欢迎您访问程序员文章站本站旨在为大家提供分享程序员计算机编程知识!
您现在的位置是: 首页

ArangoDB压测小记

程序员文章站 2022-07-01 08:35:42
...

压测环境

6C 16G 测试服务器

服务器本机部署 ArangoDB3.6

手动编写压测程序,Jar包形式本机启动

场景用例

插入/更新

CASE 1(批量更新):

  • 使用ArangoDB自带的 WEB Interface
  • 相对复杂的CI模型结构
  • Update语句,更新单一属性
  • 20W条更新,首次执行59秒,
  • TPS ≈ 3300
FOR c IN col_CI

UPDATE c WITH { numericVal: 5 } IN col_CI

CASE 2(单条循环插入):

  • JAVA程序
  • 单线程
  • 10000条数据
  • 单文档插入形式,模拟CI模型场景(单个Document容量相对较大)
  • 执行时间:2.9s
  • TPS ≈ 3000
start time:1594706936441; end time:1594706939380; Run Time:2939(ms)
//将模型提供的XML文档进行解析,拆出CI相关的内容,转换成JsonObj形式进行局部加工,单条插入
JSONObject jsonCi = JSONObject.parseObject(XmlConverUtil.xmltoJson(typeXml));

for (int i=0;i<10000;i++) {
 BaseDocument document = new BaseDocument();
 document.setKey("key" + jsonCi.get("Unique_code")+"_"+i);
 document.addAttribute("Ref", jsonCi.get("Ref"));
 document.addAttribute("Name", jsonCi.get("Name")+"_"+i);
 document.addAttribute("Attributes", jsonCi.get("Attributes"));
 collection.insertDocument(document);

}
//Json示例:
{
  "Attributes": {
    "Assets": [
      {
        "Description": "设备名称",
        "MetriesCode": "Network.System.DevieName",
        "ID": "DeviceCode",
        "Code": "Network.Switch.DeviceCode",
        "Source": "Manual,Metrics",
        "Name": "设备名称"
      },
      {
        "Description": "显示名称",
        "MetriesCode": "Network.System.DevieName",
        "ID": "DisplayCode",
        "Code": "Network.Switch.DisplayCode",
        "Source": "Manual,Metrics",
        "Name": "显示名称"
      },
      {
        "Description": "IP地址",
        "ID": "IPAddress",
        "Code": "Network.Switch.IPAddress",
        "Source": "Manual,Metrics",
        "Name": "IP地址"
      },
      {
        "Description": "MAC地址",
        "MetriesCode": "Network.System.MACAddress",
        "ID": "MAC",
        "Code": "Network.Switch.MACAddress",
        "Source": "Manual,Metrics",
        "Name": "MAC地址"
      },
      {
        "Description": "OID",
        "MetriesCode": "Network.System.SystemOID",
        "ID": "SystemOID",
        "Code": "Network.Switch.SystemOID",
        "Source": "Manual,Metrics",
        "Name": "OID"
      },
      {
        "Description": "设备***",
        "MetriesCode": "Network.System.SerialNum",
        "ID": "SerialNum",
        "Code": "Network.Switch.SerialNum",
        "Source": "Manual,Metrics",
        "Name": "设备***"
      },
      {
        "Description": "厂商",
        "MetriesCode": "",
        "ID": "Manufactures",
        "Code": "Network.Switch.Manufactures",
        "Source": "Manual,Metrics",
        "Name": "厂商"
      }
    ],
    "Configs": {
      "Attribute": {
        "Description": "该配置项的名称 ",
        "MetriesCode": "Network.System.SoftwareVersion",
        "ID": "SoftwareVersion",
        "Code": "Network.Switch.SoftwareVersion",
        "Source": "Manual,Metrics",
        "Name": "软件版本"
      }
    },
    "RuningStatus": {
      "Attribute": {
        "Description": "设备运行状态(正常、异常、未监控等)",
        "MetriesCode": "Network.System.SNMPReachable",
        "ID": "Snmpstatus",
        "Code": "Network.Switch.Runstatus",
        "Source": "Metrics",
        "Name": "Snmp状态"
      }
    }
  },
  "Ref": "Base.CI",
  "Name": "交换机_0"
}

CASE 3(批量导入大数据集):

  • JAVA程序
  • 单线程
  • 5W条数据
  • 批量插入/导入的方法,模拟CI模型场景(单个Document容量较大
  • 执行时间:80S
  • TPS ≈ 620
start time:1594633266617; end time:1594633347172; Run Time:80555(ms)
//Json示例

{
  "Attributes": {
    "Assets": [
      {
        "Description": "设备名称",
        "MetriesCode": "Network.System.DevieName",
        "ID": "DeviceCode",
        "Code": "Network.Switch.DeviceCode",
        "Source": "Manual,Metrics",
        "Name": "设备名称"
      },
      {
        "Description": "显示名称",
        "MetriesCode": "Network.System.DevieName",
        "ID": "DisplayCode",
        "Code": "Network.Switch.DisplayCode",
        "Source": "Manual,Metrics",
        "Name": "显示名称"
      },
      {
        "Description": "IP地址",
        "ID": "IPAddress",
        "Code": "Network.Switch.IPAddress",
        "Source": "Manual,Metrics",
        "Name": "IP地址"
      },
      {
        "Description": "MAC地址",
        "MetriesCode": "Network.System.MACAddress",
        "ID": "MAC",
        "Code": "Network.Switch.MACAddress",
        "Source": "Manual,Metrics",
        "Name": "MAC地址"
      },
      {
        "Description": "OID",
        "MetriesCode": "Network.System.SystemOID",
        "ID": "SystemOID",
        "Code": "Network.Switch.SystemOID",
        "Source": "Manual,Metrics",
        "Name": "OID"
      },
      {
        "Description": "设备***",
        "MetriesCode": "Network.System.SerialNum",
        "ID": "SerialNum",
        "Code": "Network.Switch.SerialNum",
        "Source": "Manual,Metrics",
        "Name": "设备***"
      },
      {
        "Description": "厂商",
        "MetriesCode": "",
        "ID": "Manufactures",
        "Code": "Network.Switch.Manufactures",
        "Source": "Manual,Metrics",
        "Name": "厂商"
      }
    ],
    "Configs": {
      "Attribute": {
        "Description": "该配置项的名称 ",
        "MetriesCode": "Network.System.SoftwareVersion",
        "ID": "SoftwareVersion",
        "Code": "Network.Switch.SoftwareVersion",
        "Source": "Manual,Metrics",
        "Name": "软件版本"
      }
    },
    "RuningStatus": {
      "Attribute": {
        "Description": "设备运行状态(正常、异常、未监控等)",
        "MetriesCode": "Network.System.SNMPReachable",
        "ID": "Snmpstatus",
        "Code": "Network.Switch.Runstatus",
        "Source": "Metrics",
        "Name": "Snmp状态"
      }
    }
  },
  "Ref": "Base.CI",
  "Name": "交换机_0"
}

CASE 4(批量导入小数据集):

  • JAVA程序
  • 单线程
  • 5W条数据
  • 批量插入/导入的方法,模拟CI模型场景(单个Document容量较小
  • 执行时间:150S
  • TPS ≈ 6600
start time:1594624916535; end time:1594625066607; Run Time:150072(ms)
//100W个简单document批量执行导入

collection = db.collection(COLLECTION_CI);
List<BaseDocument> arr = new ArrayList<>();

for (int i=0;i<1000000;i++) {
    BaseDocument document = new BaseDocument();
    document.setKey("key" + i);
    document.addAttribute("a", "aaaa");
    document.addAttribute("b", "bbbb");
    document.addAttribute("c", "cccc");
    arr.add(document);

}

collection.importDocuments( arr);

CASE 5(分批导入大数据集):

  • JAVA程序
  • 单线程
  • 5W条数据
  • 分50批插入/导入,每批次1000条,模拟CI模型场景(单个Document容量较大
  • 执行时间:7.3s
  • TPS ≈ 6800
start time:1594720911016; end time:1594720918366; Run Time:7350(ms)
//循环嵌套,实现分批导入,性能表现良好

for (int i = 1; i <= 50; i++) {
  List<BaseDocument> arr = new ArrayList<>();
  for (int j =1 ;j <= 1000; j++){
    BaseDocument document = new BaseDocument();
    document.setKey("key" + jsonCi.get("Unique_code") + "_" + i*j);
    document.addAttribute("Ref", jsonCi.get("Ref"));
    document.addAttribute("Name", jsonCi.get("Name") + "_" + i*j);
    document.addAttribute("Attributes", jsonCi.get("Attributes"));
    arr.add(document);
  }
  String values = mapper.writeValueAsString(arr);
  collection.importDocuments(values);

}

CASE 6(批量更新10W个CI的属性):

  • JAVA程序
  • 单线程
  • 10W条数据
  • 批量更新官方压测数据中的一个属性
  • 执行时间:7.2s
  • TPS ≈ 13000
start time:1594884931949; end time:1594884939175; Run Time:7226(ms)
 // 选择分组为『Music』的数据集,批量更新其属性 my_prop
	public Collection<String> batchUpdateCiByAttribute() {
		String aql = "FOR p IN products " +
				" FILTER p.group == 'Music' " +
				" UPDATE p WITH { my_prop: 9 } IN products";
		arangoDB = new ArangoDB.Builder().build();
		final ArangoCursor<String> cursor = arangoDB.db().query(aql,null,null,String.class);
		Collection<String> result = cursor.asListRemaining();
		return result;
	}
2、查询

CASE 7(根据主键查询指定CI设备关系到的1到4层之间的所有设备):

  • JAVA程序

  • 10个线程

  • 10W条数据

  • 最长响应0.656s

  • 最短响应0.110s

  • 平均响应0.312s

  • 90%线程响应时间小于0.413s

  • 95%线程响应时间小于0.434s

  • 错误率 0

ArangoDB压测小记

 // 查询 _key 为 "products/532035" 的CI,1到4层关系到的CI设备集合
	public Collection<String> getAllCiByKey() {
		String aql = "FOR v,e,p IN 1..4 OUTBOUND 'products/532035' GRAPH 'g_products'" +
				" OPTIONS {uniqueVertices:'none',uniqueEdges:'path'} " +
				"RETURN v";
		arangoDB = new ArangoDB.Builder().build();
		final ArangoCursor<String> cursor = arangoDB.db().query(aql,null,null,String.class);
		Collection<String> result = cursor.asListRemaining();
		return result;

	}

CASE 8(查询54W 全集数据):

  • JAVA程序
  • 单线程
  • 54W条数据
  • 直接遍历全集数据,不加任何条件
  • 执行时间:35s
  • QPS ≈ 15000
start time:1594871712833; end time:1594871747875; Run Time:35042(ms)
 // 过滤掉重复路过的节点
	public Collection<String> getAllCi() {
		String aql = "FOR p IN products " +
				" OPTIONS {uniqueVertices:'none',uniqueEdges:'path'} " +
				" RETURN p ";
		arangoDB = new ArangoDB.Builder().build();
		final ArangoCursor<String> cursor = arangoDB.db().query(aql,null,null,String.class);
		Collection<String> result = cursor.asListRemaining();
		return result;
	}

CASE 9(查询10W数据集,同时匹配两个属性):

  • JAVA程序
  • 单线程
  • 10W条数据
  • 同时匹配两个属性
  • 执行时间:3.7s
  • QPS ≈ 27000
start time:1594877416617; end time:1594877420311; Run Time:3694(ms)
 // 查询 products 集合中 group == "Book" 且 salesrank >300000 数据,限制返回10W条结果集
	public Collection<String> getCiByAttributes() {
		String aql = "FOR p IN products " +
				" OPTIONS {uniqueVertices:'none',uniqueEdges:'path'} " +
				" FILTER p.group == 'Book'  &&  p.salesrank > 300000" +
				" LIMIT 100000 " +
				" RETURN p";
		arangoDB = new ArangoDB.Builder().build();
		final ArangoCursor<String> cursor = arangoDB.db().query(aql,null,null,String.class);
		Collection<String> result = cursor.asListRemaining();
		return result;
	}

CASE 10(统计全集的数量):

  • JAVA程序
  • 单线程
  • 54W条数据
  • 字节统计记录数,输出结果
  • 执行时间:0.16s
start time:1594877254329; end time:1594877254496; Run Time:167(ms)
 public Collection<String> getAllCiAmount() {
		String aql = "FOR p IN products " +
				" OPTIONS {uniqueVertices:'none',uniqueEdges:'path'} " +
				" COLLECT WITH COUNT INTO counter " +
				" RETURN counter ";
		arangoDB = new ArangoDB.Builder().build();
		final ArangoCursor<String> cursor = arangoDB.db().query(aql,null,null,String.class);
		Collection<String> result = cursor.asListRemaining();
		return result;
	}

问题发现

  • 本地如果超过5W的CI量级导入(单体容量相对大),会出现 Java Heap OutOfMemory的异常。
  • 超过1W以上的量级导入,最好走批量接口,InsertLoop 方式超过1W会出现处理性能下降。
  • 整体测试场景是直接部署在机器上的,Docker环境中的没有测试

结论

  • 建议大于5W 的XML数据集直接走批量导入接口,InsertLoop方式适合小体量的数据集。

  • 插入、更新的TPS单机环境表现来看,应对普通的业务场景还可以。

  • 初步来看,批量导入的场景,性能表现会随着数据集的增大而下降,是由于DB底层Socket写序列化输出流到缓冲区的机制导致。解决方案初步思路是,遇到相对较大的数据集的时候,我们可以合理的将数据切分,分批导入即可。