ArangoDB压测小记
程序员文章站
2022-07-01 08:35:42
...
压测环境
6C 16G 测试服务器
服务器本机部署 ArangoDB3.6
手动编写压测程序,Jar包形式本机启动
场景用例
插入/更新
CASE 1(批量更新):
- 使用ArangoDB自带的 WEB Interface
- 相对复杂的CI模型结构
- Update语句,更新单一属性
- 20W条更新,首次执行59秒,
- TPS ≈ 3300
FOR c IN col_CI
UPDATE c WITH { numericVal: 5 } IN col_CI
CASE 2(单条循环插入):
- JAVA程序
- 单线程
- 10000条数据
- 单文档插入形式,模拟CI模型场景(单个Document容量相对较大)
- 执行时间:2.9s
- TPS ≈ 3000
start time:1594706936441; end time:1594706939380; Run Time:2939(ms)
//将模型提供的XML文档进行解析,拆出CI相关的内容,转换成JsonObj形式进行局部加工,单条插入
JSONObject jsonCi = JSONObject.parseObject(XmlConverUtil.xmltoJson(typeXml));
for (int i=0;i<10000;i++) {
BaseDocument document = new BaseDocument();
document.setKey("key" + jsonCi.get("Unique_code")+"_"+i);
document.addAttribute("Ref", jsonCi.get("Ref"));
document.addAttribute("Name", jsonCi.get("Name")+"_"+i);
document.addAttribute("Attributes", jsonCi.get("Attributes"));
collection.insertDocument(document);
}
//Json示例:
{
"Attributes": {
"Assets": [
{
"Description": "设备名称",
"MetriesCode": "Network.System.DevieName",
"ID": "DeviceCode",
"Code": "Network.Switch.DeviceCode",
"Source": "Manual,Metrics",
"Name": "设备名称"
},
{
"Description": "显示名称",
"MetriesCode": "Network.System.DevieName",
"ID": "DisplayCode",
"Code": "Network.Switch.DisplayCode",
"Source": "Manual,Metrics",
"Name": "显示名称"
},
{
"Description": "IP地址",
"ID": "IPAddress",
"Code": "Network.Switch.IPAddress",
"Source": "Manual,Metrics",
"Name": "IP地址"
},
{
"Description": "MAC地址",
"MetriesCode": "Network.System.MACAddress",
"ID": "MAC",
"Code": "Network.Switch.MACAddress",
"Source": "Manual,Metrics",
"Name": "MAC地址"
},
{
"Description": "OID",
"MetriesCode": "Network.System.SystemOID",
"ID": "SystemOID",
"Code": "Network.Switch.SystemOID",
"Source": "Manual,Metrics",
"Name": "OID"
},
{
"Description": "设备***",
"MetriesCode": "Network.System.SerialNum",
"ID": "SerialNum",
"Code": "Network.Switch.SerialNum",
"Source": "Manual,Metrics",
"Name": "设备***"
},
{
"Description": "厂商",
"MetriesCode": "",
"ID": "Manufactures",
"Code": "Network.Switch.Manufactures",
"Source": "Manual,Metrics",
"Name": "厂商"
}
],
"Configs": {
"Attribute": {
"Description": "该配置项的名称 ",
"MetriesCode": "Network.System.SoftwareVersion",
"ID": "SoftwareVersion",
"Code": "Network.Switch.SoftwareVersion",
"Source": "Manual,Metrics",
"Name": "软件版本"
}
},
"RuningStatus": {
"Attribute": {
"Description": "设备运行状态(正常、异常、未监控等)",
"MetriesCode": "Network.System.SNMPReachable",
"ID": "Snmpstatus",
"Code": "Network.Switch.Runstatus",
"Source": "Metrics",
"Name": "Snmp状态"
}
}
},
"Ref": "Base.CI",
"Name": "交换机_0"
}
CASE 3(批量导入大数据集):
- JAVA程序
- 单线程
- 5W条数据
- 批量插入/导入的方法,模拟CI模型场景(单个Document容量较大)
- 执行时间:80S
- TPS ≈ 620
start time:1594633266617; end time:1594633347172; Run Time:80555(ms)
//Json示例
{
"Attributes": {
"Assets": [
{
"Description": "设备名称",
"MetriesCode": "Network.System.DevieName",
"ID": "DeviceCode",
"Code": "Network.Switch.DeviceCode",
"Source": "Manual,Metrics",
"Name": "设备名称"
},
{
"Description": "显示名称",
"MetriesCode": "Network.System.DevieName",
"ID": "DisplayCode",
"Code": "Network.Switch.DisplayCode",
"Source": "Manual,Metrics",
"Name": "显示名称"
},
{
"Description": "IP地址",
"ID": "IPAddress",
"Code": "Network.Switch.IPAddress",
"Source": "Manual,Metrics",
"Name": "IP地址"
},
{
"Description": "MAC地址",
"MetriesCode": "Network.System.MACAddress",
"ID": "MAC",
"Code": "Network.Switch.MACAddress",
"Source": "Manual,Metrics",
"Name": "MAC地址"
},
{
"Description": "OID",
"MetriesCode": "Network.System.SystemOID",
"ID": "SystemOID",
"Code": "Network.Switch.SystemOID",
"Source": "Manual,Metrics",
"Name": "OID"
},
{
"Description": "设备***",
"MetriesCode": "Network.System.SerialNum",
"ID": "SerialNum",
"Code": "Network.Switch.SerialNum",
"Source": "Manual,Metrics",
"Name": "设备***"
},
{
"Description": "厂商",
"MetriesCode": "",
"ID": "Manufactures",
"Code": "Network.Switch.Manufactures",
"Source": "Manual,Metrics",
"Name": "厂商"
}
],
"Configs": {
"Attribute": {
"Description": "该配置项的名称 ",
"MetriesCode": "Network.System.SoftwareVersion",
"ID": "SoftwareVersion",
"Code": "Network.Switch.SoftwareVersion",
"Source": "Manual,Metrics",
"Name": "软件版本"
}
},
"RuningStatus": {
"Attribute": {
"Description": "设备运行状态(正常、异常、未监控等)",
"MetriesCode": "Network.System.SNMPReachable",
"ID": "Snmpstatus",
"Code": "Network.Switch.Runstatus",
"Source": "Metrics",
"Name": "Snmp状态"
}
}
},
"Ref": "Base.CI",
"Name": "交换机_0"
}
CASE 4(批量导入小数据集):
- JAVA程序
- 单线程
- 5W条数据
- 批量插入/导入的方法,模拟CI模型场景(单个Document容量较小)
- 执行时间:150S
- TPS ≈ 6600
start time:1594624916535; end time:1594625066607; Run Time:150072(ms)
//100W个简单document批量执行导入
collection = db.collection(COLLECTION_CI);
List<BaseDocument> arr = new ArrayList<>();
for (int i=0;i<1000000;i++) {
BaseDocument document = new BaseDocument();
document.setKey("key" + i);
document.addAttribute("a", "aaaa");
document.addAttribute("b", "bbbb");
document.addAttribute("c", "cccc");
arr.add(document);
}
collection.importDocuments( arr);
CASE 5(分批导入大数据集):
- JAVA程序
- 单线程
- 5W条数据
- 分50批插入/导入,每批次1000条,模拟CI模型场景(单个Document容量较大)
- 执行时间:7.3s
- TPS ≈ 6800
start time:1594720911016; end time:1594720918366; Run Time:7350(ms)
//循环嵌套,实现分批导入,性能表现良好
for (int i = 1; i <= 50; i++) {
List<BaseDocument> arr = new ArrayList<>();
for (int j =1 ;j <= 1000; j++){
BaseDocument document = new BaseDocument();
document.setKey("key" + jsonCi.get("Unique_code") + "_" + i*j);
document.addAttribute("Ref", jsonCi.get("Ref"));
document.addAttribute("Name", jsonCi.get("Name") + "_" + i*j);
document.addAttribute("Attributes", jsonCi.get("Attributes"));
arr.add(document);
}
String values = mapper.writeValueAsString(arr);
collection.importDocuments(values);
}
CASE 6(批量更新10W个CI的属性):
- JAVA程序
- 单线程
- 10W条数据
- 批量更新官方压测数据中的一个属性
- 执行时间:7.2s
- TPS ≈ 13000
start time:1594884931949; end time:1594884939175; Run Time:7226(ms)
// 选择分组为『Music』的数据集,批量更新其属性 my_prop
public Collection<String> batchUpdateCiByAttribute() {
String aql = "FOR p IN products " +
" FILTER p.group == 'Music' " +
" UPDATE p WITH { my_prop: 9 } IN products";
arangoDB = new ArangoDB.Builder().build();
final ArangoCursor<String> cursor = arangoDB.db().query(aql,null,null,String.class);
Collection<String> result = cursor.asListRemaining();
return result;
}
2、查询
CASE 7(根据主键查询指定CI设备关系到的1到4层之间的所有设备):
-
JAVA程序
-
10个线程
-
10W条数据
-
最长响应0.656s
-
最短响应0.110s
-
平均响应0.312s
-
90%线程响应时间小于0.413s
-
95%线程响应时间小于0.434s
-
错误率 0
// 查询 _key 为 "products/532035" 的CI,1到4层关系到的CI设备集合
public Collection<String> getAllCiByKey() {
String aql = "FOR v,e,p IN 1..4 OUTBOUND 'products/532035' GRAPH 'g_products'" +
" OPTIONS {uniqueVertices:'none',uniqueEdges:'path'} " +
"RETURN v";
arangoDB = new ArangoDB.Builder().build();
final ArangoCursor<String> cursor = arangoDB.db().query(aql,null,null,String.class);
Collection<String> result = cursor.asListRemaining();
return result;
}
CASE 8(查询54W 全集数据):
- JAVA程序
- 单线程
- 54W条数据
- 直接遍历全集数据,不加任何条件
- 执行时间:35s
- QPS ≈ 15000
start time:1594871712833; end time:1594871747875; Run Time:35042(ms)
// 过滤掉重复路过的节点
public Collection<String> getAllCi() {
String aql = "FOR p IN products " +
" OPTIONS {uniqueVertices:'none',uniqueEdges:'path'} " +
" RETURN p ";
arangoDB = new ArangoDB.Builder().build();
final ArangoCursor<String> cursor = arangoDB.db().query(aql,null,null,String.class);
Collection<String> result = cursor.asListRemaining();
return result;
}
CASE 9(查询10W数据集,同时匹配两个属性):
- JAVA程序
- 单线程
- 10W条数据
- 同时匹配两个属性
- 执行时间:3.7s
- QPS ≈ 27000
start time:1594877416617; end time:1594877420311; Run Time:3694(ms)
// 查询 products 集合中 group == "Book" 且 salesrank >300000 数据,限制返回10W条结果集
public Collection<String> getCiByAttributes() {
String aql = "FOR p IN products " +
" OPTIONS {uniqueVertices:'none',uniqueEdges:'path'} " +
" FILTER p.group == 'Book' && p.salesrank > 300000" +
" LIMIT 100000 " +
" RETURN p";
arangoDB = new ArangoDB.Builder().build();
final ArangoCursor<String> cursor = arangoDB.db().query(aql,null,null,String.class);
Collection<String> result = cursor.asListRemaining();
return result;
}
CASE 10(统计全集的数量):
- JAVA程序
- 单线程
- 54W条数据
- 字节统计记录数,输出结果
- 执行时间:0.16s
start time:1594877254329; end time:1594877254496; Run Time:167(ms)
public Collection<String> getAllCiAmount() {
String aql = "FOR p IN products " +
" OPTIONS {uniqueVertices:'none',uniqueEdges:'path'} " +
" COLLECT WITH COUNT INTO counter " +
" RETURN counter ";
arangoDB = new ArangoDB.Builder().build();
final ArangoCursor<String> cursor = arangoDB.db().query(aql,null,null,String.class);
Collection<String> result = cursor.asListRemaining();
return result;
}
问题发现
- 本地如果超过5W的CI量级导入(单体容量相对大),会出现 Java Heap OutOfMemory的异常。
- 超过1W以上的量级导入,最好走批量接口,InsertLoop 方式超过1W会出现处理性能下降。
- 整体测试场景是直接部署在机器上的,Docker环境中的没有测试。
结论
-
建议大于5W 的XML数据集直接走批量导入接口,InsertLoop方式适合小体量的数据集。
-
插入、更新的TPS单机环境表现来看,应对普通的业务场景还可以。
-
初步来看,批量导入的场景,性能表现会随着数据集的增大而下降,是由于DB底层Socket写序列化输出流到缓冲区的机制导致。解决方案初步思路是,遇到相对较大的数据集的时候,我们可以合理的将数据切分,分批导入即可。