基于Python组装jmx并调用JMeter实现压力测试
jmeter可以通过os命令调用python脚本,python同样可以通过系统命令调用jmeter执行压测
python调用jmeter
首先要安装jmeter,
解压并配置配置环境路径或建立软连,使得在命令输入jmeter便可以执行,如
unzip apache-jmeter-5.3.zip
mv apache-jmeter-5.3 /usr/loca/jmeter
ln -s /usr/local/jmeter/bin/jmeter /usr/bin/jmeter
ln -s /usr/local/jmeter/bin/jmeter-server /usr/bin/jmeter-server
打开jmeter并设计一个测试计划保存为testplan.jmx
使用python调用jmeter压测并生成报告
python中可以使用os.system()或supprocess.popen()调用系统命令,前者实时显示在屏幕上,后者可以获取到屏幕输出信息。
使用python调用jmeter运行及生成报告的命令如下。
import subprocess jmx_file = 'testplan.jmx' # jmx文件路径 result_file = 'result.jtl' # log_file = 'run.log' report_dir = 'report' run_cmd = f'jmeter -n -t {jmx_file} -l {result_file} -j {log_file}' # *面运行jmeter压测命令 report_cmd = f'jmeter -g {result_file} -o {report_dir}' # 生成html报告命令 # 不需要获取屏幕输出是,可以使用os.system() # os.system(run_cmd) # os.system(report_cmd) # 需要获取屏幕输出是,可以使用subprocess.popen() p1 = subprocess.popen(run_cmd, shell=true, stdout=subprocess.pipe) print(p1.stdout.read().decode('utf-8')) p2 = subprocess.popen(report_cmd, shell=true, stdout=subprocess.pipe) print(p2.stdout.read().decode('utf-8'))
组装jmx
每一测试计划为一个jmx文件,jmx实际上是xml格式的,包含一些jmeter自定义的格式规范。
常用的组件有:
- : 测试计划
- : 线程组
- : csv数据文件
- : http请求
- : http请求头管理器
- : cookies管理器
- : dns缓存管理器
- : 监听器(包括查看结果树、聚合报告等)
- : 响应断言
- <io.github.ningyu.jmeter.plugin.dubbo.sample.dubbosample></io.github.ningyu.jmeter.plugin.dubbo.sample.dubbosample>: 三方dubbo请求插件
dubbo插件jmeter-plugins-dubbo下载链接
jmx中,如果一个组件有子组件,格式为
<threadgroup 组件基本属性> ...线程组配置 </threadgroup> <hashtree> ...内部子组件 </hashtree> ··· 如果不包含子组件,则后面接一个<hashtree/> 单标签直接结束,例如: ```xml <csvdataset> ... csv数据组件配置 </csvdataset> <hashtree/>
详细的格式可以自己使用jmeter创建一个测试计划,使用文本文件打开jmx文件查看。
使用python组装jmx文件的方式有两种,一种是固定模板的数据渲染,一种类似jmeter的逐个组件添加,第一种比较简单。
通过分析jmx文件中的变量,我们使用jinja2模板语法,将其中的变量进行参数化,对需要判断和循环的变量设置if和for循环。
假设我们的测试计划结构为:
测试计划 dns缓存管理器 cookies管理器 csv文件(多个) ... 聚合报告 线程组(多个) csv文件(多个) http请求(或dubbo请求) http请求头管理器 csv文件(多个) 响应断言 察看结果树
将jmx中的关键数据抽取并组合,我使用的完整数据格式如下:
data.yaml
test_plan_name: 测试计划 comments: 测试计划描述 hosts: - name: las.secoo.com address: 112.126.120.128 cookies: clear_each_iteration: 'true' csv_files: - {'name': '数据文件1', 'path': 'data.csv', 'varnames': 'a,b', 'delimiter': ','} - {'name': '数据文件2', 'path': 'data.csv', 'varnames': 'c,d', 'delimiter': ','} thread_groups: - thread_group_name: 线程组1 comments: 线程组1描述 enabled: 'true' num_threads: 50 loops: -1 ramp_time: 0 scheduler: 'true' duration: 30 delay: '' http_samples: - request_name: http-get enabled: 'true' csv_files: - {'name': '数据文件4', 'path': 'data.csv', 'varnames': 'a,b', 'delimiter': ','} request: protocol: https domain: httpbin.org port: '' encoding: '' path: /get method: get connect_timeout: '' response_timeout: '' params: {'a': 1, 'b': 2} headers: {'token': 'aaaaaa', 'x-text': 'bbbbb'} follow_redirects: 'false' auto_redirects: 'false' use_keepalive: 'false' validate: - test_field: response_data # response_data响应文本 response_code响应代码response_message响应信息response_headers # 请求头request_headers sample_label url样本 response_data_as_document文档(文本) 请求数据request_data test_type: 2 # 2 包括 1匹配 8 相等 16字符串 否+4 或者+32 strings: ['a', 'b'] - request_name: http-post enabled: 'true' request: protocol: https domain: httpbin.org port: '' encoding: '' path: /post method: post data: {'c': 3, 'd': 4} follow_redirects: 'false' auto_redirects: 'false' use_keepalive: 'false' connect_timeout: '' response_timeout: '' - request_name: http-json enabled: 'true' request: protocol: https domain: httpbin.org port: '' encoding: '' path: /post method: post connect_timeout: '' response_timeout: '' raw_data: '{"e": 5, "f": 6}' follow_redirects: 'false' auto_redirects: 'false' use_keepalive: 'false' - thread_group_name: 线程组2 comments: 线程组2描述 enabled: 'false' num_threads: 50 loops: -1 ramp_time: 0 scheduler: 'true' duration: 30 delay: '' csv_files: - {'name': '数据文件3', 'path': 'data.csv', 'varnames': 'a,b','delimiter': '\t'} dubbo_samples: - request_name: 查询运费接口-dubbo enabled: 'true' registry: type: zookeeper group: '' address: 'zk-mall1.secoolocal.com:5181?backup=zk-mall2.secoolocal.com:5181,zk-mall3.secoolocal.com:5181' dubbo: timeout: 100 retires: 0 group: '' connections: 100 load_balance: random cluster: failfast service: 'com.secoo.business.config.rpc.service.businessconfigstoreservice' method: queryfreight headers: content-type: 'application/json' params: - type: java.util.list value: ${freight_warehouseid_sendareaid} - type: java.lang.string value: 110100 - type: java.util.list value: ${freight_warehouseid_sendareaid} validate: - test_field: response_data # response_data响应文本 response_code响应代码response_message响应信息response_headers test_type: 16 # 2 包括 1匹配 8 相等 16字符串 否+4 或者+32 strings: ['"code": 0']
对应的模板文件tpl.xml代码如下:
<?xml version="1.0" encoding="utf-8"?> <jmetertestplan version="1.2" properties="5.0" jmeter="5.3"> <hashtree> <testplan guiclass="testplangui" testclass="testplan" testname="{{test_plan_name}}" enabled="true"> <stringprop name="testplan.comments">{{comments}}</stringprop> <boolprop name="testplan.functional_mode">false</boolprop> <boolprop name="testplan.teardown_on_shutdown">true</boolprop> <boolprop name="testplan.serialize_threadgroups">false</boolprop> <elementprop name="testplan.user_defined_variables" elementtype="arguments" guiclass="argumentspanel" testclass="arguments" testname="用户定义的变量" enabled="true"> <collectionprop name="arguments.arguments"/> </elementprop> <stringprop name="testplan.user_define_classpath"></stringprop> </testplan> <hashtree>{% if hosts %} <dnscachemanager guiclass="dnscachepanel" testclass="dnscachemanager" testname="dns缓存管理器" enabled="true"> <collectionprop name="dnscachemanager.servers"/> <collectionprop name="dnscachemanager.hosts">{% for host in hosts %} <elementprop name="las.secoo.com" elementtype="statichost"> <stringprop name="statichost.name">{{host.name}}</stringprop> <stringprop name="statichost.address">{{host.address}}</stringprop> </elementprop>{% endfor %} </collectionprop> <boolprop name="dnscachemanager.cleareachiteration">false</boolprop> <boolprop name="dnscachemanager.iscustomresolver">true</boolprop> </dnscachemanager> <hashtree/>{% endif %} {% if cookies %} <cookiemanager guiclass="cookiepanel" testclass="cookiemanager" testname="http cookie管理器" enabled="true"> <collectionprop name="cookiemanager.cookies"/> <boolprop name="cookiemanager.cleareachiteration">{{cookies.clear_each_iteration}}</boolprop> </cookiemanager> <hashtree/>{% endif %} {% if csv_files %}{% for csv_file in csv_files %} <csvdataset guiclass="testbeangui" testclass="csvdataset" testname="{{csv_file.name}}" enabled="true"> <stringprop name="filename">dat/{{csv_file.path}}</stringprop> <stringprop name="fileencoding">utf-8</stringprop> <stringprop name="variablenames">{{csv_file.varnames}}</stringprop> <boolprop name="ignorefirstline">true</boolprop> <stringprop name="delimiter">{{csv_file.delimiter}}</stringprop> <boolprop name="quoteddata">false</boolprop> <boolprop name="recycle">true</boolprop> <boolprop name="stopthread">false</boolprop> <stringprop name="sharemode">sharemode.group</stringprop> </csvdataset> <hashtree/>{% endfor %}{% endif %} <resultcollector guiclass="statvisualizer" testclass="resultcollector" testname="聚合报告" enabled="true"> <boolprop name="resultcollector.error_logging">false</boolprop> <objprop> <name>saveconfig</name> <value class="samplesaveconfiguration"> <time>true</time> <latency>true</latency> <timestamp>true</timestamp> <success>true</success> <label>true</label> <code>true</code> <message>true</message> <threadname>true</threadname> <datatype>true</datatype> <encoding>false</encoding> <assertions>true</assertions> <subresults>true</subresults> <responsedata>false</responsedata> <samplerdata>false</samplerdata> <xml>false</xml> <fieldnames>true</fieldnames> <responseheaders>false</responseheaders> <requestheaders>false</requestheaders> <responsedataonerror>true</responsedataonerror> <saveassertionresultsfailuremessage>true</saveassertionresultsfailuremessage> <assertionsresultstosave>0</assertionsresultstosave> <bytes>true</bytes> <sentbytes>true</sentbytes> <threadcounts>true</threadcounts> <idletime>true</idletime> </value> </objprop> <stringprop name="filename"></stringprop> </resultcollector> <hashtree/>{% for thread_group in thread_groups %} <threadgroup guiclass="threadgroupgui" testclass="threadgroup" testname="{{thread_group.thread_group_name}}" enabled="{{thread_group.enabled}}"> <stringprop name="testplan.comments">{{thread_group.comments}}</stringprop> <stringprop name="threadgroup.on_sample_error">continue</stringprop> <elementprop name="threadgroup.main_controller" elementtype="loopcontroller" guiclass="loopcontrolpanel" testclass="loopcontroller" testname="循环控制器" enabled="true"> <boolprop name="loopcontroller.continue_forever">false</boolprop> <intprop name="loopcontroller.loops">{{thread_group.loops}}</intprop> </elementprop> <stringprop name="threadgroup.num_threads">{{thread_group.num_threads}}</stringprop> <stringprop name="threadgroup.ramp_time">{{thread_group.ramp_time}}</stringprop> <boolprop name="threadgroup.scheduler">{{thread_group.scheduler}}</boolprop> <stringprop name="threadgroup.duration">{{thread_group.duration}}</stringprop> <stringprop name="threadgroup.delay">{{thread_group.delay}}</stringprop> <boolprop name="threadgroup.same_user_on_next_iteration">false</boolprop> </threadgroup> <hashtree>{% if thread_group.csv_files %}{% for csv_file in thread_group.csv_files %} <csvdataset guiclass="testbeangui" testclass="csvdataset" testname="{{csv_file.name}}" enabled="true"> <stringprop name="filename">dat/{{csv_file.path}}</stringprop> <stringprop name="fileencoding">utf-8</stringprop> <stringprop name="variablenames">{{csv_file.varnames}}</stringprop> <boolprop name="ignorefirstline">true</boolprop> <stringprop name="delimiter">{{csv_file.delimiter}}</stringprop> <boolprop name="quoteddata">false</boolprop> <boolprop name="recycle">true</boolprop> <boolprop name="stopthread">false</boolprop> <stringprop name="sharemode">sharemode.group</stringprop> </csvdataset> <hashtree/>{% endfor %}{% endif %} {% if thread_group.http_samples %}{% for http_sample in thread_group.http_samples %} <httpsamplerproxy guiclass="httptestsamplegui" testclass="httpsamplerproxy" testname="{{http_sample.request_name}}" enabled="{{http_sample.enabled}}"> <elementprop name="httpsampler.arguments" elementtype="arguments" guiclass="httpargumentspanel" testclass="arguments" testname="用户定义的变量" enabled="true"> <collectionprop name="arguments.arguments">{% if http_sample.request.params %}{% for name, value in http_sample.request.params.items() %} <elementprop name="{{name}}" elementtype="httpargument"> <boolprop name="httpargument.always_encode">false</boolprop> <stringprop name="argument.value">{{value}}</stringprop> <stringprop name="argument.metadata">=</stringprop> <boolprop name="httpargument.use_equals">true</boolprop> <stringprop name="argument.name">{{name}}</stringprop> </elementprop>{% endfor %}{% endif %} {% if http_sample.request.data %}{% for name, value in http_sample.request.data.items() %} <elementprop name="{{name}}" elementtype="httpargument"> <boolprop name="httpargument.always_encode">false</boolprop> <stringprop name="argument.value">{{value}}</stringprop> <stringprop name="argument.metadata">=</stringprop> <boolprop name="httpargument.use_equals">true</boolprop> <stringprop name="argument.name">{{name}}</stringprop> </elementprop>{% endfor %}{% endif %} {% if http_sample.request.raw_data %} <elementprop name="" elementtype="httpargument"> <boolprop name="httpargument.always_encode">false</boolprop> <stringprop name="argument.value">{{http_sample.request.raw_data}}</stringprop> <stringprop name="argument.metadata">=</stringprop> </elementprop>{% endif %} </collectionprop> </elementprop> <stringprop name="httpsampler.domain">{{http_sample.request.domain}}</stringprop> <stringprop name="httpsampler.port">{{http_sample.request.port}}</stringprop> <stringprop name="httpsampler.protocol">{{http_sample.request.protocol}}</stringprop> <stringprop name="httpsampler.contentencoding">{{http_sample.request.encoding}}</stringprop> <stringprop name="httpsampler.path">{{http_sample.request.path}}</stringprop> <stringprop name="httpsampler.method">{{http_sample.request.method}}</stringprop> <boolprop name="httpsampler.follow_redirects">{{http_sample.request.follow_redirects}}</boolprop> <boolprop name="httpsampler.auto_redirects">{{http_sample.request.auto_redirects}}</boolprop> <boolprop name="httpsampler.use_keepalive">{{http_sample.request.use_keepalive}}</boolprop> <boolprop name="httpsampler.do_multipart_post">false</boolprop> <stringprop name="httpsampler.embedded_url_re"></stringprop> <stringprop name="httpsampler.connect_timeout">{{http_sample.request.connect_timeout}}</stringprop> <stringprop name="httpsampler.response_timeout">{{http_sample.request.response_timeout}}</stringprop> </httpsamplerproxy> <hashtree>{% if http_sample.request.headers %} <headermanager guiclass="headerpanel" testclass="headermanager" testname="http信息头管理器" enabled="true"> <collectionprop name="headermanager.headers">{% for name, value in http_sample.request.headers.items() %} <elementprop name="" elementtype="header"> <stringprop name="header.name">{{name}}</stringprop> <stringprop name="header.value">{{value}}</stringprop> </elementprop>{% endfor %} </collectionprop> </headermanager> <hashtree/>{% endif %} {% if http_sample.csv_files %}{% for csv_file in http_sample.csv_files %} <csvdataset guiclass="testbeangui" testclass="csvdataset" testname="csv 数据文件设置" enabled="true"> <stringprop name="delimiter">{{csv_file.delimiter}}</stringprop> <stringprop name="fileencoding">utf_8</stringprop> <stringprop name="filename">dat/{{csv_file.path}}</stringprop> <boolprop name="ignorefirstline">true</boolprop> <boolprop name="quoteddata">false</boolprop> <boolprop name="recycle">true</boolprop> <stringprop name="sharemode">sharemode.group</stringprop> <boolprop name="stopthread">false</boolprop> <stringprop name="variablenames">{{csv_file.varnames}}</stringprop> </csvdataset> <hashtree/>{% endfor %}{% endif %} {% if http_sample.validate %}{% for assertion in http_sample.validate %} <responseassertion guiclass="assertiongui" testclass="responseassertion" testname="响应断言" enabled="true"> <collectionprop name="asserion.test_strings">{% if assertion.strings %}{% for string in assertion.strings %} <stringprop name="97">{{string}}</stringprop>{% endfor %}{% endif %} </collectionprop> <stringprop name="assertion.custom_message"></stringprop> <stringprop name="assertion.test_field">assertion.{{assertion.test_field}}</stringprop> <boolprop name="assertion.assume_success">false</boolprop> <intprop name="assertion.test_type">{{assertion.test_type}}</intprop> </responseassertion> <hashtree/>{% endfor %}{% endif %} <resultcollector guiclass="viewresultsfullvisualizer" testclass="resultcollector" testname="察看结果树" enabled="true"> <boolprop name="resultcollector.error_logging">false</boolprop> <objprop> <name>saveconfig</name> <value class="samplesaveconfiguration"> <time>true</time> <latency>true</latency> <timestamp>true</timestamp> <success>true</success> <label>true</label> <code>true</code> <message>true</message> <threadname>true</threadname> <datatype>true</datatype> <encoding>false</encoding> <assertions>true</assertions> <subresults>true</subresults> <responsedata>false</responsedata> <samplerdata>false</samplerdata> <xml>false</xml> <fieldnames>true</fieldnames> <responseheaders>false</responseheaders> <requestheaders>false</requestheaders> <responsedataonerror>false</responsedataonerror> <saveassertionresultsfailuremessage>true</saveassertionresultsfailuremessage> <assertionsresultstosave>0</assertionsresultstosave> <bytes>true</bytes> <sentbytes>true</sentbytes> <url>true</url> <threadcounts>true</threadcounts> <idletime>true</idletime> <connecttime>true</connecttime> </value> </objprop> <stringprop name="filename"></stringprop> </resultcollector> <hashtree/> </hashtree>{% endfor %}{% endif %} {% if thread_group.dubbo_samples %} {% for dubbo_sample in thread_group.dubbo_samples %} <io.github.ningyu.jmeter.plugin.dubbo.sample.dubbosample guiclass="io.github.ningyu.jmeter.plugin.dubbo.gui.dubbosamplegui" testclass="io.github.ningyu.jmeter.plugin.dubbo.sample.dubbosample" testname="{{dubbo_sample.request_name}}" enabled="{{dubbo_sample.enabled}}"> <stringprop name="field_dubbo_registry_protocol">{{dubbo_sample.registry.type}}</stringprop> <stringprop name="field_dubbo_registry_group">{{dubbo_sample.registry.group}}</stringprop> <stringprop name="field_dubbo_rpc_protocol">dubbo://</stringprop> <stringprop name="field_dubbo_address">{{dubbo_sample.registry.address}}</stringprop> <stringprop name="field_dubbo_timeout">{{dubbo_sample.dubbo.timeout}}</stringprop> <stringprop name="field_dubbo_version"></stringprop> <stringprop name="field_dubbo_retries">{{dubbo_sample.dubbo.retries}}</stringprop> <stringprop name="field_dubbo_group">{{dubbo_sample.dubbo.group}}</stringprop> <stringprop name="field_dubbo_connections">{{dubbo_sample.dubbo.connections}}</stringprop> <stringprop name="field_dubbo_loadbalance">{{dubbo_sample.dubbo.load_balance}}</stringprop> <stringprop name="field_dubbo_async">sync</stringprop> <stringprop name="field_dubbo_cluster">{{dubbo_sample.dubbo.cluster}}</stringprop> <stringprop name="field_dubbo_interface">{{dubbo_sample.dubbo.service}}</stringprop> <stringprop name="field_dubbo_method">{{dubbo_sample.dubbo.method}}</stringprop> <intprop name="field_dubbo_method_args_size">1</intprop>{% for param in dubbo_sample.dubbo.params %} <stringprop name="field_dubbo_method_args_param_type1">{{param.type}}</stringprop> <stringprop name="field_dubbo_method_args_param_value1">{{param.value}}</stringprop>{% endfor %} <intprop name="field_dubbo_attachment_args_size">0</intprop> <stringprop name="field_dubbo_config_center_protocol"></stringprop> <stringprop name="field_dubbo_config_center_group"></stringprop> <stringprop name="field_dubbo_config_center_namespace"></stringprop> <stringprop name="field_dubbo_config_center_user_name"></stringprop> <stringprop name="field_dubbo_config_center_password"></stringprop> <stringprop name="field_dubbo_config_center_address"></stringprop> <stringprop name="field_dubbo_config_center_timeout"></stringprop> <stringprop name="field_dubbo_registry_user_name"></stringprop> <stringprop name="field_dubbo_registry_password"></stringprop> <stringprop name="field_dubbo_registry_timeout"></stringprop> </io.github.ningyu.jmeter.plugin.dubbo.sample.dubbosample> <hashtree>{% if dubbo_sample.dubbo.headers %} <headermanager guiclass="headerpanel" testclass="headermanager" testname="http信息头管理器" enabled="true"> <collectionprop name="headermanager.headers">{% for name, value in dubbo_sample.dubbo.headers.items() %} <elementprop name="" elementtype="header"> <stringprop name="header.name">{{name}}</stringprop> <stringprop name="header.value">{{value}}</stringprop> </elementprop>{% endfor %} </collectionprop> </headermanager> <hashtree/>{% endif %} {% if dubbo_sample.validate %} {% for assertion in dubbo_sample.validate %} <responseassertion guiclass="assertiongui" testclass="responseassertion" testname="响应断言" enabled="true"> <collectionprop name="asserion.test_strings">{% if assertion.strings %}{% for string in assertion.strings %} <stringprop name="97">{{string}}</stringprop>{% endfor %}{% endif %} </collectionprop> <stringprop name="assertion.custom_message"></stringprop> <stringprop name="assertion.test_field">assertion.{{assertion.test_field}}</stringprop> <boolprop name="assertion.assume_success">false</boolprop> <intprop name="assertion.test_type">{{assertion.test_type}}</intprop> </responseassertion>{% endfor %} {% endif %} <hashtree/>{% endfor %}{% endif %} {% endfor %} </hashtree> </hashtree> </hashtree> </jmetertestplan>
组装出类似data.yaml格式的数据,并使用jinja2渲染模板即可得到完整的jmx文件
pip install pyyaml jinja2
import yaml import jinja2 # 组装或读取数据 with open('data.yaml', encoding='utf-8') as f: data = yaml.safe_load(f) # 读取模板 with open('tpl.xml', encoding='utf-8') as f: tpl = f.read() # 渲染模板生成jmx jmx = jinja2.template(tpl).render(data) with open(jmx_file, 'w', encoding='utf-8') as f: f.write(jmx)
后计
在实际项目中,还涉及数据文件的拷贝,节点环境的部署,脚本的分发,报告的下载等等,可以使用paramiko或者fabric或ansible完成,压测节点数据分发的服务管理。
以上就是本文的全部内容,希望对大家的学习有所帮助,也希望大家多多支持。