Become a JMeter and Continuous Testing Pro

Start Learning

Test Your Website Performance NOW!

arrow Please enter a URL with http(s)
Jul 06 2015

Guest Post: Continuous Integration Performance Testing

Discovering performance problems as early as possible would save you time and money. How about finding them out right after a commit? What about changing environment settings? How could you be sure that all 20+ nodes of your application are healthy running your changes? Some manual testing or one-thread automated testing simply wouldn’t do it. How about generating some smoke load?

To solve the challenges above and even more, you can add Smoke and Benchmark performance tests to your continuous integration workflow. To do so I used:


  • Jenkins as a continuous integration (CI) agent
  • JMeter for load and performance testing
  • Taurus to orchestrate JMeter in Jenkins
  • BlazeMeter for tests reporting

The nice part about these tools? They are all free.




  1. Configure continuous deployment of code to environments.
  2. Create a JMeter script to cover your key functionality: add assertions to all requests, parameterize the script with all possible input values (it’s not manual testing, you can literally add all 10K search strings that your customers are using in production), make it real-world like with addition of probabilities of each of the request.

Configuring Performance Tests in CI


Now let’s add performance tests to our CI. We want to run a smoke load test with a few virtual users after each build and run a benchmark test for every release build.


  1. Install Taurus. Taurus is an open source tool to configure and run different load/performance test setups. A single JMeter script can be used to run various types of tests: ramp-up, smoke, benchmark etc. Here are the installation instructions
  2. Create configuration YML files for each of the environments and test types. See example below.
  3. Run the test with command:bzt smoke.yml
  4. Configure Jenkins to run a test on each build. The Jenkins freestyle project can be used.

Taurus Yml file example for a smoke test (example gist)


#! /usr/local/bin/bzt
   concurrency: 10
   ramp-up: 30s
   hold-for: 5m
     script: Smoke_script.jmx
         my-hostname: YOUR_HOST.COM
   - module: blazemeter
     test: Smoke Test
   - module: junit-xml
     filename: report.xml
     data-source: pass-fail
   - module: fail-criteria
     - avg-rt>250ms for 30s, continue as failed
     - failures>5% for 5s, continue as failed
     - failures>50% for 10s, stop as failed
     - rc5??>0 for 10s, continue as failed
     token: Your Blazemeter Token
     browser-open: false
     disable: true
   check-interval: 5


The sweet part is that now you can use one JMeter script.jmx for all environments and test types: all you need to do is change the hostname and runtime settings, like the number of concurrent users in the yml configuration.


Enjoy testing!




What is Taurus Doing?


When you are running Taurus (bzt) it:


- Overrides settings in the JMeter script and creates a new one in the test directory (name will be prompted to the console);

- Starts JMeter in a non-UI mode using the new script;

- Disables some of the plugins specified in the script like Loadsophia plugin;

- Sends report data to BlazeMeter in a real time, so you can analyze and share it on the fly;


YML Options


Lets walk through some of the options in the YML file:


- Execution is mainly used by Taurus to override your script settings in the JMeter script “Thread group” settings. Settings that are not specified in the section will be taken from the JMeter script.

- 'Properties' is custom parameters. In the JMeter script you can use with property functions: ${__P(my-hostname,YOUR.DEFAULT.HOST.COM)}


More about options in Taurus docs.




Reporting module BlazeMeter adds test data to the BlazeMeter account reporting specified in the test section. Your API key can be found under profile settings. You might as well use anonymous reporting, but you will lose the feature I’m enjoying the most: comparison trends for response times, errors and hits/sec for different builds:



Another nice trend you can configure is right in the Jenkins with the junit plugin and Taurus module junit-xml to report your criterias status to xml file. Criterias can be specified in Taurus configuration yml, check the config above.



You could do much more with Taurus and Jenkins. Check following Taurus + Jenkins YouTube tutorial.


This is a guest post by Marianna Bezler, a software performance engineer based in San Francisco. It first was published on her site, and has been posted here with her permission. 

arrow Please enter a URL with http(s)

You might also find these useful:

Interested in writing for our Blog?Send us a pitch!