Run massively scalable performance tests on web, mobile, and APIs

Request a Demo
Jul. 6th, 2015

Guest Post: Continuous Integration Performance Testing

Discovering performance problems as early as possible would save you time and money. How about finding them out right after a commit? What about changing environment settings? How could you be sure that all 20+ nodes of your application are healthy running your changes? Some manual testing or one-thread automated testing simply wouldn’t do it. How about generating some smoke load?


To solve the challenges above and even more, you can add Smoke and Benchmark performance tests to your continuous integration workflow. To do so I used:

 

  • Jenkins as a continuous integration (CI) agent
  • JMeter for load and performance testing
  • Taurus to orchestrate JMeter in Jenkins
  • BlazeMeter for tests reporting


The nice part about these tools? They are all free.

 

Prerequisites

 

  1. Configure continuous deployment of code to environments.
  2. Create a JMeter script to cover your key functionality: add assertions to all requests, parameterize the script with all possible input values (it’s not manual testing, you can literally add all 10K search strings that your customers are using in production), make it real-world like with addition of probabilities of each of the request.

Configuring Performance Tests in CI

 

Now let’s add performance tests to our CI. We want to run a smoke load test with a few virtual users after each build and run a benchmark test for every release build.

 

  1. Install Taurus. Taurus is an open source tool to configure and run different load/performance test setups. A single JMeter script can be used to run various types of tests: ramp-up, smoke, benchmark etc. Here are the installation instructions
  2. Create configuration YML files for each of the environments and test types. See example below.
  3. Run the test with command:bzt smoke.yml
  4. Configure Jenkins to run a test on each build. The Jenkins freestyle project can be used.

Taurus Yml file example for a smoke test (example gist)

 

#! /usr/local/bin/bzt
 ---
 execution:
   concurrency: 10
   ramp-up: 30s
   hold-for: 5m
   scenario:
     script: Smoke_script.jmx
     properties:
         my-hostname: YOUR_HOST.COM
  
 reporting:
   - module: blazemeter
     test: Smoke Test
   - module: junit-xml
     filename: report.xml
     data-source: pass-fail
   - module: fail-criteria
     criterias:
     - avg-rt>250ms for 30s, continue as failed
     - failures>5% for 5s, continue as failed
     - failures>50% for 10s, stop as failed
     - rc5??>0 for 10s, continue as failed
     
 modules:
   blazemeter:
     token: Your Blazemeter Token
     browser-open: false
   console:
     disable: true
     
 settings:
   check-interval: 5
 

 

The sweet part is that now you can use one JMeter script.jmx for all environments and test types: all you need to do is change the hostname and runtime settings, like the number of concurrent users in the yml configuration.

 

Enjoy testing!

 

Notes

 

What is Taurus Doing?

 

When you are running Taurus (bzt) it:

 

- Overrides settings in the JMeter script and creates a new one in the test directory (name will be prompted to the console);

- Starts JMeter in a non-UI mode using the new script;

- Disables some of the plugins specified in the script like Loadsophia plugin;

- Sends report data to BlazeMeter in a real time, so you can analyze and share it on the fly;

 

YML Options

 

Lets walk through some of the options in the YML file:

 

- Execution is mainly used by Taurus to override your script settings in the JMeter script “Thread group” settings. Settings that are not specified in the section will be taken from the JMeter script.

- 'Properties' is custom parameters. In the JMeter script you can use with property functions: ${__P(my-hostname,YOUR.DEFAULT.HOST.COM)}

 

More about options in Taurus docs.

 

Reporting

 

Reporting module BlazeMeter adds test data to the BlazeMeter account reporting specified in the test section. Your API key can be found under profile settings. You might as well use anonymous reporting, but you will lose the feature I’m enjoying the most: comparison trends for response times, errors and hits/sec for different builds:

 

 

Another nice trend you can configure is right in the Jenkins with the junit plugin and Taurus module junit-xml to report your criterias status to xml file. Criterias can be specified in Taurus configuration yml, check the config above.

 

 

You could do much more with Taurus and Jenkins. Check following Taurus + Jenkins YouTube tutorial.

 

This is a guest post by Marianna Bezler, a software performance engineer based in San Francisco. It first was published on her site, and has been posted here with her permission. 

 

You might also find these useful:

Interested in writing for our Blog? Send us a pitch!