Automated Performance Tests in Jenkins Continuous Integration Environments
1. Why is Continuous Integration (CI) Important?
Continuous Integration (CI) is important for developers, as they can integrate code into a shared repository and verify each check-in with an automated build, deploy and test the process. This enables them to quickly find and remove bugs and measure the impact of each code change to application performance.
Users who want to add performance testing should identify their goals together with business analytics. This is important for determining which types of test should be run at each stage of the CI process.
Here are the types of tests you can run:
● Load Tests are conducted to understand the behavior of the system under a specific expected load.
● Stress Tests are used to understand the upper limits of capacity within the system.
● Soak Tests determine if the system can sustain the continuous expected load.
● Spike Tests determine if the system can sustain a suddenly increasing load generated by a large number of users.
● Isolation Tests determine if a previously detected system issue has been fixed by repeating a test execution that resulted in a system problem.
So for example, for Continuous Delivery, running soak and spike tests in a pre-production environment helps understand how applications perform with continuous expected loads and spikes, with each production deployment.
2. Using Taurus to Easily Run Load Tests
Taurus simplifies the automation of performance testing, is built for developers and DevOps, and relies on JMeter, Selenium, Gatling and Grinder as underlying engines. It also enables parallel testing, its configuration format is readable and can be parsed by your version control system, it’s tool friendly and tests can be expressed using YAML or JSON.
If you don’t have Taurus yet, install it.
Let’s look at this simple configuration in YAML.
This test ramps-up to 5 users for 30 seconds, then sustains this load for 1 more minute. The number of hits per second (throughput) is 3, and they send requests to the url: http://aws-uut:8000/
- concurrency: 5
This is a clear and simple test. The same test in JMeter would be very long, with lots of text that is hard to understand, and changes made into it would be unclear.
You can also split the file into 2 (or more) config files which will make it easier to maintain the test code with the source code, especially if many developers are working on the same test.
- label: Home
- concurrency: 5
- include-scenario: my_scenario
The first config file defines the scenario.
The second config file defines the execution parameters.
More complex scenarios can include assertions for validating response data, group requests into transactions and pass-fail criteria.
To run the test, type bzt and the name of the YAML file.
In this case: bzt simple-travel.yml scenario-home-page.yml
Give it a few seconds to run and look at the dashboard. These include response times, percentiles, response codes and local CPU.
This is the dashboard for the test after 83% of the test. All response codes are 200s, CPU is 6% and memory is 70%. The two labels that are being hit are the Home Page and the Reservation page, and it’s successful with no failures.
After the test is finished, the results are shown collectively:
Now, let’s understand how Taurus works with BlazeMeter.
3. Viewing Taurus Results on BlazeMeter
By using the -report parameter, we can send the results to BlazeMeter in real-time.
Taurus is working in the background, but we can see all the reports and analyze the results on BlazeMeter, in a more sophisticated and attractive fashion.
Finally, let’s see how it all comes together with Jenkins so you can include it in your Continuous Integration process.
4. Using Jenkins with Taurus and BlazeMeter for Load Testing
Now, open Jenkins.
1. Click ‘Configure’ on the left.
2. Choose the Jenkins Build Triggers tab on top.
3. Scroll down to the Build section and enter the command for the test.
In this case: bzt simple-travel.yml -report
If you have a complete CI deployment process, make sure the test is kicked off with it.
To see the test details, check out the Console Output.
Results can also be shown on BlazeMeter.
We can add details easily by changing the test config file in one place on Taurus.
The test will now run on Taurus and on Jenkins. Congratulations! Now you can see the results and analyze load testing as a part of your full development process.