Running Your Load Tests With Test Failure Criteria - A Guide
Test Failure criteria for tests metrics and KPIs are an inherent part of performance tests, since they ensure your product is performing according to the requirements. For example, “The average response time for the users needs to be no more than 600 milliseconds”, or “I expect the average latency of my servers to be no more than 200 milliseconds”.
How can you check if a specific test breached your thresholds? In this blog post, I will elaborate about two ways: by using Taurus in PassFail mode, and by using Jenkins with Taurus or JMeter.
You can also test manually by checking if the metrics you wanted to measure reached values that are above your thresholds. But this is a thorough but very slow process, and it can be a real pain! You can also use the Apache JMeter™ AutoStop Listener. Learn more from this blog post.
Running Load Tests with PassFail Criteria in Taurus
Taurus is an open-source test automation framework. It uses simple YAML syntax to run open-source load and performance tests, for tools like JMeter, Selenium, Gatling and Locust.
Taurus provides a module called PassFail, which helps you automate the threshold testing process. In Taurus you can set pass/fail criteria in a YAML file, and add this YAML file to your Taurus execution. The Taurus pass/fail module supports many different load metrics, such as Response Time, Latency, Error rate and more. You can also assign the criteria to all the labels in your test, or only to specific labels. Learn about all the Test Failure options in Taurus from this link.
Let's look at an example. Here is a Taurus execution configuration:
Save this file as execution.yml. Now lets add pass/fail criteria.
The above configuration sets two pass/fail criteria:
- If the average response time (for all the labels in the test) goes over 10 milliseconds for 7 seconds in a row - stop the test as failed.
- If the number of hits (number of responses) for a specific label called ‘reserve’ is more than 10 for 13 seconds in a row - continue with the test, but mark it as failed.
Save this pass/fail configuration in a different YAML file and called it passfail_config.yml
Now that we have finished with the configuration, let’s run the test.
Taurus tests are run directly from the terminal, with the ‘bzt’ command. When we have multiple YML files that we want to use in our test we simply add them one after the other:
After a few seconds, the Taurus console report will be displayed automatically.
As the average response time for the test was more than 10 milliseconds for 1 second - we get an alert. The alert appears even though we haven’t reached the threshold yet. The purpose is to let us know we might have a problem soon.
After 7 seconds from the beginning of the test, the average response time was more than 10 milliseconds for 7 seconds in a row. This means that our first criteria was met, so the test will be stopped and marked as failed as expected.
Since the first condition was met, we didn't have a chance to check out the second condition. The two are unrelated and the test checks each one separately.
By using the Taurus passfail module, you don’t have to manually examine tests results anymore. If one of your tests met the criteria you set, it will fail the test automatically and you will get a notification for this in the report and logs.
Taurus is a very good tool for test automation. Now let’s see how we can schedule Taurus tests, by using open-source CI tool Jenkins.
Running Load Tests with Test Failure Criteria in Jenkins
We recommend running load tests as part of your Continuous Integration process. This ensures you are constantly ensuring your code changes or product updates don’t crash your website or app. One of the most popular open-source tools for continuous integration is Jenkins.
Both JMeter and Taurus can be integrated into Jenkins, and Jenkins can then schedule automated tests.
We will explain how to run Taurus with its PassFail configuration, with Jenkins. You will see it is quite easy. To learn how to run Jenkins with JMeter, click here.
As Taurus is a command line tool, you can simply use the ‘Execute Shell’ build step in your Jenkins project, to run the same execution command we ran locally:
After saving our Jenkins project, we can go ahead and build it by clicking on the ‘Build Now’ button (or use a cron job for a scheduled execution).
Once the build started, open the Console Output:
In the Console Output you can see the results of your Taurus load test, and in this case, the failure of the test due to the pass/fail criteria being met.
As the test failed, it will fail the Jenkins build.
To schedule jobs in Jenkins you can use the ‘Build Triggers’ section in your Jenkins project. There are a number of ways to schedule jobs: Periodically by using CRON (see more details here), by clicking ‘Build when a change is pushed to GitHub’ and more. This way you can easily integrate your Performance Tests into your CI process, and see which builds failed due to a pass/fail criteria being met.
Congratulations! You now know how to run tests with thresholds by using Taurus and Jenkins.
To learn more about BlazeMeter, which can show Taurus and Jenkins test analytics and run scalable load tests, just put your URL or JMX file in the box below and your test will start in minutes. Or, request a BlazeMeter demo.