Taurus - Working with Multiple JMeter Tests
In this next post in our series on Taurus, we look at overcoming the challenge of working with multiple test script files, with multiple engineers involved, when load testing complex applications. With the help of Taurus, combining several JMeter scripts into a single unified test is not only achievable but easy.
From Complexity to a Single Unified Test
The load testing of complex applications assumes that at some point you’ll have to work with multiple test script files if more than one engineer is working on test development. It means that the engineers will deliver several load test scripts, which will need to be integrated into a single load test. This is partially resolved by using the Test Fragments feature, but it requires manual management of the main test module in order to add test fragments, arrange them into Thread Groups, calculate load patterns, etc.
The situation gets more challenging when test scripts are developed by one engineer and executed by a different one, who might not have the relevant JMeter experience needed for playing with load scenarios.
This post will shed some light on the various options of running JMeter tests, setting and overriding certain test parameters of a JMeter test from YAML, and using several YAML files to build a unified single test.
JMeter .jmx Tests
In the previous article, we provided instructions on how to kick off a single JMeter .jmx test via Taurus. Now let’s learn how to start multiple JMeter tests at the same time.
Running several JMeter tests is as simple as passing paths of .jmx test script files to the bzt tool, which might look like this:
bzt ExhibitA.jmx ExhibitB.jmx /somefolder/someotherfolder/ExhibitC.jmx
Taurus automatically picks up JMeter .jmx test scripts and executes them at the same time. In regards to reporting artifacts, Taurus generates an extra set of results for each JMeter test script:
- ExhibitA.jmx artifacts:
- modified_ExhibitA.jmx - actual .jmx file with result writers injected
- kpi.jtl - main results file (CSV)
- errors.jtl - file, containing comprehensive information on errors (XML)
- jmeter.log - JMeter engine log file
- ExhibitB artifacts:
- modified ExhibitB.jmx - the same as for ExhibitA - the original .jmx with a couple of listeners added
As you can see, each JMeter script executed via Taurus will have its own set of artifacts.
In regards to reporting, the BlazeMeter Reporting module will show an aggregate output from all JMeter test scripts so that it will look like a single test, rather than a combination of 3 different ones. This is beneficial both for the test execution and the results analysis. It is also possible to use the Merge Results plugin which can plot either distinguished or consolidated graphs based on the .jtl results the files provided.
YAML Configuration File
Taurus supports running more than one existing JMeter .jmx test script from a YAML file as well.
You can instruct Taurus to kick off two (or more) JMeter .jmx tests at the same time by using 2 Scenario entries, as simple as this:
Make sure to correctly input the dash sign and indents, as they are as important as the keys and values.
The Taurus Console displays that the two tests are currently being executed in parallel:
So as you can see, Taurus makes the process of executing several tests at the same time simple and fast.
Overriding JMeter Test Parameters
We previously covered the process of executing existing JMeter tests with Taurus. Now let’s go deeper and see how we can control and override JMeter test settings from YAML configuration files.
Single Thread Group
We initially have a very simple JMeter test which hits example.com with one thread at one time.
Now let’s run it via a YAML configuration file combining the two scenarios mentioned in the previous post on Taurus:
- Run an existing .jmx script
- Set the following load pattern:
- Concurrency: 10 virtual users
- Ramp-up time: 1 minute
- Time to hold the load: 2 minutes 30 seconds
Which will result in the following YAML file (store it as i.e. ExhibitC.yml)
It’s important to keep in mind that:
- Concurrency - is the number of threads (virtual users); it is for 10 threads, not for 10 requests per second. In order to set the desired requests execution rate, consider the Throughput parameter (will be covered later on)
- Test Duration - is the sum of the ramp-up and hold-for values, so that 10 threads will be ramping up for 1 minute (start with 1 thread and add 1 more every 6 seconds), after 60 seconds when all 10 threads will be up-and-running, the test will last for another 2 minutes and 30 seconds. The total test duration will be 1m + 2m30s = 3 minutes and 30 seconds.
So if you employ Taurus in order to see a generated JMeter script in the JMeter GUI you’ll see the following:
The relevant Taurus command to open up a generated .jmx file in the JMeter GUI is as follows:
bzt ExhibitC.yml -gui
As you can see, the original values of “Number of Threads”, “Ramp-Up” and “Loop Count” were amended, the test duration was set according to YAML file and the 2 listeners were added in order to store the test metrics and detailed information on the errors.
Apart from these baseline changes it is also possible to:
- Add (or override) any property (including JMeter properties and system properties)
- Add (or override) user-defined variables
- Enable/Disable any test element starting from the Thread Group and ending up with listeners
- Amend any test element value
- And much more - to keep this post reasonably short we recommend seeing documentation on JMeter Executor for details, if any questions remain - please raise them via Taurus Support Forum
Multiple Thread Groups - Same Number of Threads
So far, so good. We see that Taurus is capable of amending the baseline Thread Group parameters (as well as other test properties). But what if we have 2 Thread Groups in the JMeter test plan? Or 3 Thread Groups? Let’s take a look.
Using the same YAML file for the “ExhibitC” and almost the same .jmx script, let’s add the second Thread Group, copy the existing one and paste it below:
Save this file as “ExhibitD.jmx” and amend the YAML configuration file to take this modified script as the scenario execution target:
Let’s run Taurus again and choose that it show the JMeter gui for the generated file:
bzt ExhibitD.yml -gui
As you can see, the “Number of Threads” for the first Thread Group is now reduced to 5. This is due to the Taurus modification. Taurus is smart enough to calculate concurrency and equally distribute it among existing (and enabled, of course) Thread Groups. The second Thread Group also has 5 virtual users. Disabled Thread Groups are not considered and left as they are in the generated .jmx test script. Taurus also does not modify the setUp and tearDown Thread Groups
Multiple Thread Groups and a Different Number of Threads
Now let’s see how Taurus handles the situation where there are multiple Thread Groups having different thread numbers.
Using the next JMeter Test Plan structure:
- Test Plan
- Thread Group 1 - 5 virtual users
- Thread Group 2 - 10 virtual users
And a Taurus YAML configuration file with:
The resulting load test configuration will look like this:
- Test Plan
- Thread Group 1 - 10 virtual users
- Thread Group 2 - 20 virtual users
Taurus proportionally distributes the configured concurrency value between 2 Thread Groups, keeping the original load pattern ratio.
Multiple JMeter .jmx Test Scripts
It is possible to override execution parameters for multiple JMeter test scripts from a YAML file as well. But you will have to set settings like concurrency, ramp-up, hold-for, etc. for each .jmx file individually. The behavior described previously (Taurus will distribute a defined concurrency among enabled Thread Groups in the .jmx file) will still persist.
The relevant configuration which executes 2 JMeter .jmx scripts provided in parallel, and that overrides virtual users, ramp-up and execution time values according to our “favorite” configuration (10 threads, 1 minute ramp-up time, 2 and a half minutes execution time) will look like:
This is how Taurus works with multiple JMeter tests and Thread Groups. If anything remains unclear, you experience any problems or have an idea on how Taurus may be improved - don’t hesitate to raise it via the Taurus Support Forum.
You might be interested in viewing our webinar, Automating Performance Tests with Taurus, which will show a live demo of how to use Taurus for both small and large-scale tests, as well as how to integrate with CI tools like Jenkins for continuous testing.
As always, leave your comments below.