Navigating your First Steps Using Taurus
This post is part of a continuing series on Taurus, an open source test automation tool that extends and abstracts JMeter, and helps to overcome various challenges. Taurus provides a simple way to create, run and analyze performance tests. In the first post, we looked at an overview of what Taurus is, the installation process and some usage scenarios. Here we will look at some of the first steps.
Execution and Reporting
As per current version 0.3.1, Taurus supports the following underlying tools:
- Selenium - execution of tests written in Python or Java
- Gatling - execution of tests written in Scala, and using the BlazeMeter reporting module
- The Grinder - execution of Jython tests, parametrizing via YAML, and using the BlazeMeter reporting module
- JMeter - execution of JMX files, parametrizing via YAML, development of new tests in YAML, merging multiple JMX and YAML combinations, console, JUnit and BlazeMeter reporting.
For the moment, JMeter support is better, but version 0.3.1 is far not the final release so we’re open to any suggestions, feature and improvements requests and bug reports. The best place to share your thoughts or problems is the Taurus Support Forum.
So in the case of JMeter, Taurus solves the following problems:
- Setting a Load Test Profile: you don’t need to calculate proper ramp-up, ramp-down nor iteration numbers, values, duration, concurrency, time to hold the load, etc. Now these can be set as name-value pairs
- Running several tests at once. It is possible to feed several .jmx input files into Taurus, moreover, it is possible to use YAML to modify .jmx scripts as required or run YAML tests in combination with .jmx
- You can use YAML only to create a JMeter test.
- You don’t need to worry about Assertions, pass-fail criteria can be defined via YAML and it is extremely easy to define the criteria and to understand them.
- Improved reporting: Taurus injects “good” reporting configuration into an existing or new JMeter test so the test results details level would be enough for an analysis and won’t cause overhead during execution.
Executing An Existing JMeter Test
There are several options on how a JMeter .jmx test script can be executed: in command-line non-GUI mode via a shell script or a java -jar ApacheJMeter.jar command, an Ant Task, via a Maven plugin or a Jenkins plugin, from Java code, etc. Taurus can be used to execute your vanilla JMeter test as well. The easiest option is passing your .jmx test script to Taurus as a parameter. Just type the following command:
Where “example.jmx” is the name of your JMeter test file, and Taurus will kick off the test and provide a real-time report and optimal results postprocessing.
Other existing JMeter .jmx test execution options via Taurus are: passing JMeter test script(s) via “Taurus options” and passing JMeter test script(s) via YAML or JSON configuration files.
You can set various Taurus options via -o command line argument, so “telling” Taurus to use your test script is as simple as:
bzt -o execution.scenario.jmx=example.jmx
With regards to YAML, feeding .jmx files into Taurus is also a fairly easy process and assumes only 3 lines:
It is also possible to provide more than one JMeter .jmx test script via any of the above execution approaches. In such a case, Taurus will merge the load profiles and calculate the effective load pattern based on the merge rules set. This will be covered in depth in an upcoming post, so stay tuned and keep our blog on your radar to learn about the most prominent Taurus features. If it is something you need right away don’t hesitate to raise the question at the Taurus Support Forum and if your question is “hot” - the resolution will be added to the Taurus Documentation.
Viewing a Generated Test in JMeter GUI
If you’re a JMeter expert, you might want to see the resulting .jmx script prior to execution. Opening the modified_requests.jmx file in JMeter GUI can be done as simple as appending a -gui argument to a bzt command line like:
bzt example.yml -gui
It will automatically parse the config files provided, merge them into one aggregate .jmx script and open it in JMeter GUI so you could see what the final test plan will look like:
If you have ever opened a .jmx script with a text editor, you can guess how it looks like under the hood. So 9 lines of YAML code (155 bytes) results in 127 lines of XML the size of 5kb which is 50 times larger.
As was already mentioned, in the case of JMeter, Taurus provides the following reports:
- a summary console during run time
- statistics at the test’s end
- 2 .jtl result files:
- CSV - for successful samplers
- XML - for failed samplers containing full details of requests/responses
In addition there are 2 more reporting options:
JUnit XML Reporter
If you need to export your test results in the JUnit XML format, the JUnit Reporting module can be enabled by simply adding - module: junit-xml line to “reporting” section of the YAML config file. Let’s add it to our example.yml file.
Possible options are:
- filename - you can set an output file full path. By default it will be a xunit.xml file in the Artifacts folder
- data-source - you can choose between
- sample-labels - the report will contain URLs (or sample names) with errors (if any)
- pass-fail - the report will be based on pass/fail criteria information
(See the JUnit XML Reporter chapter for comprehensive information on JUnit reporting module configuration, as the reporting module configuration mentioned above only enables the JUnit output module with very default settings.)
Taurus can send interim and final test results to BlazeMeter. Even more, you can watch the progress of the test currently being executed on the BlazeMeter website. This is available on all BlazeMeter pricing tiers including the free version and it even works in anonymous mode - you don’t even have to be registered with BlazeMeter to use the feature. However, it is recommended to have your own account at least for the following 3 reasons:
- You will be able to use another great feature - the ability to compare different test run results
- The reports retention period will be guaranteed. We try to keep reports as long as it is possible, but when it comes to some thresholds, we have to delete them to free up some capacity for the new ones.
- It will be possible to track KPI trends for the test series on a single graph.
In order to see a report for the running a test on BlazeMeter systems, just append the -report parameter to the Taurus startup command and it will automatically open your default browser and navigate to the page with the results.
So if you run the aforementioned example.yml file as follows:
bzt example.yml -report
Taurus will kick off test scenario defined in the example.yml file and open the test report page in your browser. Data is being sent every 30 seconds (the default value, though it can be amended via YAML) and the report can be shared with any person who is interested in it. If you’re not very familiar with BlazeMeter reporting, the process of the test execution via Taurus with the BlazeMeter reporting module enabled appears onscreen as:
Regarding, the “external” source name, this means that the test originates from an external source. In our case, it originates from Taurus and the report URL, which is the same for the console and browser instance.
To get an idea on which metrics you can see, take a look at the Sample Report and remember that that the report for running a test is updated almost in real time (every 30 seconds by default or according to what you specify in configuration).
You can configure Taurus to use your own BlazeMeter API key in a couple of lines of YAML code.
First of all, obtain your API key from your BlazeMeter Profile page.
Once you have the key you can add to it ~/.bzt-rc .
Here is an example of setting the BlazeMeter API key directly in the .bzt-rc file just to give you an idea of how it should look like.
Never share your API key with anyone as it gives full programmatic access to your account and your test data may be accessed or deleted.
If you’re sitting behind the proxy server you can set the proxy host, port and credentials under top-level configuration settings.
Limitations and Constraints
Kindly keep in mind the next few recommendations while using Taurus:
Wait for a Graceful Shutdown of Taurus
If you need to stop the test for any reason, you can do it via hitting Control+C keys combination in the console session where Taurus is running. But please do it only once and be patient, as a “massive” load test cannot stop immediately. It is still possible to terminate the test session by typing the Control+C combination one more time, but in this case the threads will be ending in an abnormal way, resulting in extra errors. Additionally, the post-processing logic may fail to produce a well-formed CSV and/or XML files so you’ll have to spend extra time sorting that out.
Mind YAML Formatting
Remember that YAML files are very sensitive to indents, so it’s important to mind the gaps i.e. before the “concurrency”, “requests” or “url” stanzas. If a single space break is missing, Taurus won’t be able to interpret the configuration file. Unfortunately, neither YAML nor JSON has a DTD schema, so currently it isn’t possible to provide an IDE with auto-completion and syntax checking, so you need to be careful.
Keep Your BlazeMeter API Key Safe
Never share your BlazeMeter API key with anyone.
- Don’t commit it to VCS
- make sure that the log files you post on the Support Forum don’t contain the API key
- the best place to keep it is either a local .bzr-rc file or i.e. passing it as a command-line argument like:
bzt -o modules.blazemeter.token=5b06f9816960191f49c4 -report
If running through a Continuous Integration system. Replace the reference token 5b06f9816960191f49c4 with your own one.
As always if you have any questions or comments just leave a message in the discussion form below this post and we’ll get back to you shortly.