Taurus: A New Star in the Test Automation Tools Constellation
When it comes to performance testing, JMeter is fantastic...but not perfect. Automation and integration with other systems can be a pain, and the tool itself comes with a steep learning curve. In this post, we'll take a look at Taurus, an open source test automation tool that extends and abstracts JMeter, and helps to overcome these challenges. Taurus provides a simple way to create, run and analyze performance tests.
Intro to Taurus - Making Test Automation Simple
Here is a bit of the obvious: the load testing process isn’t limited to developing and running a load test. Moreover, those don’t cover even close to 50% of associated activities which includes:
- load test scenario definition and design
- load test scenario parameterization and execution
- load test scenario results analysis
- integration of all above processes into the software development life cycle
- automation of all of the above
Here at BlazeMeter, we’re strong believers of the open-source doctrine and big JMeter fans, but we also know that JMeter is not perfect. We’re actively contributing to make it better, but there are still a few drawbacks. Some of these potentially will never be truly fixed, like scary XML .jmx files syntax, poor reporting capabilities and not-so-obvious approaches to set load test scenario details i.e. ramp-up, ramp-down, time to hold the load, concurrency, etc.
Beyond JMeter, the list of popular free and open source tools include alternatives such as The Grinder or Gatling, each having its own advantages and disadvantages. (If you aren’t familiar with them you’re welcome to read Open Source Load Testing Tools: Which One Should You Use?). Each tool needs to be configured differently and the load test results output format differs as well.
This is why we encourage you to start using Taurus - our free and open source automation framework, which is basically an abstraction layer over JMeter (or Grinder or Gatling or Selenium, with more tool support upcoming). Taurus provides:
- extremely simple setup and upgrading
- an easily-readable, version control friendly and unified DSL (Domain-specific language) to define load test scenarios
- the ability to execute existing JMeter (or Grinder or Gatling or Selenium) tests
- the ability to create new tests from scratch using the aforementioned DSL
- the ability to merge multiple existing test scripts and/or DSL-driven tests into a single scenario
- real-time reporting
- multiple test output formats, including but not limited to:
- a native results format of the underlying testing tool
- integration with the blazemeter.com reporting service
- console stats and pseudo-graphic charts during the test execution (which can be combined with the BlazeMeter integration
- a JUnit-compatible test results format
- an easy way to define flexible pass/fail criteria. In case the results exceed a threshold, it is possible to automatically mark test(s) as failed
- platform-independency - runs where Python and Java can run
Taurus is an acronym that stands for Test Automation Running Smoothly, which reflects its baseline mission - making the process of test configuration and execution as simple as possible.
Want to know more? Keep reading!
Taurus Installation and Usage Scenarios
Installation and Upgrade
The installation process is fairly simple, it can be done using the pip - Python Package Management system. The pip comes with every up-to-date Python distribution so given you have python executable in system PATH you should have pip as well.
The command to install Taurus (or upgrade it to the most recent version) is:
pip install --upgrade bzt
(where “bzt” is BlazeMeter Taurus)
You may also have to install/upgrade the setuptools package via the following command:
pip install --upgrade setuptools
If you’re Windows user you may also have to install Microsoft Visual C++ Compiler for Python 2.7
Make sure that you always use the most recent version of Taurus, as it contains the most recent fixes and new functionality.
You can validate the installation by typing
This command will print the Taurus version and all the possible command-line arguments.
Upon installation, the Taurus package creates a configuration folder /etc/bzt.d folder (in Windows it will be located under %DRIVE%\etc\bzt.d, where DRIVE is a pseudo-environment variable which returns the drive letter where Python is installed - usually the C drive, so the configuration directory will be located at C:\etc\bzt.d in the majority of cases). Taurus creates a 10-base.json file under that folder, and the file contains initial a default configuration that can be amended according to your requirements. Keeping default values under the /etc/bzt.d folder reduces the Taurus configuration file’s size as the settings are shared among all Taurus tests which are being executed from this machine.
Taurus Syntax: Adding the Missing DSL to JMeter
Taurus can consume configuration files written in JSON or YAML languages. The main advantage is human readability in contradiction to the XML, Python, or Scala languages, which are being used in JMeter, Grinder and Gatling respectively. So, Taurus adds a missing DSL for JMeter and other supported underlying tools. For instance, JMeter’s way of the load scenario definition isn’t very obvious and you need to perform some calculations with regards to setting virtual users’ arrival and ramp-down rates, play with the number of iterations or schedule options to set a desired load test duration, etc. This is partially resolved by custom Thread Groups provided via the JMeter Plugins project (i.e. Ultimate Thread Group or Stepping Thread Group) but other limitations persist. Moreover, JMeter .jmx files are in fact XML files, so even if you make small changes, the version control system diff won’t be very informative and it will be quite hard to figure out the nature of the commit. Taurus simplifies the aforementioned processes by providing easily readable configuration files that can be interpreted even by someone without experience with any of the supported tools.
For instance, a simple load test with 10 concurrent users, a ramp-up time of 1 minute, a duration of 2.5 minutes and hitting an example.com site with HTTP GET requests will look as simple as:
As you can see, it’s only 8 lines of simple human-readable language that represents a (more or less) complete load scenario. It’s really simple to use.
This is the whole idea of Taurus - to provide a unified simplified way of configuring and running automated tests, and represent the results in the most effective form.
Let’s assume that the above YAML configuration is stored in example.yml file. In that case, you can invoke it as simple as:
Once you hit Enter, the Taurus engine will start the test execution, including:
- downloading the latest JMeter version (with the latest plugins) to the ~/.bzt/jmeter-taurus folder (there is also an option to specify an existing JMeter location if you use custom plugins, your load generator machine is in DMZ and cannot access the Internet)
- preparing the JMeter .jmx script based on the example.yml provided
- kicking off the actual JMeter test
- displaying real-time statistics and basic ASCII-art graphs in the text console
- printing a summary to the console upon the test’s completion
- saving JMeter test results in format, understandable by your favorite JMeter Listeners, both out-of-the-box and provided by JMeter Plugins project.
This is how the Taurus test execution appears:
The post-test summary report looks like:
As you can see, initial test results can be analyzed just looking into the console output. Full details are available under the “Artifacts dir” which either has a timestamp-based, auto-generated name (as in above case), or can be explicitly set via the -d command line argument.
The contents of “Artifacts dir” are as follows:
- bzt.log - the Taurus tool log. Contains the full output of the Taurus tool, very useful for troubleshooting
- effective.json - the final Taurus configuration file in JSON format. It contains merged down input file(s), configuration defaults, any overridden properties, etc.
- effective.yml - same as an effective.json but in YAML format. Taurus doesn’t differentiate between YAML and JSON and is able to consume any format so either option is fine.
- errors.jtl - the JMeter result file in XML format with all diagnostic fields enabled. You will be able to see the full request and response details in View Results Tree listener.
- example.yml - the YAML configuration file(s) provided via the command line to Taurus
- jmeter-bzt.properties - any JMeter Properties overrides
- jmeter.log - a very default JMeter log file
kpi.jtl - the main results file, its name stands for Key Performance Indicators, it includes:
- the sampler timestamp, label and duration
- the response message and code
- whether the sampler was successful or not
- connect and latency metrics
- active thread numbers
- merged.json - a semi-final Taurus config file. It contains merged down, user-provided configuration files (YAML or JSON) but without defaults or overrides applied
- merged.yml - same as above but in YAML format
- modified_requests.jmx.jmx - same as requests.jmx plus Taurus-driven changes via YAML
- requests.jmx - YAML configuration file converted to JMeter .jmx format
- system.properties - effective JVM System Properties
So kpi.jtl and errors.jtl are for results analysis, while the others are for debugging purposes.
You can learn more about Taurus syntax at the http://gettaurus.org/ website, in particular:
You can also inspect an effective.yml file to see what does generated YAML configuration look like.
Issues, Improvements and Other Feedback
It’s worth mentioning that the main developer of Taurus is Andrey Pokhilko - the person, who stands behind Loadosophia.org and JMeter Plugins projects, and is now BlazeMeter’s Chief Scientist. So if you have any defects, suggestions, improvements requests and so on - you’re very welcome to share your thoughts via the Taurus Support Forum.
You might be interested in viewing our webinar, Automating Performance Tests with Taurus, which will show a live demo of how to use Taurus for both small and large-scale tests, as well as how to integrate with CI tools like Jenkins for continuous testing.
Start testing with BlazeMeter to enhance your Taurus script, by outting your URL in the box below: