Andrey Pokhilko is Chief Scientist at BlazeMeter. Andrey is a renowned thought-leader and innovator within the Apache JMeter performance testing community. He has worked extensively to create and develop tools that enhance the Apache JMeter protocol coverage and analytical reporting capabilities. In 2009, he founded and led this open source project to its undisputed status as the premier source of JMeter Plugins in the world. Loadosophia was similarly born from Andrey’s desire to complete JMeter’s offerings. He has world-class experience with large scale performance testing, having led the load testing team for five years at search engine giant Yandex.

Become a JMeter and Continuous Testing Pro

Start Learning

Test Your Website Performance NOW! |

arrowPlease enter a URL with http(s)
Jan 27 2016

How to Keep Using Your Favorite Open Source Tools in the Continuous Delivery Era

Are tools developed in the ‘pre-DevOps days’ doomed to extinction? By facilitating the release of software to production at any time, DevOps methodologies and Continuous Delivery processes cut the time-to-release of websites and apps from weeks to hours.


There’s no denying that these disciplines are an integral part of our future - if not our present. But how do they fit in with our past?  


Most of us are probably still using tools that were developed before words like ‘DevOps’ and ‘Continuous Delivery’ became part of our everyday vocabulary. These tools, like JMeter, Siege and Gatling, are tried and trusted and most of us don’t really want to give them up, but they weren’t created with a DevOps mindset and this is causing us big problems.


4 Ways Your Open Source Tool is Failing Your DevOps Processes (& What to do About it)


Whether you’re using JMeter, Siege, Gatling, or The Grinder, you’re probably noticed some gaps between the capabilities of your tool and the requirements of a successful Continuous Delivery process. The new open source source tool Taurus closes these gaps by creating an extra level on top of existing open source tools to enable automation.


Here are four of the biggest gaps in your tools when it comes to implementing a successful Continuous Delivery process, along with a quick explanation as to how Taurus can help you overcome them.


1. They Rely on a HUMAN Presence


Most of the tools were originally created to be used by a dedicated performance tester - but roles these days are blurred. We often have a single person writing, testing and deploying the code and/or a heavily automated process - and the tool can’t fully support this.


For example: when JMeter fails to execute a test, it will give you the exit code 0 - which actually means ‘Success’. When you build a job on the Jenkins Continuous Integration (CI) application, and you use JMeter to execute that test, you won’t be able to detect the failure in your regular ways because it will always give the exit code 0.   The tool essentially ‘lies’ to you.


Why? JMeter was created before tests were run in an automated and unattended way. When the tests are launched in the User Interface (UI), an error message will pop up to tell the performance tester that there’s an issue or he/she will immediately see that all the results are flagged red.  It relies on a human presence in the loop.


What You Can Do About This


Taurus helps with problems like this. In this case, it enables you to set the thresholds and define the failure criteria in advance so the exit code won’t appear as ‘successful’ if your KPIs aren’t met or if the test failed to launch at all due to technical issues.


2. They Don’t Support Automated Results Analysis

Let’s say you’ve managed to successfully execute the load test and now you need to analyse the results.


You probably want to know key facts like whether your results hit your success criteria, if the response time was too high, and if they fell within the expected ranges. When you automate load testing, you need a way to check the results and raise a flag if the Key Performance Indicators (KPIs) don’t fit into Service Level Agreement (SLA) ranges or pre-set thresholds. However, none of the popular open source tools provide much in terms of automating this part of the process. Most focus on generating the load but neglect the results analysis (the only exception being Gatling, which provides a rich report at the end of its run).


There are some plugins that mitigate this problem. For example: the Jenkins plugin for JMeter - which helps you set acceptance ranges. However, even this only works when you use JMeter - not The Grinder or Gatling or Locust.


What You Can Do About This


Taurus provides native formats for results from load test results (such as how many tests are failing) and presents them in a CI friendly style. Advanced CI applications like Jenkins are able to consume the numeric statistics from the tests in order to plot the graphs and trends over time.


Alternatively, these formats can be entered into a special reports server on BlazeMeter so you can easily view trends and manage results.  Here, you can actually jump from a point on a trendline into an individual test in order to investigate the source of spikes and irregularities.


3. They Don’t Give You the Required Level of Scalability


All the above named open source tools were created in the days when most of the load was generated with either one single machine or a couple of machines. Over the years, the popularity of the internet has soared and the magnitude of usage has exploded - but the tools remain the same.


For example: Gatling doesn’t have any built-in distributed testing capabilities and Siege was created before anyone started thinking about distributed testing (AWS weren’t even providing the cloud services at the time).  


But even the tools with distributed testing capabilities (i.e. JMeter and The Grinder) were built for local area networks (LAN) not cloud and geographically distributed testing. JMeter’s distributed testing allows you to easily set up the test by using the resources of 2-5 nearby machines. However, when it comes to launching several tests a day with 10-15 machines, you’ll find that you spend more time struggling with infrastructure issues than actually doing the test!


This shouldn’t be so hard. Amazon Web Services (AWS) has made it very simple for us - we should just be able to bring up a couple of machines in Amazon in different regions. There needs to be a way to set up the performance test just as easily.


What You Can Do About This


Taurus uses its distributed testing capabilities to scale what was previously unscalable. As I just mentioned, Gatling doesn't have scalability. However, with Taurus you can configure it to use the BlazeMeter cloud and then you can scale it onto hundreds of machines.


4. They Don’t Support Collaborative Ownership of Code


In the past, tests were conducted by dedicated people who maintained the test configuration in local folders or a similar form.


Today, people commit their artifacts into a version control system. According to agile practices, they create and maintain tests collaboratively and there is no single owner. Instead, the code has collaborative ownership.  Companies can’t afford to have one single engineer on it, they need transparency and to allow other staff members to take over if necessary. However, many of the open source tools don’t support this. It can be hard for new users to learn the tool, to identify changes that have occurred over time, and to understand what has changed.


For example: in JMeter’s XML-based test plan format, it is incredibly difficult to see the differences that were made through the file editing history in VCS. And if you’re unlucky to edit that file in parallel with several other people, merging the conflicts will be extremely painful! It’s much better to have a format that can merge like other source code files.


What You Can Do About This


Taurus lets you describe your load test in simple terms: you can just list the URLs to hit, the additional headers to request, and add ‘think times’. You can express several options in an easy plaintext format and get the correct load test for JMeter. It automatically translates your text configuration into a simple XML file. All of the complexity is hidden, which means that you don’t need to know Scala to create a load test on Gatling and you don’t need to know the configuration format for Siege at all. The file format used is lean and text-based - which makes it super easy for multiple people to modify it, track and merge their changes.  


JMeter, Gatling, Siege, The Grinder & Locust: Pre-DevOps but not Prehistoric

We might all be moving towards a Continuous Delivery working process, but most of us don’t want to give up on our ‘tried and tested’ load testing tools. No-one has time to learn and implement a new tool - never mind having to give up on one we trust and are comfortable with.


Ideally, we want to keep using our favorite Open Source Tools like JMeter or Siege and integrate them within our Continuous Delivery process. To facilitate this, we just need to add one more open source tool (Taurus) which was created after buzzwords like DevOps and Continuous Delivery became part of our everyday lives.


Want to Learn More About How to Evolve Your Performance Testing into the Continuous Delivery Era?


Interested in more discussion about the changing nature of performance testing in the DevOps age? View the recording of a recent webinar featuring Forrester's Diego Lo Giudice and BlazeMeter’s Michael Sage on the topic, Evolve Your Performance Testing to Succeed in the Digital Era, that will show you how leading digital brands are integrating testing into Continuous Delivery to drive business advantage.

arrowPlease enter a URL with http(s)

Interested in writing for our Blog?Send us a pitch!