3 Reasons Why Open Source Alone Isn't Enough for JMeter Experts
If you want to master JMeter performance testing, it’s worth taking a leaf out of the book from the experts.
Advanced JMeter users usually come from a performance engineering background. They devote themselves to delivering results and verifying that their systems are capable of enduring great demand and performance challenges. They need to ensure they can support extremely high volumes of concurrent users and achieve a great ‘hits’ rate.
So why do many of them rely on other tools in addition to JMeter? Are open source tools alone enough?
First off, let’s look at why so many developers and testers use JMeter. The most obvious answer is that, being an open source tool, you can use JMeter free of charge. You simply download the zip file, extract it onto your computer, and you’re good to go. Conversely, other proprietary tools, such as HP Loadrunner, are incredibly expensive to set up and run. Additionally, as an open source tool, JMeter allows you to modify your source code and create your own specific plug-ins.
Aside from being open source and free, there’s one other very important reason for JMeter’s popularity:
JMeter is THE standard in the world of performance testing.
If you’re looking for a performance testing solution or the means to perform load testing, JMeter will always be near the top of your list. It has a widespread and long standing reputation for stability and performance.
Although many experts rely on it, there are some drawbacks to using this open source solution “out-of-the-box”. So perhaps it’s no surprise that we’re seeing an ongoing trend of engineers searching for a more comprehensive, enterprise-like solution.
Want to be a JMeter Expert? Check out our JMeter 5 -day training course
Disadvantages of Open Source Solutions
While open source solutions have many advantages, they also have many disadvantages. Open documentation can be incomplete at times, bugs manage to work their way into code, and some JMeter behaviors are inflexible - meaning you can’t change them even if you want to. For example, if one of your servers doesn’t load in a JMeter distributed environment, it will cause your entire cluster to fail whatever tests are running, along with any operating engines. Let’s say you started your test with one console and 14 engines, then one of the servers did not load properly due to a communication error or some local issue with the server on which it was hosted. This would cause the console to fail the entire test and stop all 14 servers. This is especially disappointing if the problematic server was not crucial to your test results, in which case you could have still obtained meaningful results with only 12 or 13 servers.
So, for many JMeter experts, open source isn’t enough and many solutions are emerging on the market to complement JMeter’s offerings. I work with many developers and testers who are using BlazeMeter for this purpose. It’s 100% compatible with JMeter so it’s able to overcome bugs and offer features to overcome inflexible behaviors like the one mentioned above. For example, if you aim to start your test with 14 engines and one of them doesn’t load correctly, the test will still continue to run. What's more, with the dynamic Master/Slave feature, you can add the omitted server to the test afterwards for complete results.
For the second part of this post, I’m going to look at three more reasons why I see so many JMeter experts turning to tools like BlazeMeter for performance testing.
IT Management and Capacity Allocation
Aside from running load tests, performance engineering experts also have to manage their IT environments, which can be quite complex, time consuming, and expensive. For example: one of our customers maintains an environment with tens of thousands of users and relies on BlazeMeter to better manage their servers. Sometimes their internal servers are shared with other groups, in which case time slots need to be allocated to load their JMeter environment and run tests. Other times, server management is used within the same department, be it testing or DevOps, due to additional running services. As expected, the necessary JMeter and Java updates must be put in place in addition to taking care of any new or lingering issues within the operating system. As a result of the unpredictable nature of IT management tasks, time management and organization become critical to seamless testing and operations. In this particular case, BlazeMeter provided over-dedicated servers, saving this customer a real headache, and enabling them to simply load and start their test.
As you may know, out-of-the-box JMeter reporting capabilities are very basic. The graph system that JMeter offers can be difficult to understand and lots of JMeter users I have spoken with don’t find them very intuitive. For example: it’s very hard to analyze and identify bottlenecks, or critical KPI metrics, such as hit rates, requests per second, and response times using the common JMeter listeners. While raw data is available, what usually happens is that experts take their raw data and use external services, such as Loadosophia to generate the associated graphs and analyses. All you have to do is upload your results to the service and in return, it provides you with respective graphs and analyses.
BlazeMeter’s graphs allow you to identify if any bottlenecks exist within a number of seconds, providing accompanying metrics with zoom capabilities for any area of the graphs, along with all of the raw JMeter data. That way, if you need to further analyze your environment, you’re much better equipped to do so.
Dynamic and Flexible Performance Test
It’s useful to be able to edit and play around with your test while it’s running. For example, let’s say an e-Commerce website that plans on supporting 50,000 concurrent users wants to run a three hour test. After 30 minutes of generating positive, stable results, there are still 2.5 hours of testing that can be used. The ability to update the test properties is important in cases where you need to verify the environment stress and recovery point. In this case, you basically need the ability to add more and more concurrent users until a breaking point is reached where the system displays nothing but errors. Similarly, this procedure can be performed with hit rates. For instance, if the system needs to support a 2,000 request per second hit rate, you’ll need to test your current environment to see if it can withstand that hit rate. With the current JMeter version, there’s no way to dynamically increase that hit rate and pinpoint where the system will break. Moreover, you’d probably like to relieve the system and return to the originally supported 2,000 hits per second and see how long it takes for the server to recover, if at all.
[Two of BlazeMeter's most appealing features are the Dynamic Multi-Test and Dynamic Properties. These features enable you to make changes to a number of your test's KPIs and parameters (i.e. hit rate and profile) during the actual test. These two features can add a level of sophistication to tests, enabling users to start and stop different test elements or application components using 'IF' or 'WHILE' controllers. BlazeMeter provides you with all of the tools you need in one place, removing the need to run several separate tests. As a result, you can make changes in one test that will generate a single, extensive and comprehensible report. ]
Seeing that JMeter is an open source solution, it can be run on any cloud provider (i.e. AWS, RackSpace, HP, Google, and so on). However, utilizing other providers will result in initial costs as well as server and environment management. Aside from having to download the appropriate versions of JMeter and Java, any configurations, images, as well as data collection and aggregation from different servers will fall on your shoulders. The amount of time it takes to accomplish all of the tasks necessary to actually run JMeter and start testing diverts performance engineers from their initially assigned tasks.