Is JMeter a Viable Open Source Alternative to LoadRunner?
Performance testing for websites is something which, in the past, was mostly done with software tools installed on local machines. Take for example LoadRunner, probably the most well known tool in the world of performance testing (it’s been around for a while).
LoadRunner is made up of two components: the IDE, in which scripts are created using a Windows-based application (Windows support only), and the load generators, which are supported on both Windows and Linux based.
For a long time LoadRunner (previously owned by HP, then acquired by MicroFocus) was the only serious contender in the performance testing market for enterprises, with costly licensing fees and lock in plans. Additionally, as with many outdated licensing-type software, transferring the LoadRunner license from one machine to another can be a nightmare.
LoadRunner is known to cover a wide range of testing protocols, and has extensive reporting capabilities, but licensing requirements and lack of open source capabilities mean that it isn't ideal for companies working to implement continuous testing practices. When agile methodologies are implemented, but companies continue to use tools that need specialised knowledge and limited licensing, testing can become a bottleneck. This is why more and more companies (even enterprises) are turning to open source testing tools like Apache JMeter.
JMeter - Advanced Open Source Testing
Apache JMeter is an open source project that is very similar to LoadRunner as far as technical capabilities go. It essentially runs simulated, virtual users that make all of the requests that a real browser would make.
Many people were looking for an open source alternative for performance testing as a result of the high price of LoadRunner. Accordingly, JMeter was born as an Apache project, under the Apache umbrella, and has been fairly active over the past few years. Technically speaking, it is an application written in Java, meaning it can run on any machine, Windows, Linux or Mac. It is GUI based, so elements can be added via the GUI tool, such as http requests. While the interface may seem a bit overwhelming at first, in a way, JMeter is like the cockpit of a jetliner airplane. It is most likely that 4 or 5 dials and levers are actually needed to fly the plane, though supplemental equipment is there just in case. Most of the time, only 4 to 5 elements in JMeter are used, however the number of features displayed can be overwhelming.
One of the cool things about JMeter is that because it is an open source project, it supports a plug-in architecture where functionalities can easily be added by writing a Java plug-in. One of the areas where JMeter differs from some of the other commercial performance testing tools is in its focus on defining load in terms of concurrent users, opposed to defining load in terms of requests per second. For that matter, API performance testing is usually done based on requests per second, which resulted in the creation of a third party plug-in to support detailed throughput shaping based on transactions/requests per second.
The JMeter Challenge
Like many open source, developer-oriented platforms, the real challenge with JMeter is that it was written with a focus on technical functionality and not on user experience. For example, in JMeter, a virtual user is called a ‘thread’, whereas other tools use the terms 'virtual user', 'simulated user', or even simply, 'user'. JMeter also makes use of the term 'sampler', which other tools refer to as a 'request'.
While the community has added a significant amount of functionality, ultimately, JMeter lacks a number of features and capabilities that are required for enterprise usage. There are a lot of missing components in terms of scaling, server coordination, collaboration and access levels.
Conversely, JMeter is ideal for a single developer that needs to run load tests for a couple of hundred or even a thousand concurrent users on a website which makes JMeter (on it’s own) a great solution for small development shops. However, once a solution is needed for a distributed team or enterprise, where things like true scalability, ease of use, teamwork, commercial support and reporting capabilities are vital to the company's performance and survival, JMeter, alone, does not suffice.
Filling the Gap: 7 Enterprise Grade Features
Many enterprise teams recognize JMeter as an awesome performance testing “engine”, however, while JMeter and LoadRunner are comparable in terms of technical capabilities, we believe that the attention to scalability, features, and support levels, makes all the difference.
Accordingly, BlazeMeter Continuous Testing platform augments JMeter (as well as 20 other open source testing tools), successfully bridging the gap between the standalone, open source tool and a true enterprise-level solution.
1 - Commercial Support
Even though JMeter is an open source project, BlazeMeter provides enterprise-grade commercial support for JMeter, similar to that of RedHat’s support for Linux. This includes everything from a simple request to debug an existing JMeter script, to an onsite JMeter training for a group of 20 developers run by our performance testing professionals.
BlazeMeter provides both training and 24/7 support as well as professional services. BlazeMeter can write JMeter scripts for you from scratch, run them and analyze the results, (though at the end of the day we’d rather teach you to fish than fish for you).
2 - Cloud-Based SaaS & Installable Software
JMeter is an installable software tool, requiring users to own or rent a substantial number of servers to create large loads. BlazeMeter offers JMeter as a service, allowing customers to run their own existing scripts via the cloud, without building their own data center.
Customers are simply provided with their requested number of servers in real time, in a modern, cloud solution where the number of servers can increase or decrease purely in terms of necessity. In case the need arises to run a test behind a firewall, BlazeMeter provides on-premise agents that allow customers to run performance tests without relinquishing the advantages of installing software locally.
3 - Reporting & Collaboration
One of JMeter’s well known shortcomings is reporting capabilities. BlazeMeter understands the importance of sharing formatted and easy to read reports between team members, and has its own reporting engine, which outdoes JMeter's out-of-the-box reports by far.
Additionally, BlazeMeter supports team collaboration by providing all of the tests and reports through its web-based interface, as opposed to LoadRunner's software that can only be utilized when installed and run on a PC. BlazeMeter's web-based interface enables customers to test execution, run scripts or create loads from anywhere as well as view relevant reports.
4 - Browser based user recording and scriptless performance testing
BlazeMeter provides a Chrome plug-in, enabling users to record browser activity without even having JMeter installed. The extension records all of the HTTP/S requests that your browser makes, creates an Apache JMeter script, and automatically uploads it to BlazeMeter, where you can execute with a single click.
5 - “Webscale” Scalability
JMeter’s limited scaling capabilities are superseded by BlazeMeter’s own, additional scaling technology. By utilizing the cloud’s infrastructure, BlazeMeter is able to provision hundreds of test servers and manage the test distribution and data aggregation of large-scale load tests. As a result, you can run JMeter tests at any scale (Enterprises run tests with upwards of two million concurrent users with BlazeMeter).
6 - CI & APM Integration
Blazemeter’s integration with a range of APM and CI/CD tools extend your test data and support end-to-end visibility of server performance as well as end-user experience. The Jenkins, teamcity and bamboo plugins provide the ability to automatically run performance tests as part of the build process and see real performance data for every single build. The BlazeMeter labs team have also contributed many plugins for JMeter that extend it's capabilities. See more here.
7 - Full Continuous Testing Capabilities for the Enterprise
BlazeMeter isn't just a solution for performance testing. Now it is a full Continuous Testing platform, that offers a robust solution, including full 360 degree API Monitoring, Functional Testing Capabilities, and the recent addition of Mock Services integrated within the platform.
While JMeter, on its own, is definitely a viable alternative for some scenarios, mostly when dealing with smaller loads and teams, once you are looking for a solution which is truly suited for large teams and/or large loads, and/or automated performance testing, BlazeMeter gives you the best of both worlds. It is an open source solution with commercial support and an impressive feature set. BlazeMeter provides high-end enterprise-grade load testing at an affordable price.
Want to learn more in a 1-on-1 demo from one our performance engineers? Request a demo to discuss how ensure a smooth and easy transition from LoadRunner to JMeter.
For more information about JMeter check out:
Switch your LoadRunner scripts to open-source JMeter and Selenium in minutes, with our free online script converter. Learn more about how to switch, here.
This blog post was originally published in 2016, and updated in 2020 for accuracy.