Is JMeter a Viable Open Source Alternative to LoadRunner?
Performance testing for websites is something which, in the past, was mostly done with software tools installed on local machines. Take for example HP's LoadRunner, probably the most well known tool in the world of performance testing (it’s been around for a while).
LoadRunner is made up of two components: the IDE, in which scripts are created using a Windows-based application (Windows support only), and the load generators, which are supported on both Windows and Linux based.
For a long time HP was the only serious contender in the performance testing market, and they were able to charge an arm and a leg in licensing fees since at the time there weren’t many viable alternatives. HP still demands a very hefty price for LoadRunner. Some companies are paying six figures for the maintenance fee alone. Additionally, as with many outdated licensing-type software, transferring the LoadRunner license from one machine to another can be a nightmare.
Besides the cost of the software itself, running large scale performance tests with LoadRunner is no easy feat. Essentially, you have to create your own performance testing lab, including an adequate number of virtual user license purchases. Aside from proper capacity planning, HP has a very old school approach to scheduling performance tests, requiring advance coordination to secure performance testing time slots. While this approach may seem outdated, it has worked well for HP in the past, and many companies continue to do this today. Furthermore, Loadrunner uses a Windows based IDE, opposed to modern solutions which are either browser based or cross platform.
JMeter - Advanced Open Source Option
JMeter is an open source project that is very similar to LoadRunner as far as technical capabilities go. It essentially runs simulated, virtual users that make all of the requests that a real browser would make.
Many people were looking for an open source alternative for performance testing as a result of the high price of LoadRunner. Accordingly, JMeter was born as an Apache project, under the Apache umbrella, and has been fairly active over the past few years. Technically speaking, it is an application written in Java, meaning it can run on any machine, Windows, Linux or Mac. It is GUI based, so elements can be added via the GUI tool, such as http requests. While the interface may seem a bit overwhelming at first, in a way, JMeter is like the cockpit of a jetliner airplane. It is most likely that 4 or 5 dials and levers are actually needed to fly the plane, though supplemental equipment is there just in case. Most of the time, only 4 to 5 elements in JMeter are used, however the number of features displayed can be overwhelming.
Easily convert from LoadRunner to JMeter with this script converter, or get a 1-on-1 live demo to discuss how ensure a smooth and easy transition from LoadRunner to JMeter.
One of the cool things about JMeter is that because it is an open source project, it supports a plug-in architecture where functionalities can easily be added by writing a Java plug-in. One of the areas where JMeter differs from some of the other commercial performance testing tools is in its focus on defining load in terms of concurrent users, opposed to defining load in terms of requests per second. For that matter, API performance testing is usually done based on requests per second, which resulted in the creation of a third party plug-in to support detailed throughput shaping based on transactions/requests per second.
The JMeter Challenge
Like many open source, developer-oriented platforms, the real challenge with JMeter is that it was written with a focus on technical functionality and not on user experience. For example, in JMeter, a virtual user is called a ‘thread’, whereas other tools use the terms 'virtual user', 'simulated user', or even simply, 'user'. JMeter also makes use of the term 'sampler', which other tools refer to as a 'request'.
While the community has added a significant amount of functionality, ultimately, JMeter lacks a number of features and capabilities that are required for enterprise usage. Analogous to the Apache web server in the sense that while it works great as a technology, there are a lot of missing components in terms of scaling, server coordination, collaboration and access levels. Conversely, JMeter is ideal for a single developer that needs to run load tests for a couple hundred or even a thousand concurrent users on a website which makes JMeter (on it’s own) a great solution for small development shops. However, once a solution is needed for a distributed team or enterprise, where things like true scalability, ease of use, teamwork, commercial support and reporting capabilities are vital to the company's performance and survival, JMeter, alone, does not suffice.
Filling the Gap: 7 Enterprise Grade Features
BlazeMeter recognizes JMeter as an awesome performance testing “engine”, however, while JMeter and LoadRunner are comparable in terms of technical capabilities, we believe that the attention to scalability, features, and support levels, makes all the difference.
Accordingly, BlazeMeter augments JMeter, successfully bridging the gap between the standalone, open source tool and a true enterprise-level solution.
1 - Commercial Support
Even though JMeter is an open source project, BlazeMeter provides enterprise-grade commercial support for JMeter, similar to that of RedHat’s support for Linux. This includes everything from a simple request to debug an existing JMeter script, to an onsite JMeter training for a group of 20 developers run by our performance testing professionals.
BlazeMeter provides both training and 24/7 support as well as professional services. BlazeMeter can write JMeter scripts for you from scratch, run them and analyze the results, (though at the end of the day we’d rather teach you to fish than fish for you).
2 - Cloud-Based SaaS & Installable Software
JMeter is an installable software tool, requiring users to own or rent a substantial number of servers to create large loads. BlazeMeter offers JMeter as a service, allowing customers to run their own existing scripts via the cloud, without building their own data center.
Customers are simply provided with their requested number of servers in real time, in a modern, cloud solution where the number of servers can increase or decrease purely in terms of necessity. In case the need arises to run a test behind a firewall, BlazeMeter provides on-premises agents that allow customers to run performance tests without relinquishing the advantages of installing software locally.
3 - Reporting & Collaboration
One of JMeter’s well known shortcomings is reporting capabilities. BlazeMeter understands the importance of sharing formatted and easy to read analyses and reports between team members, and has its own reporting engine, which outdoes JMeter's out-of-the-box reports by far.
Additionally, BlazeMeter supports team collaboration by providing all of the tests and reports through its web-based interface, as opposed to LoadRunner's software that can only be utilized when installed and run on a PC. BlazeMeter's web-based interface enables customers to test execution, run scripts or create loads from anywhere as well as view relevant reports.
4 - Browser based user recording and scriptless performance testing
BlazeMeter provides a Chrome plug-in, enabling users to record browser activity without even having JMeter installed. The extension records all of the HTTP/S requests that your browser makes, creates an Apache JMeter script, and automatically uploads it to BlazeMeter, where you can execute with a single click.
BlazeMeter’s FollowMe technology goes one step further and provides instant performance testing without any advance preparations. Tests will run automatically as you just surf the web. You just click and go!
FollowMe works by triggering a virtual user group to follow your every move online - establishing the load of your test. Analytics are displayed in real-time so you can adapt your test accordingly as you go.
In addition, if you do have JMeter installed, you can convert your recording into a JMX file and use it on your local computer. Check it out on the Chrome Web Store.
5 - “Webscale” Scalability
JMeter’s limited scaling capabilities are superseded by BlazeMeter’s own, additional scaling technology. By utilizing the cloud’s infrastructure, BlazeMeter is able to provision hundreds of test servers and manage the test distribution and data aggregation of large-scale load tests. As a result, you can run JMeter tests at any scale (we’ve tested upwards of a million concurrent users).
6 - CI & APM Integration
Blazemeter’s integration with tools such as New Relic, AppDynamics, Dynatrace and AWS Cloud Watch extend your test data and support end-to-end visibility of server performance as well as end-user experience. For example, via New Relic integration, developers can easily monitor their application’s performance level while load testing. The Jenkins, teamcity and bamboo plugins provide the ability to automatically run performance tests as part of the build process and see real performance data for every single build.
7 - Mobile Load Test
BlazeMeter provides a series of handy options to simulate the work of users with various mobile devices, at any scale, from multiple geographic locations, without using numerous real mobile devices. BlazeMeter can be used to simulate a heavy load on a mobile application in order to test its strength or analyze its overall performance under different types of loads. The results are stored in your account and may be used as a comparison with future test results.
While JMeter, on its own, is definitely a viable alternative for some scenarios, mostly when dealing with smaller loads and teams, once you are looking for a solution which is truly suited for large teams and/or large loads, and/or automated performance testing, BlazeMeter gives you the best of both worlds. It is an open source solution with commercial support and an impressive feature set. BlazeMeter provides high-end enterprise-grade load testing at an affordable price.
Want to learn more in a 1-on-1 demo from one our performance engineers? Request a demo to discuss how ensure a smooth and easy transition from LoadRunner to JMeter.
For more information about JMeter check out:
Switch your LoadRunner scripts to open-source JMeter and Selenium in minutes, with our free online script converter. Learn more about how to switch, here.