A few years ago, when I was working as a SME Performance Testing Engineer in one of the largest banks in the world, we regularly faced challenges when executing our LoadRunner scripts. After we got past the collaboration challenges that often caused us to lose changes we’d made in the code, we had to deal with execution.
Test executions were made through LoadRunner’s Performance Center. We had about 450 Windows servers as load generators, 25 Controllers and 20 Analyzers, but these were all shared among multiple teams who each had to run their own tests. And there were a lot of tests to run: we scheduled the LoadRunner servers for 10K up to 15K Vusers, for running Load, Stress, Endurance and Resilience tests. Luckily, my team had top priority, but some teams had to wait full weeks before they were able to run their load tests. This delayed their ability to deliver and to send their code to production.
About 20 people used to work in the Performance Center support team. They would help us with Servers’ hang, and to recover results or sessions. They would also install, configure and provide maintenance to the Load Generators, Controllers, Analysis and Monitor servers. It was a complex infrastructure to maintain and we depended on them completely.
In the analysis phase, the real challenge was the interpretation of the results. LoadRunner provides a lot of information to be analyzed, like results, logs, counters and monitor´s data. This is great when you are very technical, but a challenge if you aren’t but still want to understand the results. When working in a team with different expertise levels, this can cause rifes and make more technical team members feel unsupported when they want to dive deeper into the results.
It’s been awhile since I’ve had to face these challenges. This is mostly because I am currently working as a freelancer and I changed the tool I use to Apache JMeter™. In most of the projects I execute, I decide on the approach, create the test plan, script, execute and analyze. The result is happy customers. In JMeter, the task is made easier. Here are some of the advantages of working with JMeter:
Scripting is Easier
The JMeter GUI is basically drag and drop object oriented. This helps us save mountains of time when scripting, even with complex scripts. In JMeter, scripting is optional and even when it’s done it’s more intuitive, and it all takes place in one screen.
JMeter’s easy scripting also enables smaller teams to work on each script. In LoadRunner, the complex scripting required dozens of engineers to remain in sync, an impossible mission given that each had his or her own philosophy of correlation rules, scripting, programming and knowledge of the tool. This would result in losing changes made in the code. In JMeter, smaller teams are enough, making collaboration easier.
Execution is Easier
You can manually setup your own load generators for small loads. You just need a couple of minutes to install Java, download JMeter and upload the JMeter script file. In BlazeMeter, which you can use for larger loads, you just need to upload your JMX file, select duration and number of iterations, monitors and location.
Analysis is Easier
You can either add graphs or listeners to view test results, or merge the results in JTL files. Results are easy to understand for less experienced engineers, and also allow in-depth analysis for testers with more know-how. In BlazeMeter, the SaaS dashboard shows response times, errors, bandwidth and hits per second, in real time if you want. In a separate tab you can see the load generators counters like CPU, Mem, IO and Network. If external monitors were set like StackDriver, CloudWatch or New Relic, graphs would be available for monitoring in BlazeMeter. All the results can be collaborated through the cloud, the JTL and logs are available for downloading and there is a high-level version for upper-level management.
I feel comfortable when working with JMeter, because JMeter allows me to do what I need in a simple, portable and compatible way. Most performance projects take me around 40 to 50 hours to complete, either for Web or Mobile. In LoadRunner, a similar project might take 200-300 hours, 5 or 6 times more, and would usually require 2-3 team members, instead of one.
Also, JMeter is free to use, which is a huge advantage. Finally, configuration with CI/CD tools like Jenkins for test automation and shifting left is easier.
If you are a Performance Testing Engineer and you are committed to one tool, you should consider looking at new options. Choose the tool that best suits your specific needs, be creative and stay curious.
LoadRunner users, convert your scripts to open-source for free with our LoadRunner Script Converter.
To check out BlazeMeter, which enhances JMeter, just put your URL or JMX file in the box below, and your test will start in minutes.
You might also find these useful:
Interested in writing for our Blog? Send us a pitch!