Adrian Lopera-Valle, Ph.D., is a technical consultant with Scimitar Technical Consulting, Inc. Adrian's research experience allows him to identify and address clear objectives in complex projects, and to communicate efficiently about them.

Become a JMeter and Continuous Testing Pro

Start Learning
Slack

Test Your Website Performance NOW! |

arrowPlease enter a URL with http(s)
Nov 15 2021

Should You Load Test Your Application with Open Source Tools?

 

Deciding which tools to use is one of the most important steps in the performance testing stage of a development project. However, the almost endless number of options often makes choosing which tool to go with - difficult. If you are deliberating whether to test with open source testing tools, this blog may be the place to start.

In this post, we will look at the pros and cons of using open-source testing tools, followed by a comparison between four leading open-source performance testing tools. In addition, we will show you when going for a premium paid solution is best for you.

What are Open Source Testing Tools?

As developed from the ideas of Richard M. Stallman, open-source software is software built with source code that can be copied, inspected, and modified by anyone. With the accelerated growth of open-source software, developers are increasingly showing interest in using this type of software for performance testing. These open source testing tools are used to determine how an application will perform under simulated conditions.

In addition to not having any initial distribution cost, open-source tools for testing come with other benefits:

  • There are no licensing costs, as open-source software must be free for redistribution by definition.
  • The tools can be modified and extended by users because the application code is available for anyone to study, copy, and edit. This means the capabilities of open-source tools can evolve continually based on the community’s needs.
  • There are no exclusivity or permanence contracts, so you can move to other solutions at any time, no questions asked, giving you flexibility when scaling your applications or trying new solutions.
  • These tools have a wider reach because more types of users with diverse test requirements have easy access to them, making them well-known and understood in the community.
  • In the case of popular software, it is likely to still have technical support, regular tests, and updates, though this may not be the case for every solution.
  • A wide community is formed around managing, implementing, updating, and testing the software, meaning tool capabilities will develop based directly on the needs of users. For example, JMeter has a community of developers that will help you to address bugs and by answering common questions.

Keeping this in mind, open-source testing solutions do have some limitations or disadvantages in comparison with many commercial alternatives:

  • Testing is done on your servers, under ideal conditions, making testing unrealistic. This is not much of an issue with premium commercial services as they use external servers and machines to simulate tests.
  • You should maintain the codes and server infrastructure for testing yourself, as opposed to expecting this to be done by the developers of premium services.
  • While there may be a community of users, technical support in unforeseeable circumstances may be limited.
  • You are responsible for updating and modifying the code based on your application, which will cost you time, resources, and money.

If you are interested in open-source in the broader sense, these articles cover in more detail open-source software development and its use in business.

JMeter, Gatling, Locust, and k6 Testing Tools

With all the open-source performance testing tools available today, comparing their performance and choosing the best seems difficult and time-consuming. Luckily, we recently compared the four leading open-source solutions, JMeter, Gatling, Locust and k6, for load performance testing, so you do not have to. These four tools all feature well-developed communities and comprehensive solutions, making them well suited for use by a wide range of testers. 

Before looking at which one came out on top, let’s look at a list of features of these open-source testing tools so you can compare and choose based on your particular needs. After all, while some tools may offer better testing performance, they could potentially be unsuitable for your testing application depending on project specifics. For example, while the four tools analyzed are all compatible with any major operating system, using JMeter may be the best option if you do not want to use coding to create the test script and just want to take advantage of the system’s GUI.

 

Feature

JMeter

Gatling

Locust

k6

OS

Any

Any

Any

Any

Testing with GUI

Yes

Recorder only

No

No

Test Script Recorder

Yes: HTTP, Siebel, Mainframe, Citrix

Yes: HTTP

No

Yes: HTTP

Programming/Extension Language

Groovy and any other JSR223

Scala

Python

JavaScript

Load Report Format

Console, HTML, CSV, XML

Console, HTML

Console, HTML

CSV, JSON

Protocols

HTTP, FTP, JDBC, SOAP, LDAP, TCP, JMS, SMTP, POP3, IMAP

HTTP, MQTT, JMS

HTTP

HTTP, gRPC

System under test

With plugin

No

No

No

Clustered mode

Yes

No 

Yes

No

 

Based on the list of features alone, the table above makes JMeter appear ahead of the group, as it has the most extensive capabilities, including for languages and protocols. However, we should look at the results of performance tests before jumping to conclusions. 

A comparative test was done using the open source Taurus automation framework. The CPU, memory and network resource usage, as well as the number of connections of these different tools, were compared using the BlazeMeter Engine. To make the tests fair, 20 virtual users for 1 minute were simulated for each tool. Following this, the results for the number of requests, response time, bandwidth, and other network metrics were compared.

So what were the findings? The results from the testing for these open-source tools were ultimately up for debate. On one hand, the initial results clearly put k6 as a winner. However, there seemed to be inconsistencies between the number of requests, response time, and throughput, leaving an open question about a close second place between  JMeter and Locust. This shows that we need to consider more than a small set of key performance indicators (KPIs) when comparing tools and solutions. To learn more about how this comparison was done, check our recent post.

Will Premium Paid Tools Offer You a Better Solution?

As listed above, open-source testing tools may come with some disadvantages, which may make them inadequate for your particular application and project. That is where paid tools may help. Premium tools are generally best for you when:

  • Time is limited and you may need technical support, especially to fix unforeseen problems.
  • You need a simple, user-friendly interface, which makes using the tool less time consuming.
  • You need real-time reporting and comprehensive analytics.
  • Testing under realistic conditions, in third-party servers, is what you need.
  • The load (your site visits) and services (what you offer in your site) may change abruptly in the foreseeable future. If this may happen in your project, scaling quickly may be unviable with open-source tools.
  • Running large-scale tests in a cloud environment is required.

When it comes to choosing a premium paid tool for performance testing, the task is not necessarily easier. With all the different tools available, finding the best one for you can prove time-consuming. We advise you to choose a solution that fits your budget and that gives you the set of features and technical support you need.

Closing Remarks

At BlazeMeter, we support the idea of open-source tools. We think that in some cases, pairing open-source and commercial tools may be the best approach to automatized performance testing. A perfect example of this is Taurus, an open-source test automation tool that allows you to run open source load testing scripts with BlazeMeter

It is clear that open-source software comes with many benefits, this explains their growing popularity. However, choosing to use them over premium paid solutions will depend on your particular resources and needs. We encourage you to do your research and balance the advantages and disadvantages of both types of tools. Following and subscribing to our blog are great ways to stay up to date on the latest developments on this fascinating topic.

 

   
arrowPlease enter a URL with http(s)

Interested in writing for our Blog?Send us a pitch!