Selenium vs. JMeter: Which One Should You Choose, and When?

Testing tools like Apache JMeter™, Selenium, and others are important for your DevOps cycle. Yet each has a uniquely different scope. In this article, we will Selenium vs. JMeter and determine which of them is a key tool for which scenario. Though you may expect a sort of conflict between these tools, I will rather analyze how these tools can work together.

What's the Difference Between JMeter and Selenium?

When it comes to comparing JMeter vs. Selenium, there's a lot to keep in mind. While both are used to test web applications, the main difference between JMeter and Selenium is that JMeter is best for load and stress testing while Selenium supports web test automation and cross-browser testing. 

Development teams are increasing their adoption of testing and test automation tools. Choosing the right testing tool will influence the entire development project, so it’s important to make a fully informed decision, considering factors such as the following:

• Which tool fits my project goal better?
• Which tool requires less training for my team?
• Which tool will have less of an impact on my organization’s policies?

JMeter vs. Selenium: Definitions

Both JMeter and Selenium simplify the testing process. They are both:

• created for web application testing
• relatively easily to learn
• well documented
• developed in Java
• open source with the Apache License

JMeter Definition

JMeter is an open source load and functional testing tool for various network protocols, run as a Java desktop application with a graphical interface. It supports protocols including JDBC database connections, FTP, LDAP, web services, websocket, JMS, HTTP, RTE and generic TCP connections. JMeter is developed as a pluggable and extensible tool suite with JMeter plugins, which enable advanced sampling and validation.

JMeter’s functionalities are organized in components. Each component belongs to a specific category that has a role and is used hierarchically in the test script. JMeter scripts are developed in the form of a components tree in the JMeter GUI (you can also use non-GUI mode if you prefer scripting).

As you can see in the diagram below, the same type of element can have different scopes and can appear in different places throughout the tree.

JMeter components fit into the JMeter GUI like this:

Selenium Definition

Selenium is an open source functional testing framework that automates browsers. It provides multiple facilities for web application testing across different browsers and platforms. Selenium is not just a single tool, but rather a suite of software. 

The two most important parts of this suite are the Selenium WebDriver and the Selenium IDE. Webriver is a set of language specific APIs for browser automation. The IDE is a Firefox/Chrome extension that provides a “record & replay” feature. BlazeMeter even provides it’s own free IDE, BlazeMeter Chrome Extension, which can likewise record and replay scripts, then automatically execute them through the BlazeMeter cloud.

Selenium scripts are mainly developed as code in order to obtain maximum advantage of WebDriver’s API. This API has bindings on various programming languages. Each web browser & operating system couple require a different driver implementation. WebDriver is an abstraction layer that requires correct driver installation on the execution machine.

The user can write in any of the supported languages to Selenium WebDriver’s API, which automates any of the supported drivers.

Note: BlazeMeter provides its own extremely robust and easy-to-use GUI Functional Testing feature instead of relying on the WebDriver.  This feature enables you to upload your Selenium scripts directly to BlazeMeter without needing to provide a specific driver.  Simply write your script (or record it with the Chrome Extension), then BlazeMeter will handle the rest!

JMeter vs. Selenium: Key Features

By Test Case

Now let’s see when we should use each tool. Consider the following hypothetical web application architecture:

• User Desk - a generic personal computer or mobile device
• Web Browser
• Front End - the layer seen in the world wide web, which is contacted by the browser
• Back End - the layer that supplies the services that execute the business vision of the application
• RDBMS - a data layer that returns data in SQL

(This schema is an oversimplification that takes into account technologies, but not implementations.)

This application needs testing and validation. Let’s see which tool we should use in each case.

Web Browser Testing

Web browser testing validates how an online web application is presented to a user, and is normally performed on more than one type of browser (referred to as “cross browser testing”).

Web browser testing verifies:

• How web pages are rendered
• How the user session is handled
• The quality of user experience

Selenium is the de facto standard for these types of tests.

Suppose that we have following test requirement:

1. Open the browser at https://web.whatsapp.com/
2. Verify the scan code changes every 30 seconds

Selenium is the smartest way to test this, because we need to open the web page and monitor the scan code tag in order to ensure it’s renewed. 

The code should fetch a tag with the scan code and monitor it via a Selenium object called a WebElement. This object periodically fetches a new scan code every 30 seconds and compares it with the previous one saved.

Here’s an example of Selenium code (written in Java), that performs this:

@Test
public void testScanCodeRenew() throws InterruptedException {
	
	driver.get("https://web.whatsapp.com/");
	
	By xpath = By.xpath("//div[contains(@class,'_2EZ_m')]");
	WebElement scanCode = new WebDriverWait(driver, 10)
		.until(ExpectedConditions.presenceOfElementLocated(xpath));
	
	Supplier<String> getScanCode = 
           () -> scanCode.getAttribute("data-ref");
	
	String oldCode = getScanCode.get();
			
	for(int index = 0 ; index < 3 ; index++) {
		
		Thread.sleep(30_000);
		
		String newCode = getScanCode.get();
		
		Assert.assertNotEquals(oldCode, newCode);
		
		oldCode = newCode;
	}
}

Front End: Testing Custom Communication Channels

JMeter shines best when involved in testing machine-to-machine interfaces. Consider, for example, WebSocket web applications. These applications can improve browser functionality by adding an Enhanced Application Client (typically in Javascript) to setup a communication channel based on custom protocol.

Even though a WebSocket application may be encapsulated into a web session and thus affect the browser, neither the user nor Selenium will realize it. We will therefore rely on JMeter for testing WebSockets.

Suppose that we have the following test requirements:

1. Contact the WebSocket endpoint at wss://echo.websocket.org/.
2. Send/receive the text request/response of unchanged messages.

The JMeter script will use one of JMeter’s elements: the Peter Doornbosch WebSocket Sampler. The resulting script performs the following steps:

• establish a secure WebSocket communications channel
• maintain a constant rate of requests and responses
• dynamically format request messages
• verify that response messages match corresponding requests

Though the example above describes a very basic protocol in which messages are sent back back without any alterations, it is nonetheless a good starting point toward a more complex scenario. Complexity could be added by:

• modifying the “Format Message” step to customize message generation by implementing some Groovy code
• modifying the “Message Assertion” and “Header Assertion” steps to customize message verification

Front End: Testing User Experience

Modern web applications put a lot of emphasis on producing a user interface that provides a good user experience. The interface has to be visually appealing and clear enough to convey the message in just a few seconds. For example, an interface may allow the user to interact with draggable components that can be overlapped to obtain different groupings (e.g. a cart for e-commerce site).

Suppose we have the following test requirement:

1. Open a browser at https://the-internet.herokuapp.com/drag_and_drop
2. Drag and drop one of the two displayed elements over the other.

Selenium is the best choice for this scenario because the feature we’re testing pertains to how components are visually rendered in the browser. JMeter cannot help us here because it is unable to monitor actions that render visual components.

The code will verify the position of draggable components before and after the “drag and drop” operation. The test will pass if the component positions in the resulting data structure switched, when the test ended. For the “drag & drop” actions, you can use Selenium’s capability to execute Javascript code directly into a WebDriver object, by using WebElement objects as arguments.

@Test
public void testDragAndDropWithCheck() throws InterruptedException {

	driver.get("https://the-internet.herokuapp.com/drag_and_drop");

	By css = By.cssSelector("div[id^=\"column-\"]");
	WebDriverWait wait = new WebDriverWait(driver, 10);
	
	Supplier<List<WebElement>> fetchComponents = () -> wait
			.until(ExpectedConditions.presenceOfAllElementsLocatedBy(css));
	/**
	 * Starting check for element position
	 */
	List<WebElement> startingCheck = fetchComponents.get();

	Assert.assertEquals("Starting - Draggable number does not match!", 2, startingCheck.size());
	Assert.assertEquals("Starting - A position does not match!", "A", startingCheck.get(0).getText());
	Assert.assertEquals("Starting - B position does not match!", "B", startingCheck.get(1).getText());

	int index = ThreadLocalRandom.current().nextInt(startingCheck.size());
	WebElement from = startingCheck.get(index);
	WebElement to = startingCheck.get(1 - index);

	JavascriptExecutor jse = (JavascriptExecutor) driver;

	jse.executeScript(
        dndScript() + "simulateDragAndDrop(arguments[0], arguments[1])",
        from,
        to);
	/**
	 * Ending check for element position
	 */
	List<WebElement> endingCheck = fetchComponents.get();

	Assert.assertEquals("Ending - Draggable number does not match!", 2, endingCheck.size());
	Assert.assertEquals("Ending - A position does not match!", "A", endingCheck.get(1).getText());
	Assert.assertEquals("Ending - B position does not match!", "B", endingCheck.get(0).getText());
}

Back End Testing: Validating Services and Execute the Business Logic

Back end systems export business logic via web services. The same back end can perform this operation to various front ends (such as web and mobile devices). This layer is typically an example of a machine-to-machine interface, therefore JMeter is perfect for this type of testing.

Suppose that we have the following test requirement:

1. Send a JSON payload to https://reqres.in/api/users to create a new user
2. Verify that the response payload now contains a correctly created new user

The JMeter script will simulate a user querying with a REST message. The script uses a standard HTTP Sampler to send a POST message with a JSON payload:

{
    "name": "new_user",
    "job": "user_role"
}

The JMeter script can use JMeter’s template feature with the following steps:

• Replace the SOAP request with REST, so the XML payload is replaced with a JSON payload
• Use a JSON Assertion to verify the response status, instead of an XPath assertion, to validate the JSON payload fields

A tip when developing REST test cases: when defining assertion steps without a precise value to assert, try using a smart regular expression to match a string instead. I used the following regex, which you too can use in your scripts:

• an “id" attribute with an integer value: the regex is [0-9]+
• a “createAt” attribute with a date field formatted as ISO 8601: the regex is [0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}\.[0-9]{3}Z

Now that we’ve seen in which cases you should use each tool, let’s compare JMeter and Selenium via two parameters:

• Technological knowledge prerequisites
• The ability to create testing suites

Technological Knowledge Prerequisites

The number of developers working on a test automation project can change, depending on the activity level of the project; therefore, it’s important to learn how to write and maintain scripts in each of these tools. (Of course, the more we automate, the less training is needed.)

Selenium Required Knowledge

Selenium-based testing requires knowledge of the WebDriver API. Moreover, it’s necessary to be proficient in the chosen developing language (e.g. Java, Python, C#, etc), developing a toolchain (e.g. Maven, Ant, MS Visual Studio, etc) and operating a testing framework (JUnit, TestNG, unittest, NUnit, etc). It’s possible to have a “Test Library” module for simplifying the procedures, but that also requires training on. Selenium is also suitable for BDD.

JMeter Required Knowledge

JMeter-based testing requires knowledge of the JMeter component organization, or at least of ten basic components. New developers must be trained on environment setup and the required plugins according to their test project goal; however, because JMeter has a GUI and doesn’t require scripting, it’s easier to run and learn.

Creating Test Suites

Individual test cases are not enough; rather, developers and teams should have test suites that aggregate multiple test scripts in order to ensure an adequate range of testing scenarios are covered. This is also helpful for test automation.

Automated test suites:

• Aggregate multiple test scripts
• Simplify test script selection by using custom languages (e.g. XML, YAML, etc)
• Use logic to execute tests in a certain order
• Use logic to conditionally execute (or not) test scripts
• Can execute a test script using custom parameter values
• Produce a human readable report for test execution.

Neither JMeter nor Selenium include built-in test aggregation functionality, but there are a number of solutions:

The Selenium test suite can be organized in a TestNG XML suite. As the name suggests, it's an XML file that describes an entire Selenium test suite/scenario to be executed. The result file can be shared among test machines, granting they have the same test coverage logic. To obtain this it's necessary to use TestNG as a testing framework and a Selenium test as a TestNG test. This is an easy operation but it can become expensive if there is a large number of test cases to be migrated/adapted.

The JMeter testing suite is a group of JMX scripts, though a better way to aggregate these scripts is to use Taurus, which can manage them via a single YAML configuration file without any need to refactor the scripts. Simply write your scripts as usual with JMeter, then add a YAML file to aggregate them. Taurus makes it possible to customize parameter values passed to each JMX script in the suite. 

BlazeMeter makes this even easier. Its Multi-Test feature allows you to aggregate and run a group of test scripts through a simple and intuitive web GUI. You can even run JMeter and Selenium tests simultaneously by leveraging BlazeMeter’s End User Experience Monitoring.

FAQ: Selenium vs. JMeter

Can Selenium Do Everything That JMeter Does?

No. Selenium can be used for some performance testing (e.g. limited bandwidth testing), but it cannot be used to load test with the level accuracy JMeter can. For example, Selenium cannot generate a controlled number of requests over time because it does not have an equivalent of JMeter’s Constant Throughput Timer. Moreover, targeting a specific requests-per-second rate in JMeter requires less resources from the execution machine, because Selenium tries to instantiate many web browser processes when doing so.

BlazeMeter’s knowledge base article answering the question Can I use Selenium for Load Testing? goes into a little more detail was to why Selenium is severely limited as a load testing tool.

Can JMeter Do Everything That Selenium Does?

No. JMeter’s HTTP(S) Test Script Recorder can record detailed web sessions originated by web browser applications, then replay the web session as the server sees them. Replays do not reproduce client side elaboration (e.g. JavaScript), however, nor do they render web pages. So if our test is focused on how a web page is presented to a user, Selenium is the better choice.

That said, if you like the idea of recording and replaying a script, BlazeMeter’s Chrome Extension will let you record, reply, then export both JMeter and Selenium scripts.

Can JMeter and Selenium Collaborate?

Yes. There are various situations where both tools can sync on the same testing application. We will describe some of them briefly.

If you’re simply testing on your local machine, JMeter’s WebDriver Sampler plugin lets you add a Selenium test case directly into a JMeter script. It has some restrictive resource limitations, but if there are only a few iterations, it has good practical trade offs including easy configuration.

Data-driven testing is another approach which involves running JMeter on multiple Selenium WebDriver instances. This method is also heavy on resource consumption, but its doable if you have the hardware to support it. Of course, it's not so easy, and it requires development to obtain a testing layer between JMeter and Selenium that can handle multiple instances with concurrent usage.

The WebDriver Sampler plugin and data-driven testing methods are indeed major, unforgiving resource consumers, so if you want to run more iterations or execute tests on the cloud, neither is a particularly viable option. Instead, BlazeMeter’s End User Experience Monitoring, mentioned earlier, provides a much easier alternative, letting you run your JMeter script at higher loads while a Selenium script simultaneously executes and assess what the user’s in-browser experience is like during the run.

It’s also possible to convert a Selenium test into a JMeter test using the recorders previously discussed (the HTTP(S) Test Script Recorder and BlazeMeter’s Chrome Extension). The Chrome Extension even lets you create a synced pair of Selenium and JMeter scripts which are ideal for running in parallel via the above-mentioned End User Experience Monitoring.

Use JMeter, Selenium, & Other Open-Source Tools with BlazeMeter

Regardless of your role in test automation and development, both JMeter and Selenium are  important and useful tools to be used during test project planning and execution. I hope this blog post helps you choose the right tool for you and your team, according to your project goals. You can also integrate and sync these tools with open source frameworks like Taurus. So start testing now!

To get the free recorder that records and creates JMeter and Selenium scripts, click here.

To learn about Taurus, click here.

START TESTING NOW

Related Resources

• What Is GUI Testing?
• How to Automate Selenium UI Tests
• 6 Easy Steps to Testing Your Chrome Extension With Selenium