IoT Testing Using the MQTT Protocol, JMeter and BlazeMeter
Disclaimer: While it is possible to test IoT based on the MQTT protocol using JMeter, in most cases this solution would not be sufficient. This article does not intend to promote using JMeter for IoT testing, but to show the capabilities of JMeter and discuss the protocol.
An appropriate IoT testing solution for each scenario is something that needs to be determined per case—contact our sales team to discuss your specific needs. We plan on expanding informational content in this area in order to make the available resources for IoT testing attainable.
In the first blog article in this series about IoT testing, I introduced a high-level overview of IoT and the MQTT protocol was introduced. The purpose of that article was to discuss the technologies used in the IoT world. Particularly, the use of a Broker and the Publish-Subscribe pattern, aka the pub-sub architecture. This blog post will be a hands-on tutorial of how to set up a Broker package and run a performance test on it, with JMeter and BlazeMeter.
There are many different Broker packages available, but for this article we’ll use the HiveMQ broker.
There will be four parts to these hands-on instructions:
- Setting up the cluster: We will use the HiveMQ broker.
- Testing the cluster with the Client CLI: In order to test that the broker actually works, HiveMQ has a client CLI console shell for testing the broker. We will use it for setting up a connection, publishing, subscribing, and disconnecting from the broker.
- Writing JMeter scripts: Downloading and using the JMeter MQTT plugin to test the cluster. We will see how the plugin compares with the HiveMQ Client CLI.
- Running a BlazeMeter test based on the JMX script.
Let’s get started.
Part 1: Installing HiveMQ
If you’re not familiar with HiveMQ then you can get more information from their website here or here. Although HiveMQ is an enterprise-level cloud platform for hosting an MQTT broker, it is relatively simple enough to use for training and learning purposes.
To sign-up to use HiveMQ, go to this page. You have two options, cloud access or download locally. Choose the free cloud option.
Conveniently, if you have a GitHub user, you could sign-up using it or alternatively create a new user.
The other option is to enter an email address. When you type your email, you may be surprised to see your picture added in the login script. Hmm.. I wonder where they got that from!
Login with email
You will be given a default cluster. HiveMQ "clusters" are one or more brokers serving clients. It is a means of distributing the workload seamlessly. So if one broker is overloaded, the system adds more brokers to form a cluster.
The cluster you will receive will have a URL that is made up of a GUID appended to ".s1.eu.hivemq.cloud". The default port is 8883. Next, click on the “Manage Cluster” button.
In the Cluster Details screen, you have three views: Overview, Access Management, and Getting Started. In the Overview section, you can see some general information on the capacity of the cluster.
Go to the Access Management tab. By default, your username will be listed as admin. But you’ll need to add another user for the client application. Add the new user and supply a name and password.
In the Getting Started tab, you’ll notice that there will be a check by "Setup credentials for your IoT Devices." That just means that you added the credentials for a client user in the Access Management tab successfully. Notice too that in the next section, "Connect you first MQTT clients," has several tools to choose from. We’ll be using the MQTT CLI, but more on that in the next section. If you do click on the MQTT CLI icon, it will take you to a tutorial on using the tool.
Part 2: The HiveMQ Client
MQTT CLI provides a command line interface (CLI) to interact with an MQTT broker. The new tool provides two key features: 1) a SHELL mode that allows you to start multiple MQTT clients 2) support for both MQTT pub and sub commands so you have one command line for all the key MQTT operations.
HiveMQ has a client component (shell) for testing the broker using publish and subscribe commands. You can find instructions on how to download the client from the Getting Started page mentioned above. To use the client, you need to download it (as a ZIP file) and extract the contents to a directory of your choosing.
There are two ways to run the client application: 1) using the executable and 2) using the shell. We’ll just take a look at the executable methods first.
1. Open a console window and go to the directory where you extracted the client (you should see a mqtt-cli.exe file and a mqtt-cli-shell.cmd file).
2. Copy and run the Subscriber command from the instructions. Notice that there is an error in the command and it should be mqtt-cli and not mqtt.
3. Open another console window and copy and paste the Publish command (also change mqtt to mqtt-cli here too).
4. If you get a "Hello" message in the Subscriber window, then you're all set.
The second method is to use the MQTT CLI Shell. To do this open a shell by double-clicking on mqtt-cli-shell.cmd in the Windows file explorer.
1. Type in the Connect command, in the form of:
- [Note] con -h <cluster address> -p <port> -s i- <client name> -u <client user> -pw <client password>
- cluster address is the long server address ending s1.eu.hivemq.cloud
- port is 8883 for ssl and 1883 for TCP
- the -s means you are using SSL
- the -i is the name you are giving this session. We will be using two sessions, so make sure the names are different.
- the username and password are the ones you added to the authorized users above
2. Next, type in: sub -s -t topic1 This will be the subscriber window to be used for receiving the messages
3. Click on mqtt-cli-shell.cmd again like you did above. This will open a new window. You can change the background color if you wish.
4. Enter the Connection command like before but change the identifier of the session!
5. Now type:
- pub -q 1 -t topic1 -m hello
6. Notice that the Subscriber window will get the message.
The -q attribute is for QoS (Quality of Service) which will be discussed below.
See here the screen shot of the two sessions with Publisher and Subscriber.
Now that we’ve seen how to connect to a broker, register a subscriber, and publish a message, we will test them using JMeter.
Part 3: Testing with the JMeter MQTT Plugin
In order to simulate an IoT test using the MQTT protocol, we’re going to create two JMeter scripts: one for the Publisher and one for the Subscriber. Basically, it will work similarly to the example above with two client windows, but in this case there will be two testing scripts.
- Download and install the MQTT Plugin
- Download the MQTT protocol plugin from here.
- Navigate to the downloads/v2.0.2 folder.
- Download the mqtt-xmeter-2.0.2-jar-with-dependencies.jar file
- Save it under your JMeter home directory in $JMETER_HOME/lib/ext folder.
- Finally, add the plugin to the JMeter plugin Manager. (In the JMeter menu, go to the Options menu and chose Plugins Manager)
JMeter Plugin Manager
Now we’re all set, so let’s start with the Publisher script.
The Publisher script handles publishing messages to the broker topic. Take a look at the following diagram of the script structure. We first declare some variables for the connection to the broker. Next we’ll have a thread group for making the requests. To do that we will connect to the broker one time and then send the request. Finally, we’ll disconnect from the broker.
JMeter Publisher script
Let’s look at each step separately. First is the User Defined Variables in which we specify the host address and port just like we did above. These are the HiveMQ server credentials.
For the thread group, we’ll keep it simple with one thread and 10 iterations.
Publisher Thread Group
Next we’ll add a Runtime Controller. The Runtime Controller controls the execution of its samplers/requests for the given time. We’ll have the test run for 60 seconds.
Publisher Runtime Controller
Now we’ll use the MQTT Connect element that was installed from the plugin. Here we specify the server and port. We’ll aso be using SSL that requires giving authentication with a username and password. We will also provide a client id. In this example, the id is pub_client_ but it can be any unique name. The rest of the fields will be default values.
Publisher MQTT Connector
Now that we are connected to the broker, we can publish messages to it. In the MQTT Pub Sampler, also a new component in the plugin package, there is a payload section for specifying the message. We’re going to send a simple string message to the topic "topic1". The Pub options field is the Quality of Service. This is something specific to the Pub-Sub pattern.
MQTT supports three service levels qualities:
- QoS = 0 - means one delivery at most. The message is delivered according to the capabilities of the underlying network. No response is sent by the receiver and no retry is performed by the sender. The receiver gets the message either once or not at all.
- QoS = 1 - means one delivery at least. This quality of service ensures that the message arrives at the receiver at least once, but there’s a probability of duplicating messages on the receiver’s side. If the publisher has not received the acknowledgment of delivery from the message broker, it sends the message again. After the duplicated message is received by the broker, the latter sends it again to all subscribers.
- QoS = 2 - means one delivery exactly. This is the highest quality of service. It is used when neither loss nor duplication of messages are acceptable.
Now let’s look at the Subscriber. As you can see, the Subscriber script looks a lot like the Publisher script. There is a section for the UDV which is exactly as for the Publisher. Then there is a thread group and a once-only connection.
The main difference is in the Sub Sampler (which is a new control in the plugin package). Here you need to give the Topic name which is "topic1" as was for the publisher. Also, you need to specify how often to check for a message. In this example, we’re checking every second.
That’s it for the JMeter part of the project. You can run them in JMeter. Or, stay with us and see how to upload the scripts to BlazeMeter for scaling their performance testing.
Part 4: Creating a BlazeMeter Performance Test
In BlazeMeter we are going to create a multi-test, which is a test composed of more than one script. First, upload the JMX scripts we created to a shared folder.
[Important] You must upload the plugin jar file too, otherwise the scripts won’t work!
MQTT Plugin JAR
Next, create two performance tests with the scripts that were uploaded. See for example the Subscriber test. You can set the Load Configuration or use the ones set in JMeter.
Finally, create a multi-test using the two tests previously configured.
Run the test, and view the results.
This has been a hands-on introduction to testing using the Pub-Sub pattern with a broker and the MQTT protocol in the world of IoT. JMeter can be used for testing devices of various types. In the example above, we used HiveMQ as the broker, but the same goes for Apache Mosquitto and other brokers as well.
In the IoT world, there are many possible types of systems, and using JMeter as the scripting mechanism may not be the most appropriate solution for simulating the devices. However, BlazeMeter can be used for performance testing or load testing in various environments in the IoT world because if its cloud capabilities.
To get started with BlazeMeter, start testing now.