API Performance Testing Tools: JMeter, Taurus, and BlazeMeter
May 4, 2022

API Performance Testing Tools: JMeter, Taurus, and BlazeMeter

Performance Testing
API Testing

An API (Application Programming Interface) is a software intermediary that allows applications to access data and features of other applications or servers. APIs are usually consumed by multiple systems, which makes them an important testing target. API performance testing tools will not only detect failures on the API itself, but also on other, third-party components. 

Let's talk about APIs from a performance testing perspective. Imagine you have a web app that consumes an API. Once your web application is publicly available, multiple clients will be hitting the same service at the same time. That being the case, you would definitely want to execute performance tests to be sure your servers are accessible to all users, even under a heavy load, and that there are no API delays.

But how can you do it? Many engineers are not sure how to run performance testing at the API level. In this article, I am going to walk you through different ways for running API performance testing tools using three of the most useful performance testing tools: JMeter, Taurus, and BlazeMeter

We will be using a public API called Deck of Cards in all of the examples. You can find the documentation here.

Table of Contents:

API Performance Testing Tool: JMeter 

JMeter is an open-source tool used for performance testing. It can be used on any platform that supports Java, which makes it a good option for automating and executing API performance tests.

Let’s see how to create an API test:

1. Open JMeter

2. Right-click on Test Plan - Add - Threads (Users) - Thread Group

JMeter load testing

3. Now, let’s add a Transaction Controller to identify the step of the test. For that, right-click on the Thread Group - Add - Logic Controller - Transaction Controller

JMeter load testing

4. Click on the Transaction Controller and change the name to “Shuffle the cards”

JMeter load testing

5. The first step of our flow is to shuffle the cards. For doing that, we are going to add an HTTP Request. Most of the API are typically built for HTTP, so we can use this sampler since there is not a specific one only for API requests. Right-click on Homepage - Add - Sampler - HTTP Request

JMeter load testing

6. In the following image, you can see an HTTP Request sampler with the API request for shuffling cards.

JMeter load testing

7. As a next step, we want to avoid false negatives or positives in our test, and be sure we’re on the page we’re supposed to be. To do that, we will add a Response Assertion. Right-click on the HTTP Request - Add - Assertions- Response Assertion

JMeter load testing

If we look at the response in the documentation of the API, we should get something like this:

API performance testing

8. I’ll be adding ‘ “success”: true’ as a Response Assertion

API performance testing

The second step of our flow is to draw a card. This would be a GET request that will contain the deck_id  that comes in the response to the previous request. 

10. To obtain that value, we should use a Regular Expression Extractor. Right-click on the request of Shuffle the cards - Add - Post Processor - Regular Expression Extractor

JMeter load testing

11. As we saw in step 7, the response to the request is:

{"success":true,"deck_id":"3p40paa87x90","shuffled":true,"remaining":52}

 

So if we want to extract the value of deck_id, we may add a regular expression like this one: "deck_id":\s"(.*)",

12. Going back to JMeter, we will complete the fields as follows:

JMeter load testing
  • Name of created variable: deck_id 
  • Regular expression: "deck_id": "(.*)",
  • Template: $1$
  • Match No.: 1
  • Default Value: NOT_FOUND

The template is the capturing group number. In this case, we want to match group 1 and the default value is what JMeter is going to display if the regular expression does not match.

13. Let’s add a Transaction Controller called “Draw a card”. As we showed you in steps 3 and 4, right-click on the Thread Group - Add - Logic Controller - Transaction Controller. Then change the name to “Draw a card”.

JMeter load testing

14. Right-click on “Draw a card” - Add - Sampler - HTTP Request

In the following image, you can see the HTTP Request Sampler

JMeter load testing

15. The protocol and Server Name is going to be the same as in step 6. Add “/api/deck/${deck_id}/draw/?count=2” to the path where ${deck_id} is the call to the deck_id variable that stores the value we extracted on step 9.
As we did before, for each request we will be adding a Response Assertion

16. Right-click on the request - Add - Assertions - Response Assertion. Add ´”deck_id”: “${deck_id}”´ as an assertion to check if we are getting the right response.

JMeter load testing

17. To visualize the results of executing our script when scripting or debugging, we will add a View Results Tree. Right-click on the Thread Group - Add - Listener - View Results Tree.

JMeter load testing

18. We usually also add an HTTP Cookie Manager and HTTP Cache Manager. You can find those by doing right-click on the Test Plan - Add - Config Elements

19. To configure the load of the test, we should go to the Thread Group element. We configured this example to have 5 threads and a ramp-up period of 10s.

JMeter load testing

The thread group control also allows you to set how many workflow iterations each virtual user will do (Loop Count). We will leave it in 1 so that each virtual user shuffles and draws a card once.

Our script will look something like this:

JMeter load testing

To run the test, click on the “play” button and see the results on the View Results Tree.

JMeter load testing
JMeter load testing

Remember to remove or disable the View Results Tree when running high load tests, in order to improve the performance of the load generator.

API Performance Testing Tool: Taurus

Taurus is an open-source tool that automates and runs performance tests. Unlike JMeter, Taurus doesn’t have a GUI, but it enables simplified running of other tests from tools like Gatling, Locust, Selenium, and others.

In Taurus, you can write your script on a YAML File and run it on a terminal. YAML is an easy-to-understand language (which leads to a lower learning curve). This makes it easy to maintain using version control tools such as GitHub. It is also an advantage when adding your tests to continuous integration tools like Jenkins

Here you can read the complete guide to API testing with Taurus. Now let’s see how to test our Deck of Cards API.

Every YAML file starts with an “execution” section. This is where you define the parameters corresponding to the load and the execution engine to be used (which is JMeter by default).

execution:
  - executor: jmeter
    concurrency: 5
    ramp-up: 10s
    hold-for: 60s
    scenario: deck of cards  

 

  • scenario: it is the name of the scenario that is going to be executed. It must be defined in the “scenarios” section (which we will see next). It is important to keep in mind that this field does not have a default value, therefore we must specify it. If not, Taurus will not know which scenario to run.
  • concurrency: the number of target concurrent virtual users
  • ramp-up: the ramp-up time for reaching target concurrency
  • hold-for: the time to hold the target concurrency

After defining the test, it’s time to add the “scenarios” section to the YAML file. This section tells Taurus which steps to execute during the test. 

1. First, we are going to name the scenario. This name matches the scenario in the “execution” section.

execution:
  - executor: jmeter
    concurrency: 5
    ramp-up: 10s
    hold-for: 60s
    scenario: deck of cards 
scenarios:
 deck of cards:  

 

2. Then we will add requests and transactions. The transactions are the same as the Transaction Controllers on JMeter. They are useful to identify the different steps of our tests.

execution:
  - executor: jmeter
    concurrency: 5
    ramp-up: 10s
    hold-for: 60s
    scenario: deck of cards 
scenarios:
 deck of cards:
  requests:
   - transaction: Shuffle the cards
     do:  

 

3. Inside those, we will list the individual requests to be executed. We have a GET HTTP request to Shuffle the cards. 

execution:
  - executor: jmeter
    concurrency: 5
    ramp-up: 10s
    hold-for: 60s
    scenario: deck of cards 
scenarios:
 deck of cards:
  requests:
   - transaction: Shuffle the cards
     do:
       - url: 'http://deckofcardsapi.com/api/deck/new/shuffle/?deck_count=1'
         method: GET

 

  1. Now we are going to add an assertion to check if we are getting the right response.
execution:
  - executor: jmeter
    concurrency: 5
    ramp-up: 10s
    hold-for: 60s
    scenario: deck of cards 
scenarios:
 deck of cards:
  requests:
   - transaction: Shuffle the cards
     do:
       - url: 'http://deckofcardsapi.com/api/deck/new/shuffle/?deck_count=1'
         method: GET
         assert:
         - contains:
           - deck_id

 

5. As we did on JMeter, on the “Shuffle the cards” request we extracted the value of deck_id and used it later on the “Draw a card” request.

execution:
  - executor: jmeter
    concurrency: 5
    ramp-up: 10s
    hold-for: 60s
    scenario: deck of cards 
scenarios:
 deck of cards:
  requests:
   - transaction: Shuffle the cards
     do:
       - url: 'http://deckofcardsapi.com/api/deck/new/shuffle/?deck_count=1'
         method: GET
         assert:
         - contains:
           - deck_id
         extract-regexp:
          deck_id:
           regexp: '\"deck_id\":\s\"(.*)\",'
           default: NOT_FOUND  
           match-no: 1  
           template: 1  
           subject: body  
           scope: all

 

  • regexp: the regular expression
  • default: thedefault value to use when the regular expression doesn’t match. 
  • match-no: If multiple values have matched, you specify which match to use. In this example, we are matching the first result.
  • template: Which capture group to take
  • subject: the subject for search (body, headers,http-code, url)
  • scope: If we set the scope to “all”, it will check main and sub-samples
  1. Let’s add a transaction called “Draw a card”, which is the second step of our test.
execution:
  - executor: jmeter
    concurrency: 5
    ramp-up: 10s
    hold-for: 60s
    scenario: deck of cards 
scenarios:
 deck of cards:
  requests:
   - transaction: Shuffle the cards
     do:
       - url: 'http://deckofcardsapi.com/api/deck/new/shuffle/?deck_count=1'
         method: GET
         assert:
         - contains:
           - deck_id
         extract-regexp:
          deck_id:
           regexp: '\"deck_id\":\s\"(.*)\",'
           default: NOT_FOUND  
           match-no: 1  
           template: 1  
           subject: body  
           scope: all
   - transaction: Draw a card
     do:

 

7. Inside this transaction, we’ll add a GET HTTP request to draw a card. 

execution:
  - executor: jmeter
    concurrency: 5
    ramp-up: 10s
    hold-for: 60s
    scenario: deck of cards 
scenarios:
 deck of cards:
  requests:
   - transaction: Shuffle the cards
     do:
       - url: 'http://deckofcardsapi.com/api/deck/new/shuffle/?deck_count=1'
         method: GET
         assert:
         - contains:
           - deck_id
         extract-regexp:
          deck_id:
           regexp: '\"deck_id\":\s\"(.*)\",'
           default: NOT_FOUND  
           match-no: 1  
           template: 1  
           subject: body  
           scope: all
   - transaction: Draw a card
     do:
       - url: 'http://deckofcardsapi.com/api/deck/${deck_id}/draw/?count=2'
         method: GET

 

  1. Finally, we are going to add an assertion to the “Draw a card” request.
execution:
  - executor: jmeter
    concurrency: 5
    ramp-up: 10s
    hold-for: 60s
    scenario: deck of cards 
scenarios:
 deck of cards:
  requests:
   - transaction: Shuffle the cards
     do:
       - url: 'http://deckofcardsapi.com/api/deck/new/shuffle/?deck_count=1'
         method: GET
         assert:
         - contains:
           - deck_id
         extract-regexp:
          deck_id:
           regexp: '\"deck_id\":\s\"(.*)\",'
           default: NOT_FOUND  
           match-no: 1  
           template: 1  
           subject: body  
           scope: all
   - transaction: Draw a card
     do:
       - url: 'http://deckofcardsapi.com/api/deck/${deck_id}/draw/?count=2'
         method: GET
         assert:
         - contains:
           - deck_id
 

The final script will look like this:

execution:
  - executor: jmeter
    concurrency: 5
    ramp-up: 10s
    hold-for: 60s
    scenario: deck of cards 
scenarios:
 deck of cards:
  requests:
   - transaction: Shuffle the cards
     do:
       - url: 'http://deckofcardsapi.com/api/deck/new/shuffle/?deck_count=1'
         method: GET
         assert:
         - contains:
           - deck_id
         extract-regexp:
          deck_id:
           regexp: '\"deck_id\":\s\"(.*)\",'
           default: NOT_FOUND  
           match-no: 1  
           template: 1  
           subject: body  
           scope: all
   - transaction: Draw a card
     do:
       - url: 'http://deckofcardsapi.com/api/deck/${deck_id}/draw/?count=2'
         method: GET
         assert:
         - contains:
           - deck_id

 

To execute the script, navigate to the location where you saved the script in your command line window and execute bzt.yml. Where “.yml” is the name of the script. 

As the script executes, a live report will be displayed showing various metrics.

 

https://imagecompressor.com/

 

Once it finishes, it will create a folder with the current date on the location where you ran the script which contains all the information of that execution.

 

API Performance Testing with BlazeMeter

Finally, if you want to move to the next level and improve your experience doing API performance testing you can use BlazeMeter. BlazeMeter provides advanced reporting, cloud scalability, CI/CD integrations, and more.

All you have to do is upload your JMeter or Taurus script to the BlazeMeter platform. Let’s see how to do it! 

1. Click on “Create Test”

 

BlazeMeter load testing

 

2. You can choose between “Performance Test” where you can upload a JMeter or YAML script or “Get it now” to install the BlazeMeter Chrome extension to create a new test. 

 

BlazeMeter load testing

 

3. If you choose “Performance Test¨ you’ll be redirected to this dashboard.

BlazeMeter load testing

 

4. Click on “Upload Script” and select your JMeter or YAML file.

BlazeMeter load testing

 

 

5. Specify your load configurations.

JMeter load testing

6. Click on “Run Test” to execute the script.

 

BlazeMeter load testing

 

 

7. Once it is up and running, a live report will be displayed.

 

 

BlazeMeter load testing

 

 

All of these tools are great options for API performance testing. Each offers different benefits:

  • JMeter allows you to create tests with a GUI, making it more user-friendly and letting them run from your machine.
  • Taurus enables simplified coding with quick results.
  • BlazeMeter lets you scale your JMeter and Taurus scripts to thousands of users and run your test from different locations using a user-friendly dashboard for looking at the live results.

START TESTING NOW