Stress Testing, Soak Testing and Spike Testing Best Practices with BlazeMeter

Performance testing is a critical practice for any organization that wants to ensure the stability, quality, and impeccable user experience of its apps and services. Stress testing is an important type of performance testing and one of its building blocks. Stress testing is used to understand the ability of your system to handle extremely heavy traffic loads that could affect responsiveness and service. This post explores soak testing vs stress testing and provides best practices for running these types of load testing and explains how to run stress tests with JMeter and BlazeMeter.

What is soak testing vs stress testing?

Soak testing, also known as endurance testing, takes the system and application to its limits AND over a sustained period of use.  Stress testing tests beyond normal usage of the application for a shorter time to see if it breaks or not and how quickly the system recovers from that stressful load. 

What is endurance testing? 

Endurance or soak testing takes the system and application to its limits AND over a sustained period of use.  

What is stress testing?

Stress tests are run by developers and testers to validate and examine the App-Under-Test’s (AUT) endurance under heavy user loads, the maximal load it can handle, and how the system recovers from that stressful load. Heavy traffic spikes can cause slow responsiveness, errors, and perhaps downtime. If your stress test scenario aims to cover an abrupt and aggressive ramp-up in the number of virtual users for a short span of time, it is called a Spike Test. 

Stress Testing Your Apps and Systems

Why should we stress test our apps?

There are dozens of types of software tests out there (load, unit, functional, API). While we can run them all,  it’s recommended to prioritize according to your business use case needs.  It's important to know the difference between soak testing vs stress testing. 

Stress testing makes sense if you experience patterns of real user traffic coming to your websites, apps, and services. Especially if it’s in the form of sudden spikes or intense loads. Stress testing is also helpful in identifying your AUT’s limitations, and the respective root causes. These can vary between load balancing issues, memory leaks, storage capacity issues, etc.

📕 Related Resource: Learn more about Performance vs. Load vs. Stress Testing

Soak testing vs stress testing vs spike testing?

There are two main types of Stress testing: Spike testing and Soak or Endurance testing.

Spike Testing

Spike testing Tests the reaction of the AUT to sudden large bursts or spikes in the load generated by the virtual users. 

Endurance Testing

Many developers ask what is endurance testing - endurance or soak tests are run to make sure the software can handle the expected load over a long period of time. Endurance tests are similar to spike tests, except they test the system over time.

Stress testing use cases and scenarios

• An e-commerce platform getting ready for peak online shopping season, e.g. Black Friday.
• A healthcare company ensuring its website and apps can handle the potentially extreme load of open enrollment season.
• High-demand live streaming events
• User traffic as a reaction to lucrative promotions

Stress testing best practices

To help with stress testing, here are best practices for building tests, scripting and monitoring your load tests.

1. Test calibration

Calibrate tests to ensure the load engines are not the bottleneck.

2. Add assertions to your JMeter script

Don’t forget about Assertions. Assertions help you validate the test. Not only that it’s performing as expected, but that it’s also hitting all the endpoints it should, and the responses are within the relevant criteria. This could be the actual response body, a certain JSON payload, or even how long the response time was.

Under stress, apps tend to behave oddly. An assertion will not just help you to identify such behavior, but will also give you a head start with troubleshooting. To take advantage of this ability. Use assertions with meaningful failure messages.

3. Add timers to your JMeter script

While some might think that in order to stress the AUT, it makes sense to bombard it with a continuous swarm of requests with no delays between them, this is actually counter-productive. It is a best practice to use Timers to add realistic delays between the requests, because even in a real-life stressful event, users will wait a few seconds before they browse to the next page, or submit a form. Unrealistic scenarios make bad tests, that provide skewed results.

4. Remove Listeners before uploading to BlazeMeter

BlazeMeter has advanced reporting of its own and listeners are resource-heavy.

5. Monitor performance KPIs

Monitor the following KPIs during spikes: Average and Median Response Time, the Response Time’s 90% percentile, connect time, and error rate. In addition,monitor the server-side for the Memory and CPU utilization, Disk I/O, etc.  It’s equally important to test how the AUT will recover from that burst of traffic, i.e. how fast we will go back to the “normal” behavior, if at all.

6. Establish a performance baseline

We must first establish a baseline to understand what our “normal” behavior is, and its compliance with our SLA. If we won’t have a clear and stable baseline we’ll only have half of the story when running our spike test, as we won’t be able to measure our recovery.

7. Monitor the client-side and the server-side

Run the test while monitoring closely. While this may sound obvious, it’s important enough to point out that our test represents the client-side, i.e. how the end-users will experience our AUT’s performance. However, we can’t rely solely on that. Without properly monitoring the server-side, we won’t have any visibility into the Root cause of any performance issues we may find.

8. Continuous testing for each version

Ideally, run for every new version released. Running such a test once is a good start. Running it within relevant cadence will help to ensure our quality and stability, and will help us identify regressions. Running the test every release will ensure we’re always alert and ready to address any performance issues we’ll find before we release a new version with regressions in it.

9. Isolate relevant services

Isolate services that are not ready or relevant in order to test only what’s relevant.

How to run a spike test in BlazeMeter

BlazeMeter can run many different open-source test execution tools. However, JMeter would be the best approach for this use-case as it will allow us flexibility pre-test and during.

Options 1: The JMeter Ultimate Thread Group for spike tests

This approach will require planning ahead. The Ultimate Thread Group allows the tester to add multiple ramp-ups and ramp-downs during a single test. Those could be gradual, or less so. For our matter, you could schedule a spike to be injected during the test, e.g. after 10 minutes of a gradual ramp-up and then stable load, ramp up to 10X within 10 seconds, then after 1 min, ramp down the previous load level.

The downside of this approach is that you will not have control over the concurrency during the test run. You won’t be able to determine when the spike will happen, or the concurrency it will reach.

Option 2:  JMeter Properties for spike testing

This approach will allow maximum control and flexibility. Determine over the number of threads you’ll run at the start of the test, as well as when you’ll want to dynamically generate spikes. Use JMeter Properties for the number of threads and modify the Properties while the test is running using Real-Time Control.

Step 1: When building your JMeter test plan, add a parameterized value with a JMeter function called __P  in the Thread Group’s ‘Threads’ configuration, instead of an absolute number. JMeter will look to read out a value from a Property, if it will find one. Example: {__P(threads,10)}. In this case, the __P function will read the value off of the ‘threads’ property, if it exists. If not, it will use 10 as the default value.

Step 2: After running the script successfully in your local JMeter, upload it to BlazeMeter as a new ‘Performance Test’. You will notice another feature in the test configuration page called “JMeter Properties”.Add the Property’s name from Step 1, e.g. ‘threads’, and provide the value you wish to start the test with, e.g. 100.

Step 3: After you’ve started the test, while it’s running, you will notice a ‘Run Time Control’ button on the top-right corner of the report. Click on it, and then on ‘Remote Control’. You will then be able to specify the new value of the ‘threads’ property to the preferred maximum concurrency of your spike, e.g 500.

Step 4: Once you wish to tear-down the spike, redo step 3.is time, the new value should be updated to the pre-spike concurrency level, in our example, to 100.

Step 5: Be alert for any reaction in the report in the form of an error rate or response time increase.

That’s it! You’ve created a spike in your test and you can continue to add multiple spikes during the test if needed.

Endurance testing in BlazeMeter

Step 1:

Set the test duration per your business requirements (ensure your plan allows you to reach that duration, otherwise, contact your BlazeMeter account manager).

Please note:, any cloud test in BlazeMeter, i.e. a test that utilizes BlazeMeter’s cloud locations, that is longer than a 3 hours duration, will be provisioned with XXL instances for the load engines. This is done to avoid crashing over the lengthy duration of the test, or running out of storage capacity.

Step 2:

Add load in real-time as needed, by either:

• Using the same “JMeter Properties” technique as in the Spike testing above, OR
• Starting a test dynamically and adding it to your existing test run. 

Step 3:

Configure your test in BlazeMeter. 

Step 4:

Create a new Multi-Test.

Step 5:

Add your test to the Multi-test.

Step 6:

Start the Multi-test.

Step 7:

This is a  step if  you require more load and don’t want to restart the test.

While the test is running, go to your test from Step 1, change the configuration, and restart it. You will be prompted to run it as a single test or run it as a test under the mutli-test you’ve started previously. You will want to choose the latter.

That will add the load of the “new” test to the existing Multi-test. You’ll see the report reacting accordingly in 2-4 minutes, so expect the users’ graph to increase.

That’s it! Now you know the difference between endurance or soak testing vs stress testing and how to configure and run both Spike and Endurance tests in BlazeMeter in order to stress test your App, Website, or Services.

START TESTING NOW

Related Resources

• 4 Endurance Testing Tips You Need to Know