SoapUI Load Testing Tutorial: Getting Started and Best Practices

If you are starting with load testing, you will usually conduct it in a test environment similar to the production environment, where the application is subjected to varying load patterns. You can collect performance metrics using tools such as JMeter, BlazeMeter, and SoapUI, including application response time, error rates, throughput, requests per second, memory, and CPU statistics when subjected to custom load testing cases.

In this blog, we will go through a SoapUI load testing tutorial. We will understand open-source SoapUI testing in greater depth, as well as discuss SoapUI load testing best practices, strategies, and considerations.

Table of Contents:

• What is SoapUI Testing?
• SoapUI Load Testing Tutorial: Getting Started
• Considerations for SoapUI Load Testing Strategies

What is SoapUI Testing?

SoapUI is a functional testing tool for SOAP and Web Service testing. It also offers a no-code load testing functionality for complete coverage of custom test cases in a single test environment. 

Users can extend existing functional API tests to build advanced test case scenarios with a range of features such as:

• Click and Run: Build and run new load tests with a few simple clicks.
• Multiple Load Strategies: Simulated multiple load patterns with built-in strategies.
• Performance Validation: Evaluate service performance with built-in assertions.
• Automation and Integration: Additional tooling can be integrated using a simple command line interface.

SoapUI Load Testing Tutorial: Getting Started

SoapUI load testing is based on your existing functional test cases, so this tutorial assumes that you already have a SOAP project in place. The tool allows you to create a new SOAP or REST project as well as import existing packaged projects. For this tutorial, we start the process by creating a new SOAP project of calculator functionality. 

Create a New SOAP Project and Test Suite

- Simply click the SOAP icon:

- Fill in the Project Name and add the Initial WSDL path.

For our project, we use the Project Name: the calculator and the WSDL path are available here: http://www.dneonline.com/calculator.asmx?wsdl

In order to create a load testing suite with a few simple clicks, we check the boxes shown below.

The Request checkbox refers to the functional requests of the test case and the Test Suite refers to the load testing suite for each of the test cases. Fill in the dialog box and press OK.

Generate the SoapUI Load Testing Suite

Since we are automatically creating the TestSuite, SoapUI provides more options for generating the TestSuite as shown in the dialog box below. For this project, the four test cases correspond to requests for the Add, Divide, Multiply and Subtract functionalities. As you can see, we have also chosen to generate a load test for each case independently.

Note that the load testing suite for different (sets of) Requests can also be generated and modified later as required.

On the left menu pane, we see the CalculatorSoapTestSuite, where we added all functional cases to be tested individually. For instance, the Add TestCase includes a Request for the Add Test Step only.

Build Custom Load Test Steps

In order to create a load testing suite in SoapUI with all cases, click on the CalculatorSoapTestSuite and press CTRL+N to create a new test case:

After clicking CTRL+N, a dialog box appears to name the TestCase. We proceed with creating a test suite containing all calculator test cases and naming it as AllTestCases.

We then add existing Requests to this load testing suite. Right-click on the highlight AllTestCases in the menu bar and select a new SOAP Request:

Add the new SOAP Request by choosing the available requests from the drop-down menu:

Specify the Name and add Assertions to validate performance. We will discuss this step more at a later point:

Repeat the New TestRequest process for all Test Cases and the TestEditor (left click on AllTestCases and hit Enter) will show all Requests as follows. You can also Run the functional test using the green Play button to identify possible errors.

Create and Run SoapUI Load Test

We have now created the Load Test on the AllTestCases suite. Now, all that is left to do is to highlight the Load Tests option and press CTRL+N to create a new load test.

Specifying the name of the new SoapUI load test will automatically create a load test containing all of the Requests within the corresponding Test Case. The new interface will appear as follows:

Select the Load Testing Strategy

We start off by selecting the Simple strategy that requires the following parameters:

• Threads: Number of concurrent users simulated. While there is no optimal number suggested, the open source SOAPUI version is known to perform well for up to 100 Threads.
• Test Delay: The delay (milliseconds) between each TestCase run.
• Random: The factor multiplied by Test Delay to randomly assign the delay between each case.
• Limit: Available options are Total Runs, Seconds, and Runs per Thread. We set this parameter to 60s of the complete load test run.

Once the test is executed, it returns logs for various metrics:

• min: minimum time for the load step (millisecond).
• max: maximum time for the load step (millisecond).
• avg: average time for the load step (millisecond).
• last: time for the final load step in the test (millisecond).
• cnt: number of executions per load step.
• tps: transactions per second.
• bytes: header data volume processed in the load step (bytes).
• bps: header data volume processed as bytes per second in the load step (bytes per second).
• err: number of assertions occurred in the load step.
• rat: percent of requests failed (percent).

The results can be plotted using the graph statistic options. You can select to view each Test Step (Request: Add, Divide, Multiply, Subtract) individually or in total. Resolution refers to the sample of data points plotted (250 to 1000 or the simulated available data points).

SoapUI Load Testing Strategies

We can now review the available load testing strategies in SOAPUI open source tool:

• Simple: Executes the test with a configurable delay (as demonstrated earlier).
• Burst: Executes traffic bursts for a set duration and delay.
• Thread: The number of concurrent user traffic is varied between fixed threads from start to end.
• Variance: Varies the concurrent user traffic over a set interval.

Considerations for SoapUI Load Testing Strategies

There are some important points to consider when choosing your SoapUI load testing strategy:

1. Since the Strategy script option is not available for the Free version, you can manually run different strategies and export the data statistics to a separate file for further analysis using the icon below:

2. Review the Settings option to understand how the metrics are calculated and configured. For instance, the time calculations can be based on the actual time elapsed, which may be affected by the performance of your local machine and load conditions. 

3. TPS (Transactions per Second) are calculated as (1000/avg)*threadcount) by default. For the Simple test strategy, this means that avg will be low due to the introduced Test Delay. Instead, calculate the TBS/BPS based on actual time passed which is calculated as (time-passed/cnt).

4. The options of Include Request Write and Response Read will capture the actual time including requests, proxies, and authentication process during network communications between client/server.

5. Sample intervals and maximum assertions can be used to limit extreme values of errors. Set these values based on what you consider to be adequate thresholds of load testing metrics violations.

Performance Validation

Finally, we add a performance validation step to terminate the load test by adding assertions. The LoadTest Assertions are configurable parameter thresholds that register an error when specific metric values are reached. Assertions can also be configured to terminate a test case when a configured value is reached.

Click the LoadTest Assertions icon at the bottom pane.

Select the type of Assertion required, and configure the threshold values as required. For this case, we purposely set thresholds that can be easily violated during the Load Test to evaluate the results.

As expected, the violation errors are registered and the graphs show the trajectory pattern of errors during the test interval:

Additional Scripting

You can further add custom execution properties in the Setup Script (before load test execution) and Teardown Script (after load test execution) editors. The following usage scenarios can be relevant:

• Initializing resources such as objects shared data
• Removing metrics log data
• Reporting for execution failure conditions such as exceeding execution time.
• Creating shared objects.

Bottom Line

In this tutorial, we have reviewed the end-to-end process of load testing with the free version of SoapUI. The SoapUI testing process requires a few simple clicks to evaluate the available test cases based on predefined strategies, parameter settings, and assertions. We have also discussed key considerations pertaining to SoapUI load testing strategies and ways to evaluate the metrics data using graphs. 

Let us review some additional considerations and best practices to develop an accurate and most insightful load testing program with SoapUI:

1. Extensive Testing Environment: Your test environment setup should be as close as possible to the production environment. Adequate system resources may be required to run an environment with a large set of component integrations. While SOAPUI offers sufficient capabilities for limited load testing applications as discussed earlier, you can already get a flavor of advanced capabilities with the BlazeMeter Load Testing free trial, especially for testing workflows between distributed testing machines.
2. At the early stages of development consider mocking up your API to simulate real-world app functionality and behavior. Live testing may also be useful (but risky due to potential load impact on end-users) approach for apps already in the production environment stage to extract true real-world usage metrics.
3. API Virtualization can take your testing process to the next stage that gives you granular control over a variety of environment variables such as usage, connections, and network traffic parameters.
4. Develop a careful understanding of the load testing metrics. The app can perform well if it maintains high tps/bps despite a relatively high or scaled volume of threads, but how do the error patterns and assertion violations vary in response?

The SoapUI platform can serve as a first step to load testing your applications, but the feature limitations in terms of analytics, automation scripting, and the available server resources can be a limiting factor. For massively scalable, open source-based performance tests, try BlazeMeter load testing for enhanced performance testing that scales up and runs in parallel to speed accelerate the test cycles.

Start Testing Now