Aug. 4th, 2016

Understanding Your Reports: Part 2 - KPI Correlations

Welcome to part 2 of our “Understanding Your Reports” series. In part 1, we talked about what KPIs are in the context of Performance Testing. This time, we will go over KPI correlations.

 

Last time, we reviewed the 7 most common KPIs: Number of Virtual Users (VU), Hits per Second, Errors per Second, Response Time, Latency, Connect Time and Throughput. By correlating them with each other, we can use them to discover bottlenecks.

 

Number of Virtual Users (VU), Average Response Time and Errors per Second

 

Let’s look at these three graphs, which correlate the Number of Virtual Users with Response Time and Errors per Second.

 

Graph 1-3, from left to right:

 

Graphs correlating number of virtual users (vu), average response time and errors per second

 

Graph 1 - No Bottlenecks

 

graph correlating number of virtual users (vu), average response time and errors per second. system working smoothly with no bottlenecks.

 

The first graph shows us a system with amazing abilities: The Number of Virtual Users goes up, but the Average Response Time is unaffected and unchanging. This system has no bottlenecks for these variables, and user experience is consistent. Perfect.

 

Graph 2 - Bottlenecks that Might Need Fixing


Graph number 2 shows us something different. Average Response Time is affected by the rise in the Number of VU, and rises itself. However, when the Number of VU steadies, so does the Average Response Time.

 

graph correlating number of virtual users (vu), average response time and errors per second, system might have bottlenecks.

 

Does this system have bottlenecks that need to be fixed? That depends on the Product Manager. If the heightened Average Response Time fits the product’s desired abilities, nothing needs to be changed. But if it’s too high, developers can now know what and where the problem is, and take actions to reduce the Response Time.

 

Graph 3 - System Failure

 

graph correlating number of virtual users (vu), average response time and errors per second, system failure

 

The third graph shows us a system in distress. The Response Time keeps rising even after the Number of Virtual Users remains steady, and, as a result, Errors are created and are on the rise per Second. This happens when the system is unequipped to deal with the rise in the Number of Virtual Users, because it has a huge bottleneck that needs urgent fixing and caring.

 

You can also look at these graphs while using the Hits per Second KPI instead of the Number of VU. By correlating Hits per Second, Average Response Time and Errors per Second, you can determine if the bottleneck is created by the Hit Rate.

 

Let's look at something else now: the correlation between the Number of VU and Hits per Second.


Number of Virtual Users (VU) and Hits per Second (Hit Rate)

 

These graphs correlate the Number of VU with Hits per Second. Let’s say our system requirements are 500 Hits Per second for 1,000 Virtual Users.

 

Graph 4-6, from left to right:

 

graphs correlating number of virtual users (vu) and hits per second (hit rate)


Graph 4 - No Bottlenecks

 

graph correlating number of virtual users (vu) and hits per second (hit rate), system working with no bottlenecks

 

Graph number 4 shows us a system that is working smoothly. The Number of VU corresponds with the number of Hits per Second we defined - 500. This means that the system is able to manage the rise in the Number of VU, without harming their abilities to send requests.

 

Graph 5 - Bottlenecks that Need Fixing

 

graph correlating number of virtual users (vu) and hits per second (hit rate), system has bottlenecks

 

The fifth graph presents a system that has a bottleneck. The Hits per Second rise with the Number of VU, but the stops before reaching the desired 500. This means that each user can perform less actions and send less requests than our requirements. This bottleneck needs to be fixed.

 

Graph 6 - System Failure

 

graph correlating number of virtual users (vu) and hits per second (hit rate), system failure

 

The sixth graph displays a system that can’t cope with the changes in the Number of VU. The Hits per Second rises but then fluctuates  and quickly declines. It eventually rises, but so do the Errors per Second. This means that the Hits are all actually Errors. In this scenario, users are unable to perform actions and the system has a huge bottleneck that needs urgent fixing and caring.

 

BlazeMeter enables you to correlate all KPIs with each other, for optimizing your performance analysis. For more information check out our Timeline Report.

 

Want to learn more about the latest BlazeMeter features and capabilities? Register to view our webinar - Performance Testing with BlazeMeter - New and Advanced Features

 

Now read part 3 of our "Understanding Your Reports" series, Key Statistics Performance Testers Need to Understand
 

Interested in writing for our Blog? Send us a pitch!