What to Include in a Load Test report: Technical Vs Management Reports
Ok, what's the bottom line: Why do we run load tests? For fun? (well, sometimes, load testing IS pretty cool.)
We perform load and performance tests to make certain that our hard work designing, programming, building and creating websites and applications do not fail when they go live.
We need to ensure that these apps and sites are ready to be deployed on a production server, that they can handle the varying loads or influx of concurrent users (aka traffic).
And load testing reports? Gotta have 'em.
Usually, we run our performance test check, then analyze the report. And the performance reports better have lots of pertinent info and they had better be as clear to read and analyze as a blue sky on a sunny morning.
Only after analyzing the report (and tests in action) can the right decisions be made as to whether the app or site is ready to go live.
Load testing is the dress rehearsal before the live show.
The performance report; the Director.
If there is a leg that will break, better it happen during the rehearsal when it can be handled discreetly i.e., no audience or live users.
Sometimes, us techies may....ermm..underestimate the value of the report, mostly because we are busy and writing a good report sometimes just takes more time that we have to dedicate to it.
But over and over again, it must be stated, we run load tests so that we can make the informed decision as to whether these expensive sites and apps are ready to go live or not.
We cannot afford to devalue the interpretation of raw testing data.
That's why we are often faced with ‘out of the box’ graphs, figures and percentiles presented at the end of performance test cycle with no descriptive summary.
And you better believe it is the the liability of the performance tester- who must be able to clearly explain these figures into the context of the business.
There are many ongoing forum disputes about what the ideal load testing report needs to contain. While there may not be much agreement over the perfect performance report, you can always adjust the testing results depending on the target audience.
Our targeted audience can be divided into two segments: technical specialists and managers. we will review both these types of load test reports.
Technical level report
This level of reporting should include the following sections:
I.The scenario description with main breakpoints during test runs.
The person who will read the report did not perform the tests and may have little to no knowledge about what was tested. So, be as clear as possible.
II.The total number of testing cycles.
III.Both the inscope and outscope;
Do not add all graphs rendered after testing. The best testing reports should have a few select graphs, generally those that will prove optimal output at the end of the article presented.
If your report contains only raw statistics, it is almost impossible to understand the test results. The performance tester is the point person, obviously, who must provide the interpretation.
VI.Some formulas (if they are applicable to the particular test results); Reports that are full of formulas are very challenging. It's extremelly difficult for the non-tester to wholly understand what lies behind the many sequences of math symbols.
VII.Comparison with baseline or previous results ;
Is the current situation better or worse? What changed since the last execution? What changes were made in the application code?
VIII.Identification of bottlenecks in the application;
If any issues were found, then an investigation to debug and test again should begin.
IX.Recommendations as to how to resolve the bottleneck;
Add recommendations regarding what should be checked and which parameters should be adjusted.
The Nitty Gritty: Which performance metrics should be added to a load test report?
While it is definitely up to the individual performance tester, we suggest several of the most common:
- Response time Vs Running vUsers;
- Throughput and Hits per second;
- Error vs Running vUsers;
- Number of failures Vs running vUsers;
Management level report
This type of report should have another tier of detail added. On the other hand, it has another target audience altogether.
That’s why it's not good to include a lot of statistics, because managers may not have the need or understand it.
Most important to include:
- Provide a summary that is management based, highlights risks and areas of further investigation.
- Describe the scenario as a Use Case(or user story) that contains action items which may perform like the real users of your system. This will show if the tests matches user actions.
- GO\NO-GO decision. Managers should know whether the system is release ready or not at any given moment, in the current state. Time is money.
- In other words, management level reports should be like a business summary which can be extended with technical details, if needed. And it should run between 1-2 pages, no more.
If you a hard core techie who lives and breathes technical jargon...you may want to consider asking a non-technical colleague to review for clarity before passing it forward.
Do you speak business?
A good indicator of interest in the report is QnA and by that we mean Questions and Answers. It's near impossible to write a good good report that will be crystal clear after the first reading. So, if you do not get any questions, comments or feedback, it might be time to think about the usefulness of the particular test or check to make sure the report itself wasn't gibberish to the manager.
No questions or interest is a red flag.
Sample Q n A related to load test reports:
Q:“I’ve got the results, but they seem slightly slow. What can we do with this info?”
A:”Highlight all risks in the management summary and provide it to the associated stakeholder to make a decision.”
Q:”It’s not my duty to analyze and interpret results. I’m only a performance tester, I only perform tests and give statistical analysis.”
A:”You are an expert in your area. And it is definitely your job to interpret statistics into an easy to decipher business context, it's part of your mission on these projects. You are QA engineer – if you can’t, who can?”