[Performance] Handling Errors in Test Execution  

Things don’t always go as expected, and sometimes tests fail to execute. There can be multiple reasons, from an error in provisioning load generators to a missing .csv file in the test. The result can be a test ending with no data at all, or a test with partial data. Until now, if an execution error caused partial data, it wasn’t easy for testers to realize there was a problem. In addition, whether the report had partial data or no data at all – testers had no clue what went wrong, and they had to dig in the logs to understand which engine didn’t produce data and why. 

Now we make sure to surface all kinds of errors in execution, via a new errors/warnings infrastructure on the master (report) level: reports that ended with no data will get a new status - “Error”- and information about the reason will be displayed in the Summary report, along with advice on the next action to take. Reports that ended with partial data will remain in their usual status (Pass, Fail, Not Set – according to the failure criteria), but warnings will be displayed in their Summary report, as well as in the Trend Charts page.  

As part of this change, we’ve also updated the status of test runs that were aborted during the booting phase. These reports will now be displayed as Aborted – to differentiate them from reports that ended without data due to execution errors. 

On top of the above, we’ve also added support for workspace alerts, triggered in cases when a test run ended with errors/no data (this alert previously called “Test Ended Without Data”) or with warnings/partial data. This way, you can be notified immediately in case something went wrong, reconfigure the test if needed, and ensure the test can run successfully again. 

Please note that this feature requires a new version of Taurus Cloud. If you are using private locations with an outdated Taurus Cloud image you will not be able to fully enjoy this feature.