Test Automation: The Ultimate Guide To Getting Started

The world is changing and this is also applicable to the world of software development. New methodologies and new approaches are constantly appearing. Their main goal: reduce the time to market for new pieces of software. Test automation enables quicker testing with fewer manual errors.

This blog post will explain what test automation is and why it is more necessary than ever. We will also dive into the types of tests you can automate, how to choose which use cases to automate, and the test automation tools out there. 

What is Test Automation?

Test automation is the process of assessing software in an automated manner.

It includes automation of the following:

1. setUp - prepare the test data or enter the state suitable for testing

2. Run the test - execute the smallest function, a larger piece of code or a simulation of end user activity. Compare the actual results to the anticipated ones.

3. tearDown - revert the changes to put the system into "pre-test" state, so the process can be repeated.

For example, when a new BlazeMeter version is deployed, we want to ensure users can browse the website, all links are working, images are present, and users can log in to their accounts, configure and run the tests.

In order to avoid manual executions of the same steps over and over with each release, it's better to have them automated. This solution is more robust, runs faster, can be run in parallel, and can automatically provide "go/no go" decisions.

Why Software Testing Should be Automated

Before releasing a brand new product or a new version of an existing project into the market, it’s a very good idea to have it tested. For years, the software development lifecycle used to look like this:

1. Requirements
2. Planning
3. Development
4. Testing
5. Maintenance

Testing started after software development was completed. This is also called the waterfall approach. Depending on the product complexity, testing required weeks or even months of manual work to release something with acceptable quality. 

Software testing needs to be:

• Reliable: All critical issues have to be identified. When testing manually, some might be overlooked.
• Unattended: if a developer makes a commit with a killer feature or fixes a showstopper issue at the end of day on Friday, the developer needs to be able to test the whole product immediately without any human intervention.
• Integrated with other activities: adding a new code is added to the repository should trigger the necessary subset of the downstream activities like compilation, running tests, configuration, deployment, etc.   

This means that it is absolutely necessary to have different levels of automated tests executed against the codebase before a new version is delivered to the end users. Otherwise, testing will not be able to keep up with the rest of the agile development cycle, and will impede deliveries and negatively affect code quality.

Nowadays, the shift left approach is becoming more and more popular. The idea is to start testing as soon as it is possible in order to minimize the time to market. As a result, continuous testing, or in other words, continuous test automation, has become an essential part of continuous delivery pipelines. The main goal is to add testing to the product development lifecycle.

Popular Automated Test Types

The following major test types can be automated:

1. Unit Tests

These tests validate the smallest “units” of the software, like specific functions. Unit tests are mostly written by the developers, because they’re mainly for the developers. These tests protect them from introducing new issues while doing code refactoring, bug fixing or adding a new feature.

2. Integration Tests

Integration tests validate the module interaction. Integration tests can be the responsibility of either developers, the testers or a 3rd-party team like DevOps.

3. End-to-End Tests

A set of tests that is executed against a full integrated production-like environment simulating the real-life usage of the system. End-to-end tests are mainly the responsibility of testers and their main goal is to free testers heads and hands for more creative work.

The classic approach of arranging the testing suite is The Test Pyramid. It has been introduced in the book “Succeeding with Agile” by Mike Cohn and suggests having:

1. A large number of unit tests that are “cheap” and “fast”
2. A mid-range number of “integration” tests that are slower and more expensive
3. And a very limited number of end-to-end tests, which are the slowest and the most expensive in terms of development/support 

Of course, this pyramid cannot be applied to each and every software project. If the system doesn’t have a user interface, for example, it doesn’t make sense to run UI tests. If you’re running a serverless function - integration testing can be skipped as well. But this pyramid can be a good starting point for defining the testing scope.

Choosing Test Automation Use Cases

There is no “golden rule” about which use case or test should be automated. So for every project you need to be flexible and analyze the situation “from scratch”. An approach that previously succeeded might not be applicable for the current product. 

The most commonly used tactics are:

1. Start with the “easiest” test cases that are the fastest to implement - this way you will get immediate benefits from the automation
2. Start with “Smoke/Sanity” testing scenario - this way you will quickly have “Go/No go” answer as this test type covers essential requirements
3. Calculate the “return on investment” like:
   • Come up with the list of test cases and the manual effort required to execute them (another factor could be the anticipated frequency of the execution)
   • Start with the tests that take the most time/effort - this way you will be able to reduce the manual testing backlog in the most efficient manner

Choosing Test Automation Tools

There are quite a number of applications/libraries/frameworks for test automation that have their own pros and cons. 

Unit Testing Automation Tools

For Unit Testing there is the xUnit concept, which is the cumulative name of unit testing frameworks for different programming languages like:

• NUnit or MSTest for C#
• JUnit or TestNG for Java
• Pytest or Nose for Python
• And more 

Here the choice is fairly straightforward. Normally there are up to three possible frameworks that have a limited number of features, so it’s often a matter of a personal preference or fitting to the existing ecosystem (like plugins for the Continuous Integration server).

Integration Testing Automation Tools

For integration testing you will probably need the same xUnit framework as the container that provides test configuration, execution and reporting capabilities. But, you will need to use real libraries for the main programming language of your project, for communicating with other system parts or middleware instead of using mocks.

End-to-End Testing Automation Tools

For end-to-end testing there are way more options and factors that need to be considered:

First of all, if you (or your company) already have a toolchain that has been there for ages, or a license to a very expensive and very powerful test automation solution, you might have to stick to other people’s choice, whether you like it or not.

Second, if you have some freedom to choose, the main criterion is that the tool needs to be capable of supporting all the product requirements. For example: if the application under test is cross-platform and the test tool can only be run on Windows this is not a good idea; Selenium is the de-facto standard for web applications testing automation but it doesn’t work very well with Angular apps; Cypress makes the process of tests development much faster and easier but it supports only JavaScript language and Chrome browser, etc. For a detailed comparison, check out Selenium vs. Cypress to see which better fits your needs.

Third, take your team’s and your experience with the testing tools. If several engineers are proficient with a certain tool, it might be a good idea to proceed with it instead of spending time getting familiarized with something absolutely new.

Fourth, ideally the tool should not be incompatible with the main project programming language. This way, the tests will be able to share the Continuous Integration infrastructure. In addition, the test code can be “understood” and maintained by the developers, the code can be peer-checked and so on. So if the project is written in Kotlin, a Ruby-based tool is probably not the best idea.

Fifth, the tool needs to be up-to-date. If it’s abandoned - it will probably not support your modern infrastructure.

Sixth, an active community (forums, mailing lists, the possibility to reach out to developers with questions, raise an issue, etc.) is very helpful. So if you get stuck on something you could get help.

Finally, a big advantage is using open-source tools where possible. This way you can easily troubleshoot, increase logging levels, add missing features, see usage examples, etc.

A Reference List of Test Automation Frameworks

By this time you should already understand that testing automation is super important for bringing confidence into your project lifecycle. You know which tests and use cases to automate, and how to choose the right tools for your tests.

Now the question is what tool to choose. Again, there is no “right” answer for this question, because there are many tools available, from tools that are capable of running only one type of test automation activity to “monsters” supporting different types of software. To help you to choose, here is the list of the most advanced and popular solutions, mainly for end-to-end testing: 
 

It is also worth mentioning our very own automation-friendly framework called Taurus. It is not a test automation tool per se, but it’s a wrapper for 20+ (and counting) test automation and load testing tools that simplifies test configuration and execution processes. The configuration is done via human and the machine friendly YAML syntax. 

Taurus acts like a one-stop-shop for everything connected with all possible test types for almost any software project, providing the possibility to run all the possible test types in one shot and have the results in one place.

START TESTING NOW

Related Resources

• Robot Framework: The Ultimate Guide to Running Your Tests