Test Automation Pipeline - Learn How to Build Your Own

In the past years, software delivery has made a big step forward thanks to the implementation of continuous delivery. Quality Assurance is no exception, and the acceleration of the test automation pipeline has helped teams shorten development cycles and deliver higher-quality software.

This blog will take a closer look at the test automation pipeline and offer tips for you to build one on your own by focusing on two phases: the test automation framework and continuous integration.

What is a Test Automation Pipeline

A fast and reliable test automation pipeline is a system of running a variety of tests on a daily basis against different test environments. This pipeline is usually based on multi-level automation tests and stable continuous integration.

Not long ago, setting up, configuring, and maintaining a complete test automation pipeline was a huge challenge. But innovations that are occurring in QA methodologies and infrastructure applications now allow you to spend minimum time on this front, so you can concentrate more on your application.

Choosing a Framework for Your Test Automation Pipeline

The Test Automation Framework is the structure that provides engineers with a way to develop, maintain and execute automation tests, to test their applications. In other words, it is a system designed to test another system. In order to build such an automation test framework, you need to come up with a list of tools that you are going to use to verify the many different aspects of the system which you are going to test.

There are tons of questions that you need to ask yourself when choosing the right tool to proceed with test automation. But ultimately, this choice should be based on your team’s skillset. Otherwise, you might lose several weeks fighting against the tool instead of building a solid automation framework. Usually, it is better to start with the programming language selection and select the language which most of the people in your team are familiar with.

After you made your choice about the language, it’s time to choose the framework set. Here as well, your choice should be based on your team and mainly on the team’s skill set. If you have a few experienced engineers with programming skills and a big team of less experienced or manual engineers, then you might prefer tools that allow separation of the test framework development and the test creation process, which will be mostly in human-readable language.

Cucumber and Robot Framework are good examples of such frameworks. Such tools are also better in situations when nontechnical people need clear visibility into the tests automation pipeline and the QA processes overall.

On the other hand, human readability takes time and has limitations. One example of this is that in Robot Framework, the variable scope might be less than obvious, and in Cucumber it might be a complicated workflow of sharing data between steps, in comparison to pure Java.

So, if you have a team of very skilled test automation engineers who can write very clear automation tests in code instead of plain human-readable text files, and you don’t need to share these tests with nontechnical people, then you might save time on test automation by avoiding such frameworks.

In any case, each situation is different and you need to make this choice uniquely based on your test requirements, team, supported platforms, and long-term goals. But one important tip to save you from a huge mistake - do not let managers decide which automation tools your team need to use! It should be only your team’s collective choice!

To save time and concentrate mainly on the QA test automation pipeline, I have created a small automation framework that you can use to stay on track with this article. Feel free to use it for your needs if you find it useful:
https://github.com/BushnevYuri/e2e-automation-pipeline

This project shows you one of the options to group your automation test levels. The project contains examples of smoke, API, UI, and performance tests. There is a list of prerequisites to run all the project tests:

• Java >= 1.8 version (installation link)
• Apache Maven >= 3.0 (installation link)
• Taurus (installation link)

We are not going to force you to choose one tool or another, but rather share one of the possible tool stacks. This stack is used in many companies and works very well if you are comfortable with Java programming language:

• JUnit - a basic Java unit tests framework
• Serenity - the backbone of the automation framework that provides many useful features, like detailed reporting, parallel execution and tests grouping by tags
• REST Assured - a framework for REST API  tests
• Selenium - a framework for UI automation tests and web browser interaction
• Taurus - a performance testing framework that allows you to write performance tests via yml scripts

With the help of the Serenity framework and its ability to run tests by tags, you can run smoke, API, and UI tests accordingly:

mvn clean verify -Dtags="type:Smoke"

mvn clean verify -Dtags="type:API"

mvn clean verify -Dtags="type:UI"

You might notice that the UI tests failed. The reason is pretty simple. To run UI tests you need to install the Chrome web browser and also configure webdriver, which is required as part of the Selenium framework configuration. All you need is to download the latest webdriver based on your OS and specify the path to this driver in the “src/main/resources/serenity.properties” configuration file:

webdriver.base.url=http://blazedemo.com/
webdriver.driver=chrome
webdriver.chrome.driver=[PATH TO THE DRIVER]
webdriver.timeouts.implicitlywait=10000
webdriver.wait.for.timeout=10000

Performance tests are the one exception for which you need to use a specific command to run the performance test scenario (this should be executed from the project root folder):

bzt src/test/java/performance/BlazeDemoPerformanceTest.yml

Automating Test Executions

As soon as you implemented your first test automation suite, it’s time to take care of your execution as part of a larger test automation pipeline. Of course, you can run your tests manually, but it doesn’t make sense if you can automate this step as well.

Automating execution has many benefits:

• Faster tests result in feedback (you can configure automation tests to run as soon as the code has been committed)
• Some tests might have a longer execution time and you can set up nightly runs to verify results in the morning
• Fewer mistakes can happen if steps are automated (avoid human errors)
• You can configure several agents to run automation tests in parallel in different environments

One of the most commonly used CI servers for automation needs is Jenkins. It is open source, allows you to run any kind of tests, and integrates with almost anything you can imagine.  It has one of the biggest open-source communities, and they bring Jenkins to new levels year after year.

Jenkins installation is very straightforward and you can set up a CI server on your local machine in a few minutes in several ways. One of these ways is to use Docker. Have you used Jenkins before but without Docker? It’s time to take a step into the future to see why it is actually so cool.

Dockerized Jenkins

Let’s think about issues that you might have during the installation, configuration, and maintenance of your local continuous integration server:

• Even if Jenkins can be installed in a few minutes, it still doesn’t mean that you can use it right away, as you need to install required plugins (to use git, make reports or parallel builds) as well install additional software on your local machine (like Java or Maven)
• Sometimes you might need to move Jenkins to another server (it might take you a long time to proceed with Jenkins relocation)
• You might get new requirements for running your tests from a cloud (for example from AWS servers due to some security concerns)
• Sometimes you might get different incompatibility issues during the installation of new plugins or when you proceed with the reconfiguration of your server (and it might be very useful to have the ability to revert your changes)

As you can see, there are many troubles that lie just around the corner until you use Jenkins in a real-life project. The good news is, Docker will help you avoid all these troubles.

Docker, for those who don’t know, is an open-source technology that provides you with a virtual container that can pack applications inside. These can be run in a few steps on any machine after that. Docker is a wonderful tool when you need to create reproducible infrastructures. By using Docker you can easily replicate, move, revert and do almost anything with your local Jenkins infrastructure. There are many detailed tutorials on using of Jenkins with Docker,,so let’s not waste time on details and briefly go over the main installation steps in order to build the backbone of our automation pipeline.

Installation

First of all, you need to install the Docker service on your local machine. You can find the installation steps at this link, for your local machine configuration.

After Docker is installed, you can verify if it was done correctly by using this command in your terminal:

docker -v

Now, you can setup and run Jenkins by using just the one command (assuming that you will replace ‘[/your/home]’ with some of your local folders):

docker run -p 8080:8080 -p 50000:50000 -v [/your/home]:/var/jenkins_home jenkins

This command will take the Jenkins image from the cloud Docker repository. After that you should see the logs of the running container:

In the command above, “-p 8080:8080 -p 50000:50000” means that you want to forward ports from the Docker container outside, to be able to access the Jenkins app outside the Docker container.

Another part of the command “[/your/home]:/var/jenkins_home” is also very important. Docker doesn’t save changes on local files inside the container. Since you want to have all the Jenkins configuration changes to be permanently saved, you need to use your local location as permanent storage.

As soon as the container is up and running, Jenkins will be available on the “http://localhost:8080/” address:

The administrator password can be found in the logs of your running Docker image. Just go and check out your console:

After that, proceed with the basic configuration, which requires personal data forms and the plugins setup page (to be on track with our article, please choose the suggested plugins installation to install all the required plugins).

The power of Docker is that you can install necessary updates into the base image and save the newly updated container as a separate image. Then, you can use it as a base image. To be able to run our tests from the automation framework created above, we need to have Maven installed. The default Jenkins image doesn’t have Maven onboard, so this will be a nice example of how you can extend your current Docker image with this new functionality.

First of all, you need to login into the running dockerized Jenkins instance. For that, you need to know the instance ID of your running image. You can get all running instances with this command (the container id will be in the first column):

docker ps

We need to log in to the Docker image with a root user as we are going to proceed with Maven, which requires some administration access. You can use this command to login inside the Docker instance (6d68498dd7b5 is the container id so it might be different in your case):

docker exec -u root -it 6d68498dd7b5 bash

Now, let’s proceed with the Maven installation inside dockerized Jenkins:

apt-get update
apt-get install maven
exit

We have installed Maven inside the Docker container. But if you try to stop and start the container, these steps will be lost. In order to save these steps, you need to commit your changes (it’s better to use your own image name for that):

One other step that might be wise to execute right now is to install the HTML Publisher plugin into our Jenkins instance, as it will be required to aggregate test reports in our build plans. Just go into “Manage Jenkins” -> “Manage plugins” -> “Available” -> search for ‘HTML Publisher’.

Congrats! You have installed Jenkins by using the dockerized Jenkins image. Make some updates inside the container and save your own image, which can be reused on any other machine. Now you have the automation framework, a couple of tests, and a continuous integration server where you can run them. Now it’s time to learn how!

Build Your Configuration via Jenkinsfile and Jenkins Pipelines

Jenkins UI is really great and simple. Jenkins is clear in how to configure a specific build plan and run your tests right away. But as soon as we have lots of tests, it usually becomes a pain to maintain and take care of dozens and hundreds of build plans. Why?

Sometimes you need to introduce the same change into several build plans (for example to add a certain post-build action). In this case, you need to jump from one configuration to another and make these changes, while remembering where you applied it and where you haven’t yet.

Another critical issue is that you do not have any history of the applied changes to your build configurations. You don’t know what has been changed and you are not able to revert the latest changes. Having all these issues in mind, some time ago Jenkins introduced the new way of working with builds - pipelines!

Briefly, a Jenkins pipeline is a continuous delivery through code tool, which is usually added under version control tools like Git. It basically means that you can configure, maintain, revert and add build plans the same way you work with a code. At the same time, the Jenkins pipelines workflow has a nice representation that shows you all steps of your build plan in action. This gives you a nice view on build steps with timing and results for each. A file that contains pipelines code with build configuration is called Jenkinsfile.

To add a build plan with Jenkinsfile you need to open Jenkins, click “New item”, come up with a build plan and choose ‘Pipeline’ in the list of build types:

I have found this tutorial very useful to get an idea of how to create and configure jenkinfiles. But of course, official documentation is a very valuable resource as well. As for our needs, I came up with a small Jenkinsfile based on our automation frameworks from previous paragraphs. You can use it as an example:

node {
   stage('Git checkout') { // for display purposes
      git 'https://github.com/BushnevYuri/e2e-automation-pipeline.git'
   }
   stage('Smoke') {
        try {
            sh "mvn clean verify -Dtags='type:Smoke'"
        } catch (err) {
            
        } finally {
            publishHTML (target: [
            reportDir: 'target/site/serenity',
            reportFiles: 'index.html',
            reportName: "Smoke tests report"
            ])
        }
   }
   stage('API') {
        try {
            sh "mvn clean verify -Dtags='type:API'"
        } catch (err) {
            
        } finally {
            publishHTML (target: [
            reportDir: 'target/site/serenity',
            reportFiles: 'index.html',
            reportName: "API tests report"
            ])
        }
   }
   stage('UI') {
        try {
            sh "mvn verify -Dtags='type:UI'"
        } catch (err) {
            
        } finally {
            publishHTML (target: [
            reportDir: 'target/site/serenity',
            reportFiles: 'index.html',
            reportName: "UI tests report"
            ])
        }
   }
   stage('Results') {
      junit '**/target/failsafe-reports/*.xml'
   }
}

In this Jenkinsfile, we have the step where we check out the code from the repository, a few separate steps to run tests based on their level and a result step to aggregate JUnit test results at the end of the whole test suite. If you prefer https authentication to your git or another version control repository, you need to configure your own repository and the access to this repo by using your own ssh or password.

Once you put the mentioned pipelines code into the pipeline definition windows (the biggest window in the build configuration), you can run the build by using the “Build now” button:

But what if you need to make some changes to the build config? Should you open and change the script the UI? Of course, the answer is no. This is where the main magic happens. All you need is to place this Jenkinsfile into your repository somewhere near your tests.

After that, you can configure build to use the pipeline Jenkinsfile by using the “Pipeline script from SCM” definition. That’s why we have already taken care of that and put the mentioned file into the test project under ‘jenkins-pipelines/E2E-automation-test.groovy’ path. Therefore, you can configure the build plan in a few steps:

You can execute the build again and confirm that it still works fine. At the end of the script, you will see a couple of test reports available on the main build page:

If you are following our steps, you might notice that we have some failed tests after the execution. This is because of the chrome browser and web driver which you need to install and specify in the configuration file, but this time inside the Docker. If you have completed all the previous steps with us, you should have enough experience to do that, so let’s leave this step as your homework.

It is also worth mentioning there are more advanced approaches for docker file creation and configuration. First, you can create own dockerfile that contains all the installation and configuration steps in plain text file. In this case you don’t need to commit and save docker images. You can always build a new Docker instance based on this file, which is basically just an instruction how to build this image. 

As a result, we have tests, test automation framework, dockerized Jenkins, Jenkinsfile and knowledge about how to run everything together. We can say that our QA automation pipeline is ready! But there is another step that we cannot forget. 

A Bit of Blue Ocean

Blue Ocean is a relatively new project developed by the Jenkins community team, which improves the experience of using Jenkins pipelines for continuous integration pipeline representation. We all love Jenkins because it is a stable and mature continuous integration server. But you will love it much more as soon as you try Blue Ocean yourself. 

First, you need to install required plugins. Go into “Manage Jenkins” -> “Manage plugins” -> “Available” -> search for ‘Blue Ocean’:

Once “Blue Ocean” is installed, you will see “Open Blue Ocean” icon in your build plan.

Inside the Blue Ocean you will see your newly created automation tests pipeline. On the page, you will find all the results for each completed stage, actions in progress, related logs with the ability to export them and just a simple beauty of this new continuous integration UX!

The new UX looks really great as soon as you have some parallel steps, for example, UI tests, which will be run for different browsers:

This was a last step and now you have a full dockerized QA automation pipeline with automation framework, some tests and Jenkins with its brand new beautiful UX to run them all.

Bottom Line

I hope this article was useful and gave you useful information for building your own test automation pipeline from scratch. Now, if you want to learn more about Serenity, Selenium, Taurus, Jenkins and automation tests in general, there are plenty of tutorials you can find and dive into.

You can also use a solution like BlazeMeter, which integrates with many of these frameworks and technologies, to build a scalable and sustainable test automation pipeline. Experience BlazeMeter in action by trying it for free today. 

START TESTING NOW

Related Resources

• JMeter Load Testing
• Test Automation Tutorial With BlazeMeter