Yuri Bushnev is a Head of System Quality at AlphaSense, with more than 5 years of experience in Quality Assurance, DevOps and Development.

Become a JMeter and Continuous Testing Pro

Start Learning

Test Your Website Performance NOW! |

arrowPlease enter a URL with http(s)
Oct 19 2017

QA Automation Pipeline - Learn How to Build Your Own

In the past years, software delivery has made a big step forward thanks to the implementation of continuous delivery. Quality Assurance is no exception, and the acceleration of the QA pipeline has helped shorten delivery cycles and bring higher quality software.


A fast and reliable QA pipeline is usually based on multi-level automation tests and stable continuous integration, which runs all of these tests on a daily basis against different test environments. Not long ago, setting up, configuring and maintaining a complete workflow, was a huge challenge. But innovations that are occurring in QA methodologies and infrastructure applications now allow you to spend minimum time on this front, so you can concentrate more on your application. 


In this article, we are going to provide you with a well built launch pad, which you can use to give life to your own automation pipeline. This article will provide you with everything you need to set up a whole QA automation pipeline, while focusing on two phases: the test automation framework and continuous integration.


In this article, you will put your hands on:

  • Creating a test automation framework written in Java programming language (You will also find a link to the framework sample, which contains basic test scenarios created by using some of the most popular testing frameworks (Rest Assured, Serenity and Selenium), which you might find very useful for your needs)
  • Integrating with some of the most well known automation tools
  • Executing different types of tests
  • Installing and configuring a dockerized Jenkins continuous integration server
  • Creating a test execution build using Jenkins pipelines
  • Adding beautiful visualization to created test automation pipeline with the Jenkins Blue Ocean


1. Creating a Test Automation Framework


The Test Automation Framework is the structure that provides engineers with a way to develop, maintain and execute automation tests, to test their applications. In other words, it is a system designed to test another system. In order to build such an automation test framework, you need to come up with a list of tools that you are going to use to verify the many different aspects of the system which you are going to test.


There are tons of questions that you need ask yourself when choosing the right tool to proceed with test automation. But bottom line, this choice should be based on your team’s skills set. Otherwise, you might lose several weeks fighting against the tool instead of building a solid automation framework. Usually, it is better to start with the programming language selection and select the language which most of the people in your team are familiar with.


After you made your choice about the language, it’s time to choose the framework set. Here as well, your choice should be based on your team and mainly on the team’s skill set. If you have a few experienced engineers with programming skills and a big team of less experienced or manual engineers, then you might prefer tools that allow separation of the test framework development and the tests creation process, which will be mostly in human-readable language. Cucumber and Robot Framework are good examples of such frameworks. Such tools are also better in situations when nontechnical people need clear visibility into the tests automation pipeline and the QA processes overall.


On the other hand, human readability takes time and has limitations. For example, in Robot Framework the variables scope might be less than obvious, and in Cucumber it might be a complicated workflow of sharing a data between steps, in comparison to pure Java. So, if you have a team of very skilled test automation engineers who can write very clear automation tests in code instead of plain human readable text files, and you don’t need to share these tests with nontechnical people, then you might save time on test automation by avoiding such frameworks.


In any case, each situation is different and you need to make this choice uniquely based on your test requirements, team, supported platforms and long-term goals. But one important tip to save you from a huge mistake - do not let managers decide which automation tools your team need to use! It should be only your team’s collective choice!


To save time and concentrate mainly on the QA test automation pipeline, I have created a small automation framework that you can use to stay on track with this article. Feel free to fork and use for your needs if you find it useful:


This project shows you one of the options to group your automation test levels. The project contains examples of smoke, API, UI and performance tests. There is a list of prerequisites to run all the project tests:


We are not going to force you to choose one tool or another, but rather share one of the possible tool stacks. This stack is used in many companies and works very well if you are comfortable with Java programming language:

  • JUnit - a basic Java unit tests framework
  • Serenity - the backbone of the automation framework that provides many useful features, like detailed reporting, parallel execution and tests grouping by tags
  • REST Assured - a framework for REST API  tests
  • Selenium - a framework for UI automation tests and web browser interaction
  • Taurus - a performance testing framework that allows you to write performance tests via yml scripts


With the help of Serenity framework and its ability to run tests by tags, you can run smoke, API and UI tests accordingly:


mvn clean verify -Dtags="type:Smoke"
mvn clean verify -Dtags="type:API"
mvn clean verify -Dtags="type:UI"


create your own qa automation pipeline


You might notice that the UI tests failed. The reason is pretty simple. To run UI tests you need to install the Chrome web browser and also to configure webdriver, which is required as part of Selenium framework configuration. All you need is to download latest webdriver by this link based on your OS and specify path to this driver in the “src/main/resources/serenity.properties” configuration file:


webdriver.chrome.driver=[PATH TO THE DRIVER]


Performance tests are the one exception for which you need to use a specific command to run the performance test scenario (this should be executed from the project root folder):


bzt src/test/java/performance/BlazeDemoPerformanceTest.yml


qa automation pipeline, test automation framework


2. Continuous Integration


As soon as you implemented your first tests automation suite, it’s time to take care of automation execution. Of course, you can run your tests manually, but it doesn’t make sense if you can automate this step as well.


Automating execution has many benefits:

  • Faster tests result feedback (you can configure automation tests to run as soon as the code has been committed)
  • Some tests might have a  longer execution time and you can set up nightly runs to verify results in the morning
  • Fewer mistakes can happen if steps are automated (avoid human errors)
  • You can configure several agents to run automation tests in parallel in different environments


Not long ago, configuring continuous integration environments was a big challenge, because it required a special skills set. Therefore, it was usually done by DevOps engineers. But each year, continuous integration servers require less effort to install, configure and maintain, so it can be done by only one engineer in one evening after reading a bit about it, for example an article like this.


One of the most commonly used CI servers for automation needs is Jenkins. It is open source, allows you to run any kind of tests, and integrates with almost anything you can imagine.  It has one of the biggest open source communities, and they bring Jenkins to new levels year after year. If you are new to the Jenkins, you can look at this link to learn almost everything about it, another link if you are looking for a very brief introduction or this third link if your prefer basic video tutorials.


Jenkins installation is very straightforward and you can setup a CI server on your local machine in a few minutes in several ways. One of these ways is to use Docker. Have you used Jenkins before but without Docker? It’s time to take a step into the future to see why it is actually so cool.


Dockerized Jenkins


Let’s think about issues that you might have during installation, configuration and maintenance of your local continuous integration server:

  • Even if Jenkins can be installed in few minutes, it still doesn’t mean that you can use it right away, as you need to install required plugins (to use git, make reports or parallel builds) as well install additional software on your local machine (like Java or Maven)
  • Sometimes you might need to move Jenkins to another server (it might take you a long time to proceed with Jenkins relocation)
  • You might get new requirements for running your tests from a cloud (for example from AWS servers due to some security concerns)
  • Sometimes you might get different incompatibility issues during the installation of new plugins or when you proceed with the reconfiguration of your server (and it might be a very useful to have an ability to revert your changes)


As you can see, there are many troubles which lie in wait until you use Jenkins in a real life project. There is a good news - Docker will help you avoid all these troubles.


Docker, for those who don’t know, is an open source technology that provides you with a virtual container that can pack applications inside. These can be run in a few steps on any machine after that. Docker is a wonderful tool when you need to create reproducible infrastructures. By using Docker you can easily replicate, move, revert and do almost anything with your local Jenkins infrastructure. There are many detailed tutorials on using of Jenkins with Docker (I would recommend to start with this one) so let’s not waste time on details and briefly go over the main installation steps in order to build the backbone of our automation pipeline.




First of all, you need to install the Docker service on your local machine. You can find the installation steps at this link, for your local machine configuration.


After Docker is installed, you can verify if it was done correctly by using this command in your terminal:


docker -v


dockerized jenkins in continuous integration pipelines


Now, you can setup and run Jenkins by using just the one command (assuming that you will replace ‘[/your/home]’ with some of your local folders):


docker run -p 8080:8080 -p 50000:50000 -v [/your/home]:/var/jenkins_home jenkins


This command will take the Jenkins image from the cloud Docker repository. After that you should see the logs of the running container:


how to automate your software development


In the command above, “-p 8080:8080 -p 50000:50000” means that you want to forward ports from the Docker container outside, to be able to access the Jenkins app outside the Docker container.


Another part of the command “[/your/home]:/var/jenkins_home” is also very important. Docker doesn’t save changes on local files inside the container. Since you want to have all the Jenkins configuration changes to be permanently saved, you need to use your local location as permanent storage.


As soon as the container is up and running, Jenkins will be available on the “http://localhost:8080/” address:


how to create my own automation pipeline


The administrator password can be found in the logs of your running Docker image. Just go and check out your console:


test automation pipelines - how to do it


After that, proceed with the basic configuration, which requires personal data forms and the plugins setup page (to be on track with our article, please choose the suggested plugins installation to install all the required plugins).


The power of Docker is that you can install necessary updates into the base image and save the newly updated container as a separate image. Then, you can use it as a base image. To be able to run our tests from the automation framework created above, we need to have Maven installed.The default Jenkins image doesn’t have Maven onboard, so this will be a nice example of how you can extend your current Docker image with a new functionality.


First of all, you need to login into the running dockerized Jenkins instance. For that, you need to know the instance id of you running image. You can grep all running instances with this command (the container id will be in the first column):


docker ps


solid automation testing


We need to login to the Docker image with a root user as we are going to proceed with Maven, which requires some administration access. You can use this command to login inside the Docker instance (6d68498dd7b5 is the container id so it might be different in your case):


docker exec -u root -it 6d68498dd7b5 bash


Now, let’s proceed with the Maven installation inside dockerized Jenkins:


apt-get update
apt-get install maven


your own testing launchpad


delivery pipeline for agile projects


We have installed Maven inside the Docker container. But if you try to stop and start the container, these steps will be lost. In order to save these steps, you need to commit your changes (it’s better to use your own image name for that):


Integrating QA Throughout the Deployment Pipeline


One other step that might be wise to execute right now is to install the HTML Publisher plugin into our Jenkins instance, as it will be required to aggregate test reports in our build plans. Just go into “Manage Jenkins” -> “Manage plugins” -> “Available” -> search for ‘HTML Publisher’.


Congrats! You have installed Jenkins by using the dockerized Jenkins image. Make some updates inside the container and save your own image, which can be reused on any other machine. Now you have the automation framework, a couple of tests and a continuous integration server where you can run them. Now it’s time to learn how!


Build Your Configuration via Jenkinsfile and Jenkins Pipelines


Jenkins UI is really great and simple. Jenkins is clear in how to configure a specific build plan and run your tests right away. But as soon as we have lots of tests, it usually becomes a pain to maintain and take care of dozens and hundreds of build plans. Why?


Sometimes you need to introduce the same change into several build plans (for example to add a certain post-build action). In this case, you need to jump from one configuration to another and make these changes, while remembering where you applied it and where you haven’t yet. Another critical issue is that you do not have any history of the applied changes to your build configurations. You don’t know what has been changed and you are not able to revert the latest changes. Having all these issues in mind, some time ago Jenkins introduced the new way of working with builds - pipelines!


Briefly, Jenkins pipelines is a continuous delivery through code tool, which is usually added under version control tools like Git. It basically means that you can configure, maintain, revert and add build plans the same way you work with a code. At the same time, the Jenkins pipelines workflow has a nice representation that shows you all steps of your build plan in action. This gives you a nice view on build steps with timing and results for each. A file that contains pipelines code with build configuration is called Jenkinsfile.


To add a build plan with Jenkinsfile you need to open Jenkins, click “New item”, come up with a build plan and choose ‘Pipeline’ in the list of build types:


QA Automation within CD Pipeline


I have found this tutorial very useful to get an idea of how to create and configure jenkinfiles. But of course, official documentation is a very valuable resource as well. As for our needs, I came up with a small Jenkinsfile based on our automation frameworks from previous paragraphs. You can use it as an example:


node {
   stage('Git checkout') { // for display purposes
      git 'https://github.com/BushnevYuri/e2e-automation-pipeline.git'
   stage('Smoke') {
        try {
            sh "mvn clean verify -Dtags='type:Smoke'"
        } catch (err) {
        } finally {
            publishHTML (target: [
            reportDir: 'target/site/serenity',
            reportFiles: 'index.html',
            reportName: "Smoke tests report"
   stage('API') {
        try {
            sh "mvn clean verify -Dtags='type:API'"
        } catch (err) {
        } finally {
            publishHTML (target: [
            reportDir: 'target/site/serenity',
            reportFiles: 'index.html',
            reportName: "API tests report"
   stage('UI') {
        try {
            sh "mvn verify -Dtags='type:UI'"
        } catch (err) {
        } finally {
            publishHTML (target: [
            reportDir: 'target/site/serenity',
            reportFiles: 'index.html',
            reportName: "UI tests report"
   stage('Results') {
      junit '**/target/failsafe-reports/*.xml'


In this Jenkinsfile, we have the step where we check out the code from the repository, a few separate steps to run tests based on their level and a result step to aggregate JUnit test results at the end of the whole test suite. If you prefer https authentication to your git or another version control repository, you need to configure your own repository and the access to this repo by using your own ssh or password.


Once you put the mentioned pipelines code into the pipeline definition windows (the biggest window in the build configuration), you can run the build by using the “Build now” button:


Optimizing an Effective CI/CD Pipeline


But what if you need to make some changes to the build config? Should you open and change the script the UI? Of course, the answer is no. This is where the main magic happens. All you need is to place this Jenkinsfile into your repository somewhere near your tests. After that, you can configure build to use the pipeline Jenkinsfile by using the “Pipeline script from SCM” definition. That’s why we have already taken care of that and put the mentioned file into the test project under ‘jenkins-pipelines/E2E-automation-test.groovy’ path. Therefore, you can configure the build plan in a few steps:


How QA can build quality into your software pipeline


You can execute the build again and confirm that it still works fine. At the end of the script, you will see a couple of test reports available on the main build page:


The DevOps pipeline, QA


If you are following our steps, you might notice that we have some failed tests after the execution. This is because of the chrome browser and web driver which you need to install and specify in the configuration file, but this time inside the Docker. If you have completed all the previous steps with us, you should have enough experience to do that, so let’s leave this step as your homework.


It is also worth mentioning there are more advanced approaches for docker file creation and configuration. First, you can create own dockerfile that contains all the installation and configuration steps in plain text file. In this case you don’t need to commit and save docker images. You can always build a new Docker instance based on this file, which is basically just an instruction how to build this image. This topic is quite huge and if you are interested, then you can check out this article which will give you a better picture. Another approach is the Docker Compose tool, which allows you to build instances based on Docker image combinations. However, this is topic for the separate article and official documentation will help you in this case.


As a result, we have tests, test automation framework, dockerized Jenkins, Jenkinsfile and knowledge about how to run everything together. We can say that our QA automation pipeline is ready! But let’s add some sugar on the top…


A Bit of Blue Ocean


Blue Ocean is a relatively new project developed by the Jenkins community team, which improves the experience of using Jenkins pipelines for continuous integration pipeline representation. We all love Jenkins because it is a stable and mature continuous integration server. But you will love it much more as soon as you touch Blue Ocean yourself. Intrigued? Let’s move on then!


First, you need to install required plugins. Go into “Manage Jenkins” -> “Manage plugins” -> “Available” -> search for ‘Blue Ocean’:


The Continuous Integration & Continuous Delivery Pipeline


Once “Blue Ocean” is installed, you will see “Open Blue Ocean” icon in your build plan.


Delivery Pipeline For Agile Projects


Inside the Blue Ocean you will see your newly created automation tests pipeline. On the page, you will find all the results for each completed stage, actions in progress, related logs with the ability to export them and just a simple beauty of this new continuous integration UX!


Building the Testing Pipeline


The new UX looks really great as soon as you have some parallel steps, for example, UI tests, which will be run for different browsers:


What Continuous Delivery Means for Testers, QA Teams and Software developers


This was a last step and now you have a full dockerized QA automation pipeline with automation framework, some tests and Jenkins with its brand new beautiful UX to run them all.


Final Thoughts


I hope this article was useful and gave you useful information for building your own automation tests pipeline, maybe from scratch. Now, if you want to learn more about Serenity, Selenium, Taurus, Jenkins and automation tests in general, there are plenty of tutorials you can find and dive into. Let us know your thoughts and share your own opinion on the chosen stack of technologies!


Click here to subscribe to our newsletter.


Shift left with BlazeMeter. To try out agile load testing, request a demo, or put your URL or JMX file in the box below and your test will start in minutes.

arrowPlease enter a URL with http(s)

Interested in writing for our Blog?Send us a pitch!