Continuous integration is now a vital part of the modern software development life cycle. While this blog has covered its importance and benefits before, a short review is beneficial for our look at open source Continuous Integration servers.
When developers create new features or fix defects, it’s critical that new code changes won’t harm or have a negative impact on previously existing software elements. Key objectives to keep in mind include:
The whole process of software development starting from a single commit, compilation, running tests, etc. and ending up with packaging and delivery needs to be automated to make the process as efficient as possible
Performance testing should be an inherent part of the continuous delivery process starting from early development stages
All associated actions need to be automated
If everything goes well, the resulting software package should be packaged for distribution or even deployed into production
If there is an error, the relevant team members should be notified
As software development and testing techniques are becoming more and more complex, there is a need in a special software to orchestrate the aforementioned processes, as well as to ensure that the steps required for software delivery are being executed in the correct order, nothing is missing and failures are correctly handled.
That is what a Continuous Integration Server is. In this post, we will cover the most prominent free and open source solutions and review them from the perspective of integrating performance testing into the continuous integration and delivery process.
Jenkins is the leading open source continuous integration tool. For the last few years it holds the top position in DevOps toolchains due to being free, open source and modular (although its baseline functionality is very limited, there are more than 1,000 plugins which extend Jenkins capabilities and provide extra integrations).
Jenkins is a Java-based tool so you will need Java Runtime Environment to run it. All the extensions and plugins are also to be developed in Java. Given the underlying Java Runtime, Jenkins can be installed on any operating system where Java runs.
Out of the box (or with default plugins, if you want) Jenkins supports the following:
Version Control Systems:
Set status for Github commit
You need to remember that Jenkins functionality is not limited to the above features, it can be extended via Jenkins Plugins even for very unusual scenarios. For instance you can even kick off a BlazeMeter test and display results in Jenkins dashboard using the BlazeMeter Jenkins Plugin.
Installation of Jenkins and configuration for running JMeter tests is covered in Continuous Integration 101: How to Run JMeter With Jenkins, which contains comprehensive information on getting Jenkins and JMeter up and running.
Jenkins 2.0, the first major release of the software in years, was released in April 2016.
Here is an example of the Jenkins dashboard. In the bottom left corner, you can see that a build is being executed at the moment and as well as the history of recent builds.
Buildbot is another software development continuous integration tool which is trusted by the Mozilla Foundation for bringing excellence into their products, and is now used at Mozilla, Chromium, WebKit and many other projects.
Buildbot started as a lightweight alternative to Mozilla Tinderbox and currently it’s a fully functional continuous development and build server supporting a lot of integrations, distributed builds, custom (and often unique) build steps, notifications via multiple channels and many more.
Buildbot is a Python-based tool, and as such, it runs everywhere Python runs. You can deploy it onto any POSIX-compliant operating system. Moreover, Buildbot config files are basically Python scripts, meaning that the configuration is version-control friendly and you have all the power of the Python language while defining your build plan. It means extreme and unbeatable flexibility and capabilities.
As for July 2016, Buildbot supports out of the box:
Version Control Systems:
Configure (autoconf-style projects)
Compile (C projects)
VC++ (Microsoft compilers)
Test (analogue of make test)
TreeSize - check repo size
Perl Module Test
HTTP Status Push
Gerrit Status Push
Github Status Push
Stash Status Push
Buildbot installation and configuration is a separate large topic as the instructions may differ depending on each operating system and architecture, so it is better to refer to Buildbot documentation. However if you’re looking for quick and dirty way to get it running and see if it fits your needs, here are minimal instructions:
factory.addStep(steps.ShellCommand(command=["c:/jmeter/bin/jmeter.bat -n -t c:/buildbot/Test.jmx -l C:/buildbot/test.jtl"]))
to your master.cfg file.
Here is the Buildbot dashboard with one build currently running along with the history of several previous test executions:
CruiseControl is the most mature continuous integration server out of today’s scope and probably one of the oldest continuous integration solutions in existence. The first release was 15+ years ago. CruiseControl was originally developed as a framework to orchestrate build, testing and other software development lifecycle for one particular internal project at ThoughtWorks. Later, CruiseControl became a standalone continuous integration server.
The latest CruiseControl release provides the following integrations out-of-the-box:
Version Control Systems:
CC-tray (desktop application)
CruiseControl configuration is XML-based. Check out the configuration reference for all possible settings. There is also a Java-based GUI frontend, CC-config, and perhaps it would be easier to use it instead of manually editing the XML config file.
Here is an example of the minimal configuration for JMeter execution in the command-line non-GUI mode is:
<project name="Performance Test">
<exec Command="c:\jmeter\bin\jmeter.bat" Args="-n -t c:\jmeter\extras\Test.jmx -l c:\jmeter\extras\Test.jtl" />
For your reference, below is the CruiseControl project dashboard containing a history of performance tests execution and statistics on one of the builds:
If you want to integrate results into the “Test” dashboard tab, you need to merge your result files with CruiseControl log file, through the following:
<project name="Performance Test">
<!--execute performance test -->
<!-- Add performance test results to CruiseControl -->
<merge dir="c:\PerformanceTestResults\" Pattern="*.xml" />
Integrating Performance Testing Tools
Running a performance test as a part of the build is fine, however, it doesn’t resolve the main goal - automated results analysis. The whole idea of the continuous integration process is making software build and associated processes absolutely unattended. As a result, it is important to have some form of mechanism to fail the build when the performance test produces an error or the average response time exceeds a certain threshold. In some cases it can be resolved via plugins like the Jenkins Performance Plugin. But in more advanced scenarios, there might be not enough time or budget to develop the plugin or to change the load testing script.
That’s why I would recommend running your performance tests via Taurus - an automation-friendly framework for continuous testing which supports a lot of underlying load generation tools including but not limited to JMeter, Gatling, Grinder, Tsung, Locust, and more.
The key Taurus features from the continuous integration perspective are:
You don’t need to change your existing tests scripts, Taurus can “consume” everything
You can easily define fail criteria using a flexible Pass/Fail Service service. The most important bit is that Taurus will return non-zero exit code so that any continuous integration server will treat the Taurus execution step as failed
Taurus can produce results in JUnit format via the JUnit XML Reporter. The absolute majority of continuous integration servers can “understand” the JUnit output, and as a result can display and/or plot results on the build dashboard
Hopefully, this guide sheds some light on the available open source CI servers, as well as the ways of adding performance tests to your continuous integration workflow.
If you have any questions, feel free to use the comments section below, any form of feedback will be appreciated.
Want to Know More?
Here are a few further steps that will help you to further examine the topic of performance testing and continuous integration:
Register for our upcoming webinar Automated Performance Tests in Jenkins CI Environments
You might also find these useful:
Interested in writing for our Blog? Send us a pitch!