Jul. 14th, 2014

How to Include Load Testing in your Continuous Integration Environment

It still hasn’t been around that long, but Continuous Integration (CI) is already considered a ‘Best Practice’ and is a key element of agile development methodologies.

 

What are Continuous Integration (CI) Processes?

 

If you’re following CI processes, you need to test every change made to your codebase as early as possible. Using tools like Jenkins, TeamCity and Bamboo, developers continuously integrate new or changed code into a shared repository – and verify it with an automated build.  Typical CI process steps include:

 

-       Checking out the source code

-       Running the unit tests

-       Running other build steps

-       Deploying to the QA environment

-       Deploying to production

 

These tests are run continuously, enabling teams to detect errors quickly and locate them easily. CI tools automate the steps  – so it’s easy to run these tests every day or even every hour. And, if there’s a failed unit test, or any other build step, your CI tool won’t allow you to continue.

 

Why Include Load Testing in a CI Environment?

 

Let’s say the build steps pass. Great - you’ve verified that the new code works! But then you discover that the code has an adverse effect on the performance of your application.  Suddenly your site falls dramatically short of meeting your performance metrics.

 

If you’re running CI processes without running load tests, you’re only looking at one part of the equation - and you’re going to get an incomplete answer. If your app is going to be deployed to QA or production, it’s important to integrate load testing into your processes. Or, at the very least, take your tests one step beyond the functional testing on a daily basis so you can continually get a good indication of your apps’ performance.

 

Want to learn more about combining performance testing and continuous integration? Watch our webinar recording on How to Build Testing Into Your CI Pipeline, featuring a special guest DevOps Engineer from MIT, who reveals exactly how he added testing to the CI process and eliminated performance surprises.

 

The Challenges of a Load Testing Environment

 

A typical load test involves a lot of setup, including:

 

-       Keeping dedicated systems to generate the load

-       Making sure they are available the moment the test runs

-       Adding monitoring systems to identify bottlenecks

-       Generating reports

 

But, unlike your unit tests, you probably don’t want to run load tests every hour or every day. So you’re spending a lot of money on resources (load servers, maintenance costs etc.) and they are standing idle most of the time. On top of, report generation is a huge project as you’ll need to understand how to accurately and efficiently aggregate numerous data points into your CI environment.  

 

If you do want to run load tests whenever you need without wasting time and resources, it’s worth moving over to the cloud. Your CI tool will then be able to run tests for you, validating the performance of the app under load. If it doesn’t meet the performance standards that you set, the build will break.  

 

How to Run Load Tests in the Cloud

 

The easiest way to achieve this is by integrating BlazeMeter’s load testing cloud into your CI tool. To show you how to do this, I’m going to take Jenkins as an example - but it can be done with other CI tools.

 

1. Login or create a free account with BlazeMeter (30 seconds)

2. Create a test with BlazeMeter (2 mins)

3. Download and install the BlazeMeter plugin in Jenkins (5 mins)

4. Configure the plugin to run the test with your own thresholds (30 seconds)

5. Run the build

 

For other jobs, the most you’ll have to do is complete step number 4.

 

More About Setting Your Own Performance Testing Thresholds

 

Those of you familiar with Jenkins will know that you can determine your own thresholds as to whether a test is considered a pass, unstable, or a fail.  You’re notified as soon as a test fails or is considered unstable. All passes go straight through without problems.

 

The BlazeMeter plugin gives you exactly the same options for your load test. You can set thresholds for the following parameters:

 

1.     Error Rate (by %)

2.     Response Times (in milliseconds)

 

This means that you can choose what is acceptable for your test. If it’s doesn’t pass the thresholds that you’ve set, the build will fail or be declared unstable.

 

There are also a number of advanced options that you can choose. For example: you can shorten the maximum test duration or you can change the JMX running during the test. You can find out more by reading our step-by-step guide on integrating BlazeMeter with Jenkins.

 

That’s it! Whenever you want to run a load test, you just need to check your BlazeMeter plugin on your Jenkins Console. Load tests run in minutes rather than days and you’ll get reports in an easy, comprehensive format – no need to spend hours analyzing data and messing about with excel spreadsheets.

 

Find out more

 

Want to learn more about combining performance testing and continuous integration? Watch our on-demand webcast recording on How to Build Testing Into Your CI Pipeline, featuring a special guest DevOps Engineer from MIT, who reveal exactly how he added testing to the CI process and eliminated performance surprises.

 

Load testing with Jenkins

Load testing with TeamCity

 

Interested in writing for our Blog? Send us a pitch!