Apr. 18th, 2016

How to Run JMeter Tests with TeamCity for Continuous Integration

We recently covered how to run JMeter in a Continuous Integration (CI) environment with Jenkins and with Bamboo.  Today’s article covers the process of running a JMeter test using TeamCity.

 

Continuous Integration and Continuous Delivery are vital for developing and producing complex applications, technology and automation help to move faster and more efficiently. Being an important part of the software development lifecycle, load testing needs to be included into the Continuous Integration process, bringing confidence to software performance and capacity.

 

TeamCity is a build and continuous integration server from JetBrains. While it is a commercial product, it can be used for free if you don’t need more than 20 build configurations and 3 build agents. Free license is granted to open-source projects as well. TeamCity was released 10 years ago and now it’s being used by thousands of software development companies. It has awesome features like running a build prior to VCS commit, integrated code coverage reports and code inspections, the capability of running multiple builds and/or tests on different platforms, and more.

 

Configuring TeamCity for Running JMeter Tests

 

TeamCity is capable of running a JMeter test out of the box, however, you will have to analyze the results manually. This doesn’t make a lot of sense given the goal to have load tests running in an unattended and automated manner.

 

There is a JMeter plugin for TeamCity, which has 2 main functions:

 

  1. Performance Statistics - a collection of performance metrics and visualizing them as charts on build results page
  2. PerfMon - visualizing a build agent and application under test server health

- A collection of the main OS health metrics from a build agent for the test duration

- A collection of the main OS health metrics from a remote machine

 

JMeter Plugin for TeamCity Installation

 

The JMeter plugin for TeamCity can be installed in a few easy steps:

 

  1. Download the latest version of the plugin from https://teamcity.jetbrains.com/repository/download/TeamCityPluginsByJetBrains_JMeterPlugin_Build/.lastSuccessful/jmeter.zip
  2. Drop “jmeter.zip” file to the TeamCity plugins folder - by default it is:
    • C:\ProgramData\JetBrains\TeamCity\plugins - for MS Windows Family
    • .BuildServer/plugins/ directory under the home folder of the user which is running the TeamCity server - for Linux, Unix and derivatives
  3. Restart TeamCity

 

If everything goes as expected, the JMeter plugin should be listed among others under TeamCity -> Administration -> Plugins page:

 

JMeter plugin in TeamCity

 

Want to learn more about combining performance testing and continuous integration? Watch our on-demand webcast on How to Build Testing Into Your CI Pipeline, featuring a special guest DevOps Engineer from MIT, who reveals exactly how he added testing to the CI process and eliminated performance surprises.

 

Running a JMeter Test with TeamCity

 

Configuring the JMeter Build Step

 

TeamCity supports a wide range of build steps including, but not limited to:

 

- Running an arbitrary Command Line (either a custom shell script or an executable one with parameters)
- Invoking an Apache Ant task
- Running an Apache Maven project

 

Usually it is better to use your main project build system. You may know that a JMeter test can be executed via the aforementioned methods, if not - you’re welcome to check out Five Ways To Launch a JMeter Test without Using the JMeter GUI. If your application is being built using Maven, go for Maven, and if it uses Ant, run the JMeter test using JMeter Ant Task, etc.

 

In this article, I’ll stick to the command-line non-GUI mode, as it is the most common approach of running a JMeter test.

 

In order to trigger a JMeter test from TeamCity, you need to add a Build Step which will call JMeter and provide your test script as a parameter. The minimal working configuration will look like:

 

- Runner Type: Command Line
- Run: Executable with parameters
- Command executable: path to the JMeter startup script, i.e. C:\jmeter\bin\jmeter.bat
- Command parameters: standard JMeter arguments like you run the test from the command line, i.e. -n -t c:\jmeter\test.jmx -l test.jtl
- You can also specify an arbitrary “Step Name” especially if you plan to use the Remote PerfMon feature.

 

JMeter TeamCity Build Step

 

Important! The JMeter TeamCity plugin expects the results file to be in a specific non-default format, the most important requirements being:

 

  1. The metrics need to be separated by TAB characters
  2. The first line needs to be column headers

 

The relevant JMeter properties are:

 

jmeter.save.saveservice.default_delimiter=\t
jmeter.save.saveservice.print_field_names=true

 

So in order to be able to use the JMeter TeamCity plugin, add the above 2 lines to the user.properties file (it is located under the “bin” folder of your JMeter installation).

 

Configuring the JMeter TeamCity Plugin Build Feature

 

The next step is to “point” the JMeter TeamCity plugin to the .jtl results file so you could see performance statistics along with the build results. In order to do so, go to the Build Features page in TeamCity and add the “Performance Metrics Calculation” feature.

 

The minimal working configuration should look like:

 

Configuring the JMeter TeamCity Plugin Build Feature

 

Now you should be ready to run your test. You should see an extra tab on the build results page - “Performance Statistics”.

 

Performance Statistics JMeter TeamCity

 

This is where you will be able to see the performance metrics and trends. Remember that you should have at least 2 builds in order to see the charts. 

 

Performance metrics and trends charts

 

It is also possible to set reference values to check the metrics against. This way you will be able to conditionally fail the build in the case of exceeding a threshold, such as if the average response time is above the SLA or the non-200 HTTP response codes number is higher than 1% (these are just a few examples).

 

Monitoring the Application Under Test Health Metrics

 

It is also possible to plot some health metrics from the server where your application lives in the TeamCity build results page:

 

  1. Download the PerfMon Server Agent and install it on the server your applications runs on
  2. Make sure that TeamCity is able to connect the server using a default port of 4444
  3. Configure the TeamCity JMeter plugin to collect metrics from the remote server. To do so, add the “Remote Performance Monitoring” Build Feature.

 

Remote Performance Monitoring

 

Replace “localhost” with hostname or the IP address of the machine where your application is deployed.

 

With the “Remote Performance Monitoring” build feature enabled, you will be able to see the application under the test server health metrics, along with main performance KPIs on the build results page. 

 

JMeter TeamCity main performance KPIs on the build results page

 

See How to Monitor Your Server Health & Performance During a JMeter Load Test article for more information on the application under test health monitoring concept and comprehensive installation and usage instructions.

 

Now you should be ready to add JMeter test to your TeamCity build process. If anything remains unclear, feel free to ask in the comments sections below.

 

BlazeMeter users can integrate their cloud tests with local TeamCity deployments using the BlazeMeter plugin To TeamCity:

 

  1. BlazeMeter Plugin To TeamCity
  2. Load Testing with TeamCity and BlazeMeter

 

Interested in learning more? Watch our on-demand webcast on How to Build Testing Into Your CI Pipeline, featuring a special guest DevOps Engineer from MIT.

 

Interested in writing for our Blog? Send us a pitch!