Run massively scalable performance tests on web, mobile, and APIs

Request a Demo
Jun. 13th, 2018

Running JMeter with Docker in the Cloud

In the first article in this series of Docker and JMeter, I used the word “revolution” to describe this period of digital transformation of processes and infrastructures. If you’ve followed these blog posts, you now know how to formalize an Apache JMeter™ working stack based on Docker. They are:

 

In the current article, we will be widening the adoption of this revolution by using Docker’s infrastructure to bring our JMeter based application into the Cloud. We will reuse all the procedures described until now not into a proprietary working context (e.g. our working notepad) but into the infrastructure of a Cloud Service Provider.

 

As an easy alternative to the steps described in this blog post, you can run JMeter in the cloud by uploading your JMX file to BlazeMeter and running your test. Try BlazeMeter out here.

 

Docker in the Cloud

 

Which kind of services are offered by a Cloud Service Provider?

  • Providing infrastructure components that would otherwise exist in an on-premise data-center following the IaaS model.
  • Complementing IaaS products with services such as monitoring, security, load-balancing and storage resiliency.

 

The direct implication is that we are no longer responsible for the machine that executes our docked applications. The Docker machines become a software service for our business logic and they can be invoked (and paid) on demand. We can also ignore where this infrastructure is located if we want to (even though when testing in the cloud with BlazeMeter the 56 geo-locations enable us to test web traffic from all over the world).

 

We are still responsible for our business logic consistency (e.g. Java Virtual Machine, Docker environment, etc..) so we need a cloud machine for installing dependencies installation. Each Cloud Service Provider offers a portfolio with preformatted machine images (please note the difference between a machine image and a Docker image). These images can be selected according to hardware characteristics (e.g. CPU, RAM, Disk, etc.) and software applications (e.g. Operating system, Docker installation, etc.).

 

The docker-machine Command

 

Handling Docker machines is possible via the docker-machine command. This command is available in the Docker environment and runs various actions on machines (e.g. create, start, stop, status, etc..). The complete list of possible arguments is long, Here is a screenshot of some of them to get an idea.

 

docker in the cloud

 

How to Use the docker-machine Command

 

To  follow this article it’s necessary to understand the creation command for an additional Docker machine. The base command is

 

Bash

docker-machine create \
	--driver=<machine_driver>
      [optional_customization] \
	<machine_name>

 

This command contains two arguments necessary for the creation process:

  • --driver=<machine_driver> - specific for the machine provider/infrastructure (e.g. Virtualbox, Amazon, Azure, DigitalOcean, etc.).
  • <machine_name> - a pointer to reference the created machine independently by its real location.

 

docker testing in the cloud

 

Without any [optional_customization] argument(s), the machine creation process is based on the driver default configuration, for both the emulated hardware and the installed software. The machine location is also chosen by default. This last point might generate a strange situation that the reader should be aware of. Therefore, if you need low latency and you are located in Australia, maybe it's better to avoid a default server in London.

 

The creation command with docker-machine involves various steps before completion (so take a coffee break). The final step of machine creation is the installation of the Docker environment. Docker commands, which indicate that Docker is installed, should be available in the command line on the created machine. Without the docker command the Cloud machine for this article scope is wasting your time and money.

 

If we need to know which machine is reachable from our environment, we can use the command:

 

docker-machine ls

 

running jmeter in the cloud

 

This command enumerates the handled machine and summarizes information about. An important piece of information is the ACTIVE field, where an asterisk is used to display the destination of all docker commands (e.g. docker run, docker ls, etc), i.e on which machines the command will run on.

 

It’s important to clarify that the local Docker environment can handle many machines. The active machine is the one on which the Docker commands are executed on. Of course, the active machine can be changed and this is the scope of the next command:

 

eval $(docker-machine env <machine_name>)

 

This command configures the environment by changing the focus of the successive docker commands to a specified machine. At any moment we can check the status of the Docker machine with the command:

 

docker-machine status <machine-name>

 

The status command queries the machine status anywhere a machine is located. This is useful for automatic scripts (e.g. a setup procedure can verify the machine status). If we have various Docker machines dedicated to more Docker applications, it’s important to know that a machine can be started and stopped with the commands:

 

docker-machine start <machine-name>
docker-machine stop <machine-name>

 

An important note: these commands do not alter the environment so it’s mandatory to use the env command before continuing with Docker commands, otherwise that wrong machine might receive the next command.

 

Coordinating Multiple Docker Machines

 

Using a single remote Docker machine, which might be located in IaaS infrastructure, isn’t a really new aspect compared to previous articles. Of course, this Docker machine is in a Cloud environment and not Virtualbox, but the procedures described in the previous article are still valid.

 

Therefore, in this article we are planning to:

  • Handle multi-containers applications over many machines
  • Have machines not necessarily located in the same physical local area network
  • Grant each machine communication with others like a virtual private network

 

coordinating multiple docker machines

 

As described in this series of articles, it’s reductive to consider Docker a product. Instead, it is a service platform dedicated to virtualization. In this article we are introducing two Docker facilities necessary for implementing our JMeter-based Cloud application:

  • Docker Swarm
  • Docker Overlay Network

 

Docker Swarm

 

Docker Engines v1.12.0 and onwards allow developers to deploy containers in Swarm mode. Docker Swarm is a clustering and scheduling tool for containers. Swarm mode is natively connected to a Docker Engine that, as already seen in previous articles, provides a virtualization layer between OS resources  and Docker containers/images.

 

An important difference between a single Docker machine and Docker Swarm is that we aren’t setting up containers anymore. By working with Docker Swarm it is possible to virtualize a service, leaving it to Swarm to “cluster and schedule” necessary infrastructure among nodes that compose Swarm.

 

The Docker Swarm node has two roles of a different scope relating to clustering and scheduling procedures:

  • Manager - this node receives the service definition it dispatches the task/container to worker nodes
  • Worker - this node collects and executes the task/container following received directives

 

Important details about Swarm roles:

  • It’s possible to have multiple Manager nodes that in the same swarm
  • A Manager node can execute tasks like a Worker
  • A Worker node can be “tagged” to customize scheduling/clustering tasks

 

We will be focusing on an important feature of Docker Swarm mode:

 

Swarm has scheduling filters that customize container allocation during service setup. Among these filters we will use the constraint filter that works in two steps:

  1. Add an identification tag to a machine during swarm machine creation
  2. Providing a tag to the destination machine’s address to deploy containers during service setup

 

The Docker Overlay Network

 

In the previous article we described the advantage of placing Docker containers in a virtual network to simulate the dedicated network condition (e.g. domain name resolution, private communication). This can be used only when the network connects containers located into same docker machine.

 

In this article we propose a Docker based application composed of many containers. These containers are divided among many Docker machines. Docker machines are not localized in same local network, which complicates containers messages exchange. Moreover, these containers require domain name resolution and private communication (like in the previous article). The solution for these requirements continues to be based on the docker network command, with a single difference: we need to create an “overlay” network type. The overlay network driver creates a distributed network among multiple Docker machines.

 

A docker overlay network can be logically shared among many docker machines in a swarm. The overlay network creation must be invoked from a swarm manager. The command is:

 

docker network create -d overlay --attachable --subnet=<sub_net> <network_name>

 

The arguments:

  • -d overlay - specifies the type of docker network to be created
  • --attachable - provides the ability to communicate without the need to set up routing on the individual Docker daemon hosts. Makes sense only for docker machine that operates in a swarm.
  • --subnet - specifies the subnet using the CIDR format
  • The last mandatory argument is the network name. This name is used during swarm service creation to create containers “attached” to the network.

 

After creating the overlay network, only the Manager machine recognizes this network. It's like the Worker machines do not see this network (try out the docker network ls command output on the Workers). It's normal and not an error. When the Manager allocates a container attached to the overlay network, the involved Worker becomes the owner of the new container, plus it sees the overlay network.

 

Additional Resource Handling

 

As usual in a cloud context, our example application is not closed to the rest of the world and requires dependencies to work correctly. In the context of our JMeter example we are speaking about:

  • providing additional contents required for test execution (e.g. plugins jar libraries)
  • providing a jmx script with a test to be executed in distributed mode
  • fetching a test report after the distributed test conclusion

 

Here is another difference compared to the previous articles that show a single docker machine. With a single docker machine, sharing contents with containers requires creating/mapping a docker volume from the local file system to the containers. Now, this direct way isn’t possible because docker machines, and its containers, are “far away”. So, Docker provides us a with a solution with the command docker-machine scp.

 

docker-machine scp /local/content DOCKER-MACHINE-NAME:/remote/path/

 

This command is quite similar to the Linux command Secure Copy (scp) with the following differences:

  • remember to use in combination with docker-machine command
  • a user/password is not necessary for remote machine authentication

 

With this command it is possible to:

  • provide additional jars to Docker machines where JMeter containers are executed (in both Manager and Worker nodes)
  • provide a jmx script file where aJMeter master container is executed (in Manager node)
  • fetch report files where the JMeter master container (in Manager node) is executed

 

docker machines for load testing

Provide additional jars to Docker machines

 

load testing in the cloud

Provide a jmx script file to a Manager

 

load testing through docker

Fetch report files from a Manager

 

A Multi-machine JMeter Test in the Cloud

 

So, we have a Cloud based application composed of docker machines/containers in various data centers provided by our Cloud Service Provider. We chose DigitalOcean because it provides excellent service quality with good prices. Moreover the simplicity of setting up the  API infrastructure is an important part of this decision. To try DigitalOcean services it’s necessary to set up an account (see this link for a $10 trial account).

 

The proposed example is in Bash and assumes that the execution machine has a Docker environment installed (also Docker Toolbox is fine) and a DigitalOcean account with some credit (for one hour of test our scenario costs 10 cents).  As usual, the complete source code described here is available on github.

 

Customizing the DigitalOcean Droplet

 

The first step in our script is to define the location of the Docker machines. We take a base machine image (called a Droplet in DigitalOcean). We will customize it with a location identified by a string (to_do_creation). In our example, the swarm manager will be created in the data center “nyc3” in New York. Moreover we will define three workers placed in “ams2” (Amsterdam), “fra1” (Frankfurt) and “lon1” (London).

 

As already described, the docker-machine create command depends on the Digital Ocean driver to create the machine as well as optional machine customization. Luckily, Digital Ocean has its driver well documented. The driver page has a list of all the arguments for customizing a machine.

 

One of the required arguments is called “digitalocean access token”. The token is the way to identify a virtual machine with a user so the user can be billed. To continue with the example it’s necessary to generate the token with the configured account with credit and follow this procedure.

 

Bash

# This variable describe manager location
MANAGER_REGION="nyc3"

# This array describes workers location
declare -a WORKER_REGIONS=("ams2" "fra1" "lon1")


# DigitalOcean Access Token
export DO_TOKEN="your_digital_ocean_token_string"

function to_do_creation(){
echo "--driver=digitalocean --digitalocean-access-token=${DO_TOKEN} --digitalocean-size=1gb --digitalocean-region=${1} --digitalocean-private-networking=true --digitalocean-image=ubuntu-16-04-x64"
}

 

The “to_do_creation” procedure is quite simple. It takes an input string that describes a data center location (e.g. nyc3) and returns a string filled with the custom configuration with location and token ready to be used with docker-machine create command.

 

Creating Machines and Infrastructure Globally

 

Now we will proceed with machine creation. From this point on every action has an hourly price so an invoice will be generated (on an hourly basis).

 

Bash

# Manager machine name
MANAGER_ID=manager-${MANAGER_REGION}


# Create manager machine
docker-machine create \
    $(to_do_creation $MANAGER_REGION) \
    --engine-label role=$MANAGER_ID \
    $MANAGER_ID


# This command extract real ip address
MANAGER_IP=`docker-machine ip $MANAGER_ID`

# Init docker swarm manager on machine
docker-machine ssh $MANAGER_ID "docker swarm init --advertise-addr ${MANAGER_IP}"


# Extract a token necessary to attach workers to swarm
WORKER_TOKEN=`docker-machine ssh $MANAGER_ID docker swarm join-token worker | grep token | awk '{ print $5 }'`


 

In this section of the code we are creating a Manager machine name and proceeding to use it during machine creation. This is a new argument and it requires an explanation. The “--engine-label” provides the tagging mechanism that we have describe early in article. The created machine has an attribute called “node” that contains a custom value.

 

After creation we use the command docker-machine ip to extract a real machine IP address. With the manager machine ip address we proceed with two commands related to Swarm, in two places:

  • before we initialize the manager node as the Swarm manager
  • after we extract the worker token necessary for adding workers to the Swarm

 

Bash

# this array holds worker machine names
declare -a WORKER_IDS=()

# Iterate over worker regions
for region in "${WORKER_REGIONS[@]}"
do
    # Machine name
    worker_machine_name=$(echo worker-${region})
    # Create worker machine
    docker-machine create \
    $(to_do_creation $region) \
    --engine-label role=$worker_machine_name \
    $worker_machine_name
    
    WORKER_IDS+=($worker_machine_name)

    # Join to Swarm as worker
    docker-machine ssh ${worker_machine_name} \
    "docker swarm join --token ${WORKER_TOKEN} ${MANAGER_IP}:2377"
done

 

This code section creates worker machines. There is a main loop over regions and it creates a new machine for each region. Here is a step by step description of the code for a single region:

  • already known Bash procedure for machine configuration
  • tagging each machine with a customized value based on the region (--engine-label argument)
  • finally, add a worker machine to the swarm by using the worker token and the manager ip address (docker swarm join command)

 

Bash

# Overlay network information
SUB_NET="172.23.0.0/16"
TEST_NET=my-overlay

# Switch swarm manager machine
eval $(docker-machine env $MANAGER_ID)

# From swarm manager overlay network creation
docker network create \
  -d overlay \
  --attachable \
  --subnet=$SUB_NET $TEST_NET

 

As a final step of the infrastructure setup, we create the overlay network with minimal configuration (just a name and ip range). Note that we have switched to the manager machine.

 

[You can also skip all of these steps by just uploading your JMX file to BlazeMeter and choosing the locations you want to run your test from in and easy to use GUI].

 

Creating and Executing Containers

 

Now that we have all the machines configured around the world and to the necessary network infrastructure, we are ready to interconnect our logical services. So now we will proceed with the services and containers creation.

 

Bash

# this array is necessary to hold containers name
declare -a JMETER_CONTAINERS=()

# for each worker machine
for id in "${WORKER_IDS[@]}"
do
   # for three times we create JMeter slave service 
   # using engine label for scheduling
   for index in $(seq -f "%02g" 1 3)
   do
   	jmeter_container_name=$(echo ${id}_${index}_jmeter)
    	docker service create \
      	--name $worker_container_name \
      	--constraint "engine.labels.role==$id" \
      	--network $TEST_NET \
      	vmarrazzo/jmeter \
      	-s -n \
      	-Jclient.rmi.localport=7000 -Jserver.rmi.localport=60000 \
      	-Jserver.rmi.ssl.disable=true
   	# save container name
      JMETER_CONTAINERS+=($jmeter_container_name)
    done
done

 

In the code above we loop over worker machines. For each of them we create three JMeter slave services. The base structure of the command docker service create is quite similar to the command used in the distributed testing article, with the following differences:

  • using a constraint to schedule a machine on the “role” tag
  • assigning a name to each container/service
  • attaching services to the overlay network

 

This set the created service’s location in specific data centers around the world, and we have a container name that the overlay network can resolve to real ip addresses. At the end of the loops we have a total of nine JMeter slave instance waiting for our distributed test. (Nine - we looped over three regions, and for each region we looped three times to create JMeter slave containers).

 

This concludes the main script, which set up the Docker Swarm that is composed of machines deployed in various world locations. The main goal of the swarm is to carry many JMeter slave instances waiting for test procedures on the docker overlay network.

 

By sending commands to the Manager machine we can validate our logical infrastructure with Docker commands.

 

jmeter in docker load testing

Docker Swam Machine

 

docker sward, cloud, jmeter

Docker Swarm Running Services

 

This last presented code acts on the Manager machine. It creates containers and not swarm services by bypassing the Swarm clustering logic. The only precondition is to upload a jmx script on the manager machine.

 

Bash

echo "^^^DEBUG^^^ Create client and execute NMAP on overlay network"
docker run \
    --network my-overlay \
    networkstatic/nmap \
    -n 172.23.0.0/24 -p 1099,60000

timestamp=$(date +%Y%m%d_%H%M%S)
jmeter=/mnt/jmeter

declare -a JMETER_CONTAINERS=("worker-ams3_01_jmeter" "worker-ams3_02_jmeter" "worker-ams3_03_jmeter" "worker-fra1_01_jmeter" "worker-fra1_02_jmeter" "worker-fra1_03_jmeter" "worker-lon1_01_jmeter" "worker-lon1_02_jmeter" "worker-lon1_03_jmeter")

echo "##^## Create client and execute test"
docker run \
      --network my-overlay \
      -v /mnt/jmeter:/mnt/jmeter \
      vmarrazzo/jmeter \
      -n \
 	-Jclient.rmi.localport=7000 \
      -Jserver.rmi.ssl.disable=true \
 	-R $(echo $(printf ",%s" "${JMETER_CONTAINERS[@]}") | cut -c 2-) \
 	-t ${jmeter_path}/test.jmx \
 	-l ${jmeter_path}/client/result_${timestamp}.jtl \
 	-j ${jmeter_path}/client/jmeter_${timestamp}.log

 

The first executed container is only for this article’s demonstration purposes, showing a subnet with nine virtualized machines. This container is based on the nmap command and it discovers all the open machines in the overlay subnet with TCP ports used by the JMeter distributed protocol. The outcome of Nmap container is a standard output like:

 

Console output

Starting Nmap 7.01 ( https://nmap.org ) at 2018-05-21 19:53 UTC
Nmap scan report for 172.23.0.1
Host is up (0.000066s latency).
PORT      STATE  SERVICE
1099/tcp  closed rmiregistry
60000/tcp closed unknown
MAC Address: 6A:F6:B8:D5:C0:CF (Unknown)

Nmap scan report for 172.23.0.7
Host is up (0.019s latency).
PORT      STATE SERVICE
1099/tcp  open  rmiregistry
60000/tcp open  unknown
MAC Address: 02:42:AC:17:00:07 (Unknown)

Nmap scan report for 172.23.0.9
Host is up (0.019s latency).
PORT      STATE SERVICE
1099/tcp  open  rmiregistry
60000/tcp open  unknown
MAC Address: 02:42:AC:17:00:09 (Unknown)

Nmap scan report for 172.23.0.11
Host is up (-0.058s latency).
PORT      STATE SERVICE
1099/tcp  open  rmiregistry
60000/tcp open  unknown
MAC Address: 02:42:AC:17:00:0B (Unknown)

Nmap scan report for 172.23.0.13
Host is up (-0.057s latency).
PORT      STATE SERVICE
1099/tcp  open  rmiregistry
60000/tcp open  unknown
MAC Address: 02:42:AC:17:00:0D (Unknown)

...

 

In the above standard output the Nmap container has discovered various virtual machines on subnet. Counting the ones with the open socket to accept JMeter distributed protocol we reach nine, just like the JMeter slave containers allocated in the swarm.

 

The second container executed on manager is the JMeter master that invokes the distributed test execution using the list of passed JMeter slave containers.

 

jmeter on containers

 

The picture above shows the recognized remote instances and the start of the remote test. The single JMeter instance is configured to generate a few dozen samples a second, but the aggregate sampling rate reaches over 1000 msg/s. By downloading the final jtl report file from the manager it is possible to visualize, through the JMeter GUI, the number magnitude.

 

remote jmeter load testing

 

That’s it. In this article we have taken a well known test scenario based on the JMeter distributed testing mode and exported in into Docker and Cloud as infrastructural pillars.

 

The result of this is a flexible testing environment with the scope to aggregate loads and results. This environment offers advantages for aggregating consistent loads by using virtual machines with economic savings (the single virtual machine used cost less than 1c/h).

 

Moreover, this globally spread environment provides the possibility to test an application by generating traffic coming from another country or continent.

 

Running JMeter in the Cloud with BlazeMeter

 

A simpler way to achieve everything described in this article is to take your already created JMX file, uploading it to BlazeMeter, configuring the geo-locations and starting the test.

 

jmeter in the cloud with blazemeter

 

To try out BlazeMeter, put your URL in the box below and your test will start in minutes. Or, request a demo.

     
arrow Please enter a valid URL

Interested in writing for our Blog? Send us a pitch!