Dec. 4th, 2014

The DNS Cache Manager: The Right Way To Test Load Balanced Apps

There’s no doubt that high availability and scalability are vital for today’s networking-based applications. Just think of the millions of people using services like Google or Facebook at this very moment!

 

To make sure their sites are seamless and robust, most companies turn to Network Load Balancing as a solution. This helps them ensure that their service will remain operable - even when there are unexpected circumstances or one instance failures.

 

There are many different deployment scenarios and high availability configurations. But most follow these basic ideas/requirements:

 

  1. The end users sees the whole system as a single endpoint (through the Load Balancer)

  2. The system consists of at least two instances

  3. The Load Balancer orchestrates end user requests so they are distributed between the system instances

  4. When one instance becomes unavailable (i.e. for maintenance, replacing or upgrading hardware or a power outage), the Load Balancer stops sending requests to it and the remaining instances serve the end users. As soon as the instance is operational again, the Load Balancer adds it back into its list.

 

The Right Way to Use JMeter With Your Load Balanced Systems

 

In today’s post, I’m going to be focusing on a scenario in which the load balancer has more than one IP address. I’ll show you how to ensure that JMeter is correctly sending requests to all the load balancer’s entry points.
 

It’s important to have multiple load balancer endpoints to ensure that the load balancer itself does not become a single point of failure - and that if one entry point stops serving requests, the remaining ones will continue to respond.


 

This diagram shows the network deployment which will be used in my demo.

 

 

 

 


Here’s a breakdown of each network element:

 

  • Apache JMeter. This is my workstation and requests will be sent from it.

  • Load Balancer. Amazon ELB has been configured to distribute requests across two “Application Under Test” instances. To make it easier to read and understand, I’ve used Load Balancer network details like:

 

  • DNS Name: example.com

  • LBIP1 and LBIP2 (Load Balancer IP address 1 and 2)

 

Different and real DNS names and IPs will be used for the real demo

 

  • Application Under Test Instances 1 and 2. A simple Apache HTTP Server on the Amazon EC2 instance will return the static HTML page with one sentence. I won’t be including them in today’s demo as I want to focus solely on Apache JMeter and the Load Balancer.


 

A DNS Overview

 

Already familiar with DNS?  Skip this part and move straight to the Amazon ELB Load Test Demo.

 

For everyone else, here’s a quick overview:

 

DNS, also known as the “Domain Name System”, is a naming system for computers which converts human-readable names like “example.com” to IP addresses like: “93.184.216.119”. For example: if you open “blazemeter.com” in your browser, it does a DNS lookup of the associated IP address. As such, the actual request is made from your computer’s IP address to the BlazeMeter server IP address.

 

Want to find out the IP address for a host? Use these command-line tools:

 

  1. Microsoft Windows

    • nslookup

    • ping

  2. MacOSX and Linux

    • host

    • dig

    • nslookup

    • ping

 

You can also use online services like DNSstuff or http://ping.eu/nslookup

 

So, this is how it will look when you’re determining the IP address of “example.com”:

 

C:\>nslookup example.com

Server:  127.0.1.1

Address:  127.0.1.1#53

 

Non-authoritative answer:

Name:    example.com

Addresses:  2606:2800:220:6d:26bf:1447:1097:aa7

         93.184.216.119

 

The “example.com” domain goes to two IP addresses:

  • IPv4 address of 93.184.216.119

  • IPv6 address of 2606:2800:220:6d:26bf:1447:1097:aa7

 

Now let’s try “Facebook.com”

 

C:\>nslookup facebook.com

Server:  127.0.1.1

Address:  127.0.1.1#53

 

Non-authoritative answer:

Name:    facebook.com

Addresses:  2a03:2880:2130:cf05:face:b00c:0:1

         173.252.120.6


 

C:\>

 

“facebook.com” also goes to two IP addresses:

  • IPv4: 173.252.120.6

  • IPv6: 2a03:2880:2130:cf05:face:b00c:0:1 (have you noticed this “face:b00c” hilarious bit?)




 

Now let’s try Google:

 

C:\>nslookup google.com

Server:  127.0.1.1

Address:  127.0.1.1#53

 

Non-authoritative answer:

Name:    google.com

Addresses:  2a00:1450:4009:801::1006

         74.125.230.98

         74.125.230.99

         74.125.230.100

         74.125.230.101

         74.125.230.102

         74.125.230.103

         74.125.230.104

         74.125.230.105

         74.125.230.110

         74.125.230.96

         74.125.230.97


 

C:\>

 

Ok, so here it’s a bit different. The “google.com” host goes to one IPv6 address and eleven IPv4 addresses. Although, I believe that it could be even more than this! Try running this command from your location - I think you’ll get different results. This is how Google is able to serve millions of concurrent users.  

 

Given that we still live in IPv4 world, let’s stick to the scenario where one DNS hostname goes to more than one IPv4 address - like the jmeter.apache.org site here:

 

~$ dig +short jmeter.apache.org

192.87.106.229

54.172.167.43


 

The Amazon ELB Load Test Demo

 

Default Settings

 

In this demo, I’m going to test Amazon ELB - which I have created and configured. I can’t test any of the examples above because it’s forbidden to test big websites like Google and Facebook without explicit authorization.

 

 

 

 

 

This has the DNS hostname dns-cache-manager-1494470926.eu-central-1.elb.amazonaws.com  - which goes to two IPv4 addresses

 

~$ host dns-cache-manager-1494470926.eu-central-1.elb.amazonaws.com

dns-cache-manager-1494470926.eu-central-1.elb.amazonaws.com has the address 54.93.140.145

dns-cache-manager-1494470926.eu-central-1.elb.amazonaws.com has the address 54.93.153.159


 

Let’s take a look at what happens if we send dozens of requests to the hostname:  dns-cache-manager-1494470926.eu-central-1.elb.amazonaws.com.

 

Let’s create a simple JMeter Test Plan:

 

  • Test Plan

  • Thread Group (Threads: 5, Loops: 5)

    • HTTP Request (Server Name: dns-cache-manager-1494470926.eu-central-1.elb.amazonaws.com, other fields defaults)

    • View Results Tree Listener

 

To capture the request details that were sent, I’m going to use a sniffer tool called Wireshark with this filter applied:

 

(ip.dst_host==54.93.140.145 || ip.dst_host==54.93.153.159) && http

 

This filter is necessary because Wireshark is so powerful that it will capture every single network packet. This filter limits its output to display only the following:

 

  1. HTTP requests, targetting 54.93.140.145 host

  2. HTTP requests, targeting 54.93.153.159 host

 

I’m particularly interested in the destination IP.

 

Exhibit A

 

 

 


 

As you can see, all the requests are going to the IP address: 54.93.153.159. The other one is not being hit at all.  

 

Why do the Requests go to Just One IP Address?

 

JMeter’s wiki page, JMeter and Amazon, claims that this happens because the DNS Caches on a Java Virtual Machine Level. It suggests using the sun.net.inetaddr.ttl=0 JVM property as a workaround and offers a few more recommendations. But, this doesn’t entirely solve the issue because:

 

  • It only works for Sun/Oracle Java Virtual Machines implementations

  • The underlying OS can return IP addresses for DNS hostnames from its own cache

  • Real browsers have their own DNS cache on top of OS

 

So, even with these workarounds, you might still encounter a situation where all the requests are going to a single IP address (as in Exhibit A). Here are some more examples of this happening:  

 


 

How the DNS Cache Manager Solves This Problem

 

BlazeMeter wanted to solve this problem, so for JMeter Version 2.12, its engineers contributed a brand new configuration element called the DNS Cache Manager. This works around the problem like a charm, and significantly improves the testing of the:

 

  1. CDN (Content Delivery Networks)

  2. DNS Load Balancing

  3. Load Balancers having >1 entry IP addresses

 

(Note: if you want to read more about the history and implementation of this, take a look at

Bug 56841 in the Apache Software Foundation Bug Tracking System).

 

Using the DNS Cache Manager

 

So let’s see how well it works! Let’s add a DNS Cache Manager to the Test Plan and configure it to use the Custom DNS Resolver with the two IP addresses already there. These are:

 

 

Now we can re-run the test:


 

Exhibit B

 

 

 

 

As this screenshot shows, both IP addresses (54.93.153.159 and 54.93.140.145) are now being hit by HTTP Requests.


 

The DNS Cache Manager GUI

 

The DNS Cache Manager has a very simple GUI.

 

 

 

 

Here’s a quick explanation of each element:

 

  • Name: The configuration element label. The DNS Cache Manager will be displayed in the Test Plan tree with the value provided in the input.

  •  

  • Comments: Here you can add arbitrary comments for clarification on various test elements are intended to do.

  •  

  • Clear Cache Each Iteration: The DNS Cache Manager caches the IP address -> the DNS hostname pairs and returns the cached value if the DNS hostname has a match in its internal cache. If this checkbox is ticked, the cache will be cleared at the start of the thread - just like in the HTTP Cache Manager

  •  

  • Use System DNS Resolver: The DNS Cache Manager will query DNS Server(s) defined in your OS network configuration in order to determine the hostname for each thread representing virtual users. In this case, you need to override the networkaddress.cache.ttl property in the $JAVA_HOME/jre/lib/security/java.security file and set it to 0 (the property defaults to “-1” which stands for “forever”)

  •  

  • Use Custom DNS Resolver: You can specify one or more DNS servers in the list.

  •  

    • If more than one server is provided, the DNS Cache Manager will choose a random one for each call - and will run as a round-robin mechanism.

    •  

    • If nothing is provided, the system DNS resolver will be used (the DNS server/s defined in your OS network configuration)


 

The DNS Cache Manager - Quick Tips and Troubleshooting

 

The DNS Cache Manager is simple to use and you shouldn’t run into any problems..but here are some quick tips to be sure you’ll avoid them.

 

1. Where to put the DNS Cache Manager

 

The DNS Cache Manager should be added as a child of the Test Plan or a Thread Group element.

 

2. Always Use HTTPClient4

 

Make sure that you have HTTPClient4 selected in “Implementation” drop-down

 

 

 

 


3. Enable Logging

 

It’s easy to enable logging on the DNS Cache Manager. Just append the following parameter to the JMeter startup script:

 

-Ljmeter.protocol.http.control.DNSCacheManager=DEBUG

 

as

 

java -jar ApacheJMeter.jar -Ljmeter.protocol.http.control.DNSCacheManager=DEBUG

 

or

 

jmeter.bat -Ljmeter.protocol.http.control.DNSCacheManager=DEBUG

 

You can click the yellow triangle with the exclamation sign at the upper right corner of the JMeter GUI, or select Options -> Log Viewer from the main top menu to toggle the log viewer window.

 

 

I hope (and believe) that this new feature will make your lives easier and your tests more robust and realistic. I’ve tried to cover as much as possible but if you have any questions, feel free to share them here.

 

Interested in writing for our Blog? Send us a pitch!