The DNS Cache Manager: The Right Way To Test Load Balanced Apps
There’s no doubt that high availability and scalability are vital for today’s networking-based applications. Just think of the millions of people using services like Google or Facebook at this very moment!
To make sure their sites are seamless and robust, most companies turn to Network Load Balancing as a solution. This helps them ensure that their service will remain operable - even when there are unexpected circumstances or one instance failures.
There are many different deployment scenarios and high availability configurations. But most follow these basic ideas/requirements:
The end users sees the whole system as a single endpoint (through the Load Balancer)
The system consists of at least two instances
The Load Balancer orchestrates end user requests so they are distributed between the system instances
When one instance becomes unavailable (i.e. for maintenance, replacing or upgrading hardware or a power outage), the Load Balancer stops sending requests to it and the remaining instances serve the end users. As soon as the instance is operational again, the Load Balancer adds it back into its list.
The Right Way to Use JMeter With Your Load Balanced Systems
In today’s post, I’m going to be focusing on a scenario in which the load balancer has more than one IP address. I’ll show you how to ensure that JMeter is correctly sending requests to all the load balancer’s entry points.
It’s important to have multiple load balancer endpoints to ensure that the load balancer itself does not become a single point of failure - and that if one entry point stops serving requests, the remaining ones will continue to respond.
This diagram shows the network deployment which will be used in my demo.
Here’s a breakdown of each network element:
Apache JMeter. This is my workstation and requests will be sent from it.
Load Balancer. Amazon ELB has been configured to distribute requests across two “Application Under Test” instances. To make it easier to read and understand, I’ve used Load Balancer network details like:
DNS Name: example.com
LBIP1 and LBIP2 (Load Balancer IP address 1 and 2)
Different and real DNS names and IPs will be used for the real demo
Application Under Test Instances 1 and 2. A simple Apache HTTP Server on the Amazon EC2 instance will return the static HTML page with one sentence. I won’t be including them in today’s demo as I want to focus solely on Apache JMeter and the Load Balancer.
A DNS Overview
Already familiar with DNS? Skip this part and move straight to the Amazon ELB Load Test Demo.
For everyone else, here’s a quick overview:
DNS, also known as the “Domain Name System”, is a naming system for computers which converts human-readable names like “example.com” to IP addresses like: “220.127.116.11”. For example: if you open “blazemeter.com” in your browser, it does a DNS lookup of the associated IP address. As such, the actual request is made from your computer’s IP address to the BlazeMeter server IP address.
Want to find out the IP address for a host? Use these command-line tools:
MacOSX and Linux
So, this is how it will look when you’re determining the IP address of “example.com”:
The “example.com” domain goes to two IP addresses:
IPv4 address of 18.104.22.168
IPv6 address of 2606:2800:220:6d:26bf:1447:1097:aa7
Now let’s try “Facebook.com”
“facebook.com” also goes to two IP addresses:
IPv6: 2a03:2880:2130:cf05:face:b00c:0:1 (have you noticed this “face:b00c” hilarious bit?)
Now let’s try Google:
Ok, so here it’s a bit different. The “google.com” host goes to one IPv6 address and eleven IPv4 addresses. Although, I believe that it could be even more than this! Try running this command from your location - I think you’ll get different results. This is how Google is able to serve millions of concurrent users.
Given that we still live in IPv4 world, let’s stick to the scenario where one DNS hostname goes to more than one IPv4 address - like the jmeter.apache.org site here:
~$ dig +short jmeter.apache.org
In this demo, I’m going to test Amazon ELB - which I have created and configured. I can’t test any of the examples above because it’s forbidden to test big websites like Google and Facebook without explicit authorization.
This has the DNS hostname dns-cache-manager-1494470926.eu-central-1.elb.amazonaws.com - which goes to two IPv4 addresses
~$ host dns-cache-manager-1494470926.eu-central-1.elb.amazonaws.com
dns-cache-manager-1494470926.eu-central-1.elb.amazonaws.com has the address 22.214.171.124
dns-cache-manager-1494470926.eu-central-1.elb.amazonaws.com has the address 126.96.36.199
Let’s take a look at what happens if we send dozens of requests to the hostname: dns-cache-manager-1494470926.eu-central-1.elb.amazonaws.com.
Let’s create a simple JMeter Test Plan:
Thread Group (Threads: 5, Loops: 5)
HTTP Request (Server Name: dns-cache-manager-1494470926.eu-central-1.elb.amazonaws.com, other fields defaults)
View Results Tree Listener
To capture the request details that were sent, I’m going to use a sniffer tool called Wireshark with this filter applied:
(ip.dst_host==188.8.131.52 || ip.dst_host==184.108.40.206) && http
This filter is necessary because Wireshark is so powerful that it will capture every single network packet. This filter limits its output to display only the following:
HTTP requests, targetting 220.127.116.11 host
HTTP requests, targeting 18.104.22.168 host
I’m particularly interested in the destination IP.
As you can see, all the requests are going to the IP address: 22.214.171.124. The other one is not being hit at all.
Why do the Requests go to Just One IP Address?
JMeter’s wiki page, JMeter and Amazon, claims that this happens because the DNS Caches on a Java Virtual Machine Level. It suggests using the sun.net.inetaddr.ttl=0 JVM property as a workaround and offers a few more recommendations. But, this doesn’t entirely solve the issue because:
It only works for Sun/Oracle Java Virtual Machines implementations
The underlying OS can return IP addresses for DNS hostnames from its own cache
Real browsers have their own DNS cache on top of OS
So, even with these workarounds, you might still encounter a situation where all the requests are going to a single IP address (as in Exhibit A). Here are some more examples of this happening:
How the DNS Cache Manager Solves This Problem
BlazeMeter wanted to solve this problem, so for JMeter Version 2.12, its engineers contributed a brand new configuration element called the DNS Cache Manager. This works around the problem like a charm, and significantly improves the testing of the:
CDN (Content Delivery Networks)
DNS Load Balancing
Load Balancers having >1 entry IP addresses
(Note: if you want to read more about the history and implementation of this, take a look at
Bug 56841 in the Apache Software Foundation Bug Tracking System).
Using the DNS Cache Manager
So let’s see how well it works! Let’s add a DNS Cache Manager to the Test Plan and configure it to use the Custom DNS Resolver with the two IP addresses already there. These are:
Now we can re-run the test:
As this screenshot shows, both IP addresses (126.96.36.199 and 188.8.131.52) are now being hit by HTTP Requests.
The DNS Cache Manager GUI
The DNS Cache Manager has a very simple GUI.
Here’s a quick explanation of each element:
Name: The configuration element label. The DNS Cache Manager will be displayed in the Test Plan tree with the value provided in the input.
Comments: Here you can add arbitrary comments for clarification on various test elements are intended to do.
Clear Cache Each Iteration: The DNS Cache Manager caches the IP address -> the DNS hostname pairs and returns the cached value if the DNS hostname has a match in its internal cache. If this checkbox is ticked, the cache will be cleared at the start of the thread - just like in the HTTP Cache Manager
Use System DNS Resolver: The DNS Cache Manager will query DNS Server(s) defined in your OS network configuration in order to determine the hostname for each thread representing virtual users. In this case, you need to override the networkaddress.cache.ttl property in the $JAVA_HOME/jre/lib/security/java.security file and set it to 0 (the property defaults to “-1” which stands for “forever”)
Use Custom DNS Resolver: You can specify one or more DNS servers in the list.
If more than one server is provided, the DNS Cache Manager will choose a random one for each call - and will run as a round-robin mechanism.
If nothing is provided, the system DNS resolver will be used (the DNS server/s defined in your OS network configuration)
The DNS Cache Manager - Quick Tips and Troubleshooting
The DNS Cache Manager is simple to use and you shouldn’t run into any problems..but here are some quick tips to be sure you’ll avoid them.
1. Where to put the DNS Cache Manager
The DNS Cache Manager should be added as a child of the Test Plan or a Thread Group element.
2. Always Use HTTPClient4
Make sure that you have HTTPClient4 selected in “Implementation” drop-down
3. Enable Logging
It’s easy to enable logging on the DNS Cache Manager. Just append the following parameter to the JMeter startup script:
java -jar ApacheJMeter.jar -Ljmeter.protocol.http.control.DNSCacheManager=DEBUG
You can click the yellow triangle with the exclamation sign at the upper right corner of the JMeter GUI, or select Options -> Log Viewer from the main top menu to toggle the log viewer window.
I hope (and believe) that this new feature will make your lives easier and your tests more robust and realistic. I’ve tried to cover as much as possible but if you have any questions, feel free to share them here.
You might also find these useful:
Interested in writing for our Blog? Send us a pitch!