Jul. 6th, 2014

Performance Testing in a Scaled Down Environment. Part Two: 5 Things You Can Test


In my last post I covered the Challenges of Performance Testing in a Scaled Down Environment. This week, I’m going to address what you should check when scaling down for a performance test.


When running a performance test in a scaled down environment, you use the same code as in the live environment. This means that you can get a very good idea of how the components are going to interact with each other. The results of a performance test can give key insights into how an application will perform in the live environment. So, if a scaled down performance test is the only option, you need to know what to check to get the best results.


Five Things to Test in a Scaled Down Environment


1. Code and Database Interaction

Developers should examine how a code will interact with the database when running a scaled down performance test. Developers these days don’t write any SQL queries - and they aren’t particularly concerned about them either.  This is due to the increased use of ORM (object-relational mapping) in an application. ORM usage prevents the developer from knowing how an SQL query will look, or when and how often it will run.


2. External Service Calls

It’s also important to test external service calls, in particular the number and utilization of the calls. These calls can perform very demanding, resource-consuming and long processing work, so it’s vital to understand how the application interacts with them. For example, in developing BlazeMeter, one of the external services we interacted with did some internal 'heavy lifting', and each call was very time consuming. As the number of concurrent users increased, the processing time of each request grew exponentially. We were able to solve this problem by adding a cache layer that interacted between the user and application every few seconds, limiting and controlling the number of requests. The cache layer enabled us to decrease the number of hits per second to this service. 


3. Load Allocation

You also need to know how a load is allocated amongst the servers. By means of a load balancer, load distribution to the server can be dictated, predicted, and tested. Many different algorithms can be used to perform these tests, which are numerous and range from trying to disperse the same amount of sessions on each server, to considering the resource consumption on an individual node, to finding the lowest CPU possible. A load balancer can be configured to communicate with the application in order to find out which server will best handle the load distribution. The algorithms tested in a scaled down environment are the same as those in a fully scaled environment. The load balance policy will mimic its behavior in the fully scaled environment and should be expected to distribute users efficiently.


4. Monitoring the Application Server

An application will have the same memory footprint in the scaled down environment as in the live environment, because the same code is being tested. The code will affect the same resources -  like cache size and database queries. Although these resources will not be identical in the fully scaled environment, the code’s dependency will remain the same, regardless of a faster memory or better I/O. If a bottleneck is found in the code in a smaller environment, the same can be predicted for the larger scaled environment. Monitoring the application’s server in a test environment will show how the server will behave under stress, giving insight into which metrics should be monitored and which errors should be expected in the production environment. Practicing handling and debugging crashes in the scaled down environment is an advantage, making it easier to work through the flaws in a code.


5. Run a Soak Test

Above all… we strongly urge you to run a soak test. Soak testing is used to check the performance in a simulated environment over an extended duration of time. If a performance test is usually executed for an hour, a soak test would be tested over a longer period of time. Some companies even perform soak testing for up to a few months. In a scaled down environment, the extended time needed for a soak test is feasible. When using a scaled down environment, it is important to try to obtain results that are as genuine as possible. Soak testing will allow you to collect truer results from your performance test because it takes time into consideration. Running a soak test captures the natural course of the code that can only be examined with time, because the environment becomes dirty over time and automatic actions occur - such as cleaning the cache. The impact of memory leaks may seem insignificant in a hour of testing, when in fact, it is a serious issue that should be addressed. APN providers, like New Relic, collect large amounts of data during the test and report on how an application utilizes its resources. New Relic can be used in sync with soak testing to maximize the monitoring of the application’s metrics. Soak testing gathers more authentic data that will better reflect how the code should behave in the live environment.


The Preferable Option: A Fully Scaled Environment

But when it comes down to it, a fully-scaled environment in the cloud is still the most preferable option for performance testing. Testing in the cloud is the most practical option and can be accomplished by using platforms like BlazeMeter.


I wouldn’t recommend using a scaled down environment as it is always a compromise - but it is still frequently performed by companies that don’t have enough resources. If scaling down is your only option, there are measures you can take to produce the most authentic results possible. Take into consideration the tips specified here, remember how important it is to understand how components interact and influence the code, and utilize applicable resources to produce a successful performance test.


Interested in writing for our Blog? Send us a pitch!