Ricardo Poleo is a Software Developer with 5 years of experience in languages like Java, PHP, and Javascript among others. He has been working on consultant companies in different countries. Now he is working at Abstracta with BlazeMeter Labs team to support and develop open source solutions for the community.

Become a JMeter and Continuous Testing Pro

Start Learning
Slack

Test Your Website Performance NOW!

arrow Please enter a URL with http(s)
Nov 11 2019

Introducing the JMeter HLS Plugin 2.0

BlazeMeter Labs has just released a new version of the HLS JMeter plugin, with multiple improvements and features that enable you to easily create a script to load test your video streaming services with Apache JMeter™.

 

As you may know, the HLS plugin allows you to easily test the performance of your on demand or live streaming video servers when exposed to different types of loads.

 

Let’s take a tour of the latest improvements while configuring a test, shall we?

 

Creating Your JMeter Script with the HLS Plugin

 

Getting the Plugin

 

  1. If you don’t have the Plugins Manager already installed, read the article, How to Install the JMeter Plugins Manager. After that, open it up. (Options → Plugins Manager)
  2. Select the Available Plugins and in the search field, write HLS and select the HLS Sampler plugin checkbox.
  3. Click the button Apply Changes and Restart JMeter.

 

Setting up the Test

 

4. You can add the HLS Sampler Recorder element by right-clicking on the Test Plan’s Thread Group element to get the Add menu: (Add → Sampler → bzm - HLS Sampler).

 

HLS Sampler 2.0

 

To begin with, the plugin UI has been simplified so it’s no longer required that you specify the protocol to use or type of video, but instead, the plugin automatically determines such values by itself, making the configuration easier and faster!

 

Additionally, the plugin will now also follow up redirects and continue downloading from that point without further intervention from the user.

 

5. Set the URL of the playlist you want to test out.

6. Set the video time you want to download.

 

Another thing to notice is that now you can select alternative subtitles and audio tracks to be downloaded alongside the video. If the plugin doesn’t find the selected track, then the default one will be used.

 

7. Select the language of subtitles and/or audio you want to use in the test or leave it blank to use the default one.

8. Select between Min, Max or Custom bandwidth and resolution.

 

One change in this regard is that now the video stream variant resolution (through bandwidth and resolution settings) is handled consistently regardless of the order in which the variants appear in the playlist. The bandwidth will now be considered more important (treated with higher priority) than resolution selection, unless the resolution is set to a custom value and bandwidth isn’t.

 

9. Check resume video download between iterations in case you want the plugin to download from where it left it in the last iteration.

 

HLS 2.0 jmeter plugin

 

In previous releases, the plugin automatically continued the download from where it left off between iterations. Now, the plugin gives you the possibility to decide how each iteration is going to behave, allowing through the previously mentioned checkbox to use the previous plugin version’s logic or restart from the beginning with each iteration.

 

10. Remember to add a View Results Tree to see the results. That’s it! Your script is ready to be executed.

 

In the rest of the post, we’ll cover additional changes to the HLS plugin you should know about.

 

Additional changes to the HLS Plugin

 

As expected, you asked and we listened! No longer do you have to wait until the full playlist is downloaded to see the result of each downloaded segment. To give you faster feedback on how your service is performing, the hierarchical structure has been removed, providing a way to see how the server is behaving in real time during the test execution. As you can see in the image below, the hierarchical structure is no longer used, replaced with a plain and simpler structure.

 

HLS 2.0 jmeter plugin

 

Additionally, the naming of sample results has been improved, to simplify the identification of each request and proper statistic reporting when using the Summary Report test element and similar aggregate visualizing test elements.

 

Furthermore, you can now select what particular type of sample result (master playlist, media playlist, video segment, audio segment, etc) you want to apply a post processor or assertion to, allowing you to implement specific logic for each one. Please check the Assertions and Post Processors section in the plugins Readme for more details about this.

 

NOTE: Now, the main and subsamples selection on assertions and post processors have no effect anymore.

 

Another change made over the last release is the possibility to stop the test plan execution ipso facto using the Stop button. In previous releases, you had to wait until the whole video segments were downloaded before being able to actually stop the test, but this is no longer the case. Here is a picture showing the result of stopping a test plan:

 

HLS 2.0 Jmeter Plugin

 

As a way to prevent excessive consumption of resources for both JMeter nodes and the servers under test, the plugin now uses the HTTPClient with standard JMeter reuse of connections, avoiding creating and closing connections for every request to the servers. Moreover, the plugin follows HLS RFC guidelines for solving relative URLs and waiting for the proper period between playlist reloads, avoiding potential busy waiting and providing the same behavior that video players have, simulating real life users more accurately.

 

This new this new release fixes the integration with all JMeter HTTP configuration test elements (cookies, headers, cache, etc) by using JMeter-provided HTTP sampler logic to make all requests and generate sample results. This also provides consistent formatting of sample results with the ones generated by HTTP sampler and fixes all existing issues while handling non-success status codes and connection problems.

 

Since not everything is about generating load to the servers, but also measuring the impact of such load by showing the key performance metrics and statistics, this version adds to each media segment sample result a new response header (X-MEDIA-SEGMENT-DURATION) which contains the reproduction duration for the segment. This facilitates the obtainment of video streaming KPIs to measure the user experience on the reproduction by combining it with download times. Let’s see an example of how to do this:

 

1. Add a User Defined Variables configuration element (Add → Config Element → User Defined Variables) to the created test plan with the following variables:

 

HLS 2.0 jmeter plugin

 

2. Add a JSR223 PreProcessor to the HLS Sampler with the following code, which will reset the variable values before sampler starts on each iteration:

 

vars.putObject("segmentStartEpoch", System.currentTimeMillis())
vars.putObject("bufferedTimeMillis", 0)
vars.putObject("segmentBufferingMillis", 0)

 

3. Now you can add a JSR223 Assertion to the HLS Sampler using the following code, which will calculate segmentBufferingMillis and mark any video segment that requires buffering as failed. This will let you easily identify problems with perceived user experience when reproducing the video.

 

 

long extractSegmentDurationMillis(headers) {
	def headerPrefix = "X-MEDIA-SEGMENT-DURATION: "
	def headerStart = headers.indexOf(headerPrefix)
	if (headerStart < 0)
	  return null
	headerStart += headerPrefix.length()
	def segmentDurationSeconds = Float.parseFloat(headers.substring(headerStart, headers.indexOf("\n", headerStart)))
	(long) (segmentDurationSeconds * 1000)
}

long segmentDurationMillis = extractSegmentDurationMillis(prev.responseHeaders)
if (!segmentDurationMillis) {
	return true
}
long bufferedTimeMillis = vars.getObject("bufferedTimeMillis")
long segmentStartEpoch = vars.getObject("segmentStartEpoch")
long now = System.currentTimeMillis()
bufferedTimeMillis -= (now - segmentStartEpoch)
if (bufferedTimeMillis < 0) {
	AssertionResult.setFailure(true)
	AssertionResult.setFailureMessage("Segment buffering " + (-bufferedTimeMillis) + " millis")
	vars.putObject("segmentBufferingMillis", -bufferedTimeMillis)
} else {
     vars.putObject("segmentBufferingMillis", 0)
}
vars.putObject("bufferedTimeMillis", Math.max(bufferedTimeMillis + segmentDurationMillis, 0))
vars.putObject("segmentStartEpoch", now)

 

Ok, now we have a functional logic for getting the KPIs, let’s store the values in jtls for later analysis:

 

4. You can add the following configuration to user.properties at JMeter bin folder (and restart jmeter), which will instruct JMeter to include the calculated variables in generated JTL files (generated with listeners) which can later on be used in analysis. It also defines custom graphs to be included when generating HTML reports with JMeter allowing you to easily view this information in a friendlier way.

 

 
sample_variables=bufferedTimeMillis,segmentBufferingMillis

## Custom graph definition
jmeter.reportgenerator.graph.custom_segment_buffering.classname=org.apache.jmeter.report.processor.graph.impl.CustomGraphConsumer
jmeter.reportgenerator.graph.custom_segment_buffering.title=Segment Buffering
jmeter.reportgenerator.graph.custom_segment_buffering.property.set_Y_Axis=Buffering Time (ms)
jmeter.reportgenerator.graph.custom_segment_buffering.property.set_X_Axis=Over Time
jmeter.reportgenerator.graph.custom_segment_buffering.property.set_granularity=${jmeter.reportgenerator.overall_granularity}
jmeter.reportgenerator.graph.custom_segment_buffering.property.setSampleVariableName=segmentBufferingMillis
jmeter.reportgenerator.graph.custom_segment_buffering.property.setContentMessage=Message for graph point label


jmeter.reportgenerator.graph.custom_buffered_time.classname=org.apache.jmeter.report.processor.graph.impl.CustomGraphConsumer
jmeter.reportgenerator.graph.custom_buffered_time.title=Buffered Time
jmeter.reportgenerator.graph.custom_buffered_time.property.set_Y_Axis=Buffered Time (ms)
jmeter.reportgenerator.graph.custom_buffered_time.property.set_X_Axis=Over Time
jmeter.reportgenerator.graph.custom_buffered_time.property.set_granularity=${jmeter.reportgenerator.overall_granularity}
jmeter.reportgenerator.graph.custom_buffered_time.property.setSampleVariableName=bufferedTimeMillis
jmeter.reportgenerator.graph.custom_buffered_time.property.setContentMessage=Message for graph point label

 

When you finish creating your JMeter script to test your HLS applications, you can upload it to BlazeMeter and enhance your testing abilities with better reporting, collaboration, scalability, and all their new continuous testing features! The BlazeMeter platform also allows you to generate traffic from all over the world, testing the performance of your server for users worldwide. And that’s it! This is just a part of the huge changes that BlazeMeter Labs has prepared for you. Don’t just read about them though, try them! Let us know in the comments below what other improvements or new features you’d like to see or need.

     
arrow Please enter a URL with http(s)

Interested in writing for our Blog?Send us a pitch!