How to Test Video Streaming with JMeter

Video streaming testing is essential in today's world. Learn the basics and how to get started in this blog.

What Is Video Streaming Testing?

Video streaming testing refers to the process of testing streaming services to ensure the functionality, performance, and usability match expectations. 

Why Is Video Streaming Testing Important?

Video streaming services are used in sports events, music concerts, news and more. Before such events it is important to run performance tests on your streaming services, to make sure they don’t crash when users are watching. 

The video streaming protocols we will test are taken from this list, which contains the most popular streaming protocol: HLS, Progressive Download, RTMP, MPEG-DASH. (Icecast is used mainly for audio streaming, for example, in radio services, so we excluded it from our list).

Each protocol will be discussed in two parts: script realization and KPI Monitoring. So first, let’s discuss each part and learn what is important to take into consideration about each one. After that we will proceed to the protocol performance testing examples.

How to Prepare For Video Streaming Testing

Script Realization

When creating your performance scripts, it’s important to take the following into consideration:

• Different devices - different video resolutions. When you watch a video using your phone, you have less video resolution than when you watch it using a laptop. You need to pay attention to the most popular resolutions in your tests when creating your scripts.
• You can configure different bandwidth speeds in JMeter. 
• For testing from several regions/states, you can use JMeter’s distributed mode and several AWS instances. Or, you can use BlazeMeter, which is easier and friendlier to use.
• Concerning user behavior, it is worth remembering that users do not always start watching the video from the beginning. If they’re watching VOD, they can pause the stream or choose another one. This affects the way the server is load tested.

Useful KPIs to Monitor When Load Testing

When analyzing script results, these are the KPIs we recommend you monitor:

1. Percentage of HTTP request errors
2. Average loading time of playlist for HLS, DASH
3. Average loading time of chunks for HLS, DASH
4. Bitrate (throughput);
5. Average loading time of starting stream
6. Lag time

We can use JMeter to get the first four metrics. The fifth and sixth metrics can be calculated using timers and frame load time for some of the protocols. However, I am concerned about the correctness of the results for lag time through JMeter because of the calculation complexity and the features of the player used, which can affect the playback of the video stream. Therefore, at the end I will propose a solution for obtaining these last two KPIs.

Note: items 1-6 from the list can also be obtained using the HLS commercial plugin for video streams that work through HLS, MPEG-DASH, and Smooth protocols.

That’s it for now. Let's proceed to the implementation of scripts and after that we will return to the problem of metrics calculation in the end.

How to Test Video Streaming Protocols

Load Testing HLS/MPEG-DASH

Today, most video services work with an incoming RTMP stream. Then, the online video is encoded to HLS format and delivered to users. As an example, we will use a service for online broadcasts - Smashcast.tv. When viewing the broadcast, we request a playlist (manifest), from which we get links to the chunks and load them (only the last three chunks are used). After ~2 seconds, we re-request a playlist (at the same address as we did the first time). If it contains new chunks, then we will load them.

1. You can implement your JMeter script via an HTTP Request Sampler. We recommend using the HLS plugin for your VOD testing.

This JMeter script does the following:

• After loading the manifest, the entire playlist is saved to the ArrayList object and when the manifest is reloaded, old and new playlists are compared to avoid loading old chunks.
• The Parallel Controller is used to ensure that manifest requests are not connected with chunk requests.
• A Dummy Sampler was added because if the loop was executed many times and the condition did not pass, after some time the execution will be interrupted.

2.   The script metrics:

• Average loading time of manifest - average time for GET manifest sampler
• Average loading time of chunks - average time for GET chunk load sampler
• Bitrate - throughput for GET chunk load
• Average loading time of starting stream - the sum of the average time of samplers from GET /api/media/live to GET first chunk
• Percent of HTTP request errors

MPEG-DASH works exactly the same, but the manifest is XML, so you will only need to replace regexp for chunks extraction. For VOD, we can extract all the necessary data from the manifest using regular expressions, but for live stream, we need to calculate the number of the actual chunk by the following formula:

latestSegmentNumber = (now - availabilityStartTime - duration)/duration + startNumber;

It is also important to note that the HLS Plugin for JMeter supports MPEG-DASH from version 3.0 so you can now just provide the link to the MPD manifest and the plugin will do the rest.

Load Testing Progressive Download

The Progressive Download technology is quite simple: you have a variety of videos at different resolutions that are available for downloading as video files on an HTTP Server. When a request is made for a particular video file, the video file is sent across to the player at a single Bitrate. The peculiarity of this technology is that a user of the device has a copy of this video, so this method is also called pseudo streaming. It is not used for online broadcasts, but is actively used for streaming small videos.

In fact, this is a simple download of a video file from the server and we can use the HTTP Request sampler for that. 

1. The script is fairly simple and consists of only two requests (of the video and video upload, no need to upload playlists).

If we start watching the video from the middle, the server cuts the original video and sends us the cut part. This behavior is controlled by sending bytes to the video request header via JMeter’s HTTP Header Manager.

2. As for metrics, metrics connected to playlists, chunks and bitrate are not suitable for this case, and we cannot calculate the last two (loading time of starting stream and lag time) with JMeter. So just make sure the average download time of the video file is less than its duration.

Load Testing RTMP

Another one of the most commonly used protocols for video streaming is RTMP. However, unlike the previous methods, it does not work via HTTP (due to this, its latency is lower than the protocols above). Therefore, implementation is a little more complicated.

1. You can learn how to do it from this article, and use Youtube as a service for streaming through RTMP.

2. All tools and the Java Sampler that were discussed in the RTMP article are intended only for creating the load and do not provide any additional KPI information, except for the information about the interrupted connections.

The Solution for KPIs: Running JMeter + UI Tests in Parallel

So, we are in a situation in which we can easily create a load, but we have difficulties getting some KPIs: lag time for all protocols and average loading time of starting stream for RTMP and Processing Download.

A possible solution is to launch a UI Test with the load test in parallel. As a result, we will be able to get information about the behavior of the UI under the load on the server and also if it's a live stream, it makes sense to pay attention to the video latency.

Selenium is suitable for writing the test. Also you can configure the launch of UI tests and JMeter tests by using Taurus. Apart from statistics, it also makes sense to run several UI Tests.

• Lag time can be calculated by using a timer that will be started when the download wheel appears and stops after it disappears.
• Time of starting stream can be calculated from the first download wheel.
• In addition to calculating metrics, we can check whether there are any problems with displaying a picture, like freezing, bad or black frames.

BlazeMeter offers a free recorder that will enable you to record a scenario and get a JMeter and Selenium script for it. By using the recorder, you will be able to record the live streaming scenario, get the scripts and adjust them to calculate the KPIs.

Also, many video streaming players have an API for calculating similar metrics. For example, here is a demo with metrics calculation for JWPlayer, but the source code of the application has to be modified for the correct event processing. Players based on Adobe Flash player usually have API too.

Browser engines provide some metrics too. For instance, to get access to DecodedFrameCount for an html5 player on this site you need to execute following js code:

document.getElementById("html5_player").webkitDecodedFrameCount;

For the demonstration you can paste it to the developer console:

You can see the full list of the provided metrics here on Wiki.

Start Performance Video Streaming Testing with BlazeMeter

BlazeMeter is a great option to run your video streaming performance scripts, where you can scale them for 1000s or 1000000s of VUs and run the test from all over the world. BlazeMeter's performance testing platforms also enables teams to share tests and results with managers and colleagues, as well as get insightful results both in real time and over time.

Make sure you are ready for big events that require live streaming, like soccer or football tournaments or music shows, by performance testing enough time in advance. If your website crashes, your users will move on to your competitors, who tested their website properly.

To learn more about performance testing video streaming, check out BlazeMeter University.

START TESTING NOW