Blog
June 11, 2021
HLS vs. RTMP: Which Is Best For Testing?
Open Source Automation,
Performance Testing
HLS or RTMP? What's the difference? And which is bets for streaming and testing? Find out in this blog!
The Rise of Media Streaming Applications
Media streaming applications are widespread, and the number of them is constantly growing. This is in line with the increase in people’s desire and demand to watch various events from desktop, laptop and mobile devices. For example, the upcoming world soccer games will draw huge audiences to watch the tournament on websites that live stream the event.
As a rule, most of these media streaming applications or services target a wide audience. From the development prospective this requires thorough load testing of such applications before releasing them for public usage. That’s why special attention is paid to the media streaming load testing techniques in the BlazeMeter blog, while we believe Apache JMeter™ is the best solution for this task.
Back to topWhat's the Difference Between HLS and RTMP?
HLS and RTMP are well known protocols for streaming media contents over. HLS stands for HTTP live streaming and it is the application layer protocol. RTMP stands for real time media protocol and it’s the transport layer protocol. The idea of HLS is to stream media over the HTTP protocol, whereas RTMP streams media over TCP or UDP protocols. There’s one variance of the RTMP - RTMPT, that is the application level protocol, which runs over HTTP(s).
This blog post will explain how to create JMeter load testing scripts for HLS vs. RTMP. Then, I will compare load testing the two protocols according to the following criteria:
- Script Creation Complexity
- Request Imitation with JMeter Samplers
- Monitoring Response Time and Throughput-related Metrics with JMeter Listeners
- Validating Performance Related Metrics with JMeter Assertions
- Quality Evaluation on the End User Side
HLS vs. RTMP: Scripting
HLS JMeter Scripting
HLS uses web servers to send media content through a number of streams at different bitrates. For this purpose the original stream is split into a number of chunks as sequential downloadable file components, which are then incorporated into playlist formats. To download the original media stream from a server, it’s necessary to download the playlist first, read its contents, which is a list of chunks of downloadable files, and then iterate over the playlist while sequentially downloading all the chunks.
Following this, load testing HLS services is not a difficult task. Before the HLS sampler was introduced to JMeter, the HTTP testing algorithm was implemented on a series of HTTP samplers and RegExp post-processors. The algorithm performed the following actions: getting a playlist, using the HTTP sampler, parsing its contents to get paths to the file chunks, using the RegExp post-processor and finally downloading all file chunks with HTTP samplers according to the order specified in the playlist. An example of a working script created according to this method is shown in the screenshot below.
With the creation of the HLS sampler, scripting has become much simpler. One HLS sampler can do the whole job of retrieving HLS playlists, parsing them and downloading all the chunks from the playlist.
Let’s think of an example where a streaming server creates a few sets of media chunks for various bitrates. Each set is described by a certain playlist. All chunk playlists have a reference in the master playlist. Depending on the bandwidth and the varied network throughput, the HLS player will select a proper playlist with the required bitrate of media chunks.
The HLS sampler automatically imitates this case. Without it, imitation of this case in JMeter is a complicated task. The screenshot of the script with the HLS sampler is shown below.
The standard set of JMeter listeners allows evaluating the response time, throughput and number of errors, if there are any. But if the load testing script is based on the HLS sampler the standard listeners will show only consolidated metrics for all the requests generated by HLS samplers. This is because they only display aggregated information, such as aggregate or summary report. It won’t be possible to evaluate performance metrics related to the playlists access and performance metrics related to the chunks downloading. If the script is based on the HTTP requests, instead of the HLS sampler, this information can be easily consolidated for requests related to accessed playlists and to downloaded chunks.
RTMP JMeter Scripting
When media contents are streamed over the RTMP protocol, the entire stream is split into smaller chunks, the size of which is negotiated dynamically between the client and server. There’s the handshaking phase, then the RTMP connection between client and server is established and only after that the data is transferred between the client and server.
Unfortunately, there are no official JMeter plugins that are able to implement communication over the RTMP protocol. There are 2 third-party plugins, but each of them was last updated a few years ago. Bearing in mind that for two years JMeter progressed from version 2 to version 4, these plugins have to be verified for the newer JMeter versions. That’s why it’s better to use other approaches to conduct JMeter load testing of RTMP streaming services.
Setting aside the approach of writing an RTMP sampler, other ways to load test RTMP include using the JAVA sampler, OS sampler and JSR223 sampler. The usage of a JAVA sampler in the script that imitates the RTMP player is shown in the screenshot below.
Besides the JMeter script, the screenshot displays the JMeter console, which shows information about the downloaded audio and video stream chunks: timestamp and the chunk size. Standard JMeter listeners display consolidated information for each JAVA sampler, without distinguishing time required for the negotiation, connection and the time required for downloading individual RTMP packets. The implementation of JMeter scripts imitating the behavior of the RTMP player through a JSR223 sampler requires more efforts in coding. But the advantage of this approach is the ability to get the information about performance related metrics without much effort for each chunk of media content sent from the tested server, with JMeter standard listeners,
The evaluation of the time that is required to get each chunk of media from the server can be performed with a standard listener, if the downloading of each chunk is implemented with separate HTTP samplers for HLS, or with JSR223 samplers for RTMP. Alternatively, it can be performed with a custom made JMeter listener. The time can be asserted with custom assertions for both protocols. An example of a time assertion for downloading each HLS stream media chunk is shown in the screenshot below. With this assertion it’s possible to evaluate the quality of the restored video on the end user side.
The assertion is based on the information about the duration time of the playback of each chunk. This value is extracted from the playlist for each downloaded chunk. If the download time of media chunks accumulated from the start of the test, is more than the total duration of playback time of chunks that have been downloaded, then for the end user the quality of the restored video will be degraded. As a result the assertion fails.
The assertion of downloading time of the RTMP stream is more complex, as it requires to parse this information from the RTMP packet.
Back to topComparing HLS vs. RTMP in JMeter
The table below contains information about the specifics for writing JMeter scripts for testing RTMP and HLS streaming applications and the efforts required to accomplish these scripts.
HLS | RTMP | |
Script Creation Complexity | Medium | High |
Request Imitation with JMeter Samplers | It’s possible to imitate requests with the HTTP sampler and HLS sampler. | JMeter doesn’t have standard samplers for generating RTMP requests. The RTMP requests can be generated with custom Java libraries, using the Java sampler, JSR223 sampler or OS process sampler. |
Monitoring Response Time and Throughput-related Metrics with JMeter Listeners | Standard JMeter listeners can be used to monitor and collect performance related metrics. But using standard listeners, such as aggregated report, with the HLS sampler doesn’t provide enough performance details for each separate media chunk. (These components have to be disabled in real tests). | Standard JMeter listeners can be used to monitor and collect performance related metrics. But the code developed for the JAVA and JSR223 samplers has to be developed in a way that enables using standard listeners to get performance details for each separate media chunk. (These components have to be disabled in real tests). |
Validating Performance Related Metrics with JMeter Assertions | Standard JMeter assertions can be used validation response time and code of HTTP requests for downloading of playlists and media chunks. Custom assertions, based on the JSR223 assertion JMeter component, have to be developed by QA engineers for more complex cases. | Custom assertions, based on the JSR223 assertion JMeter component, have to be developed by QA engineers for this purpose. |
Quality Evaluation on the End User Side | It’s possible to evaluate jittering and delay time before starting the video playback for the end user, with assertions. | It’s possible to evaluate jittering and delay time before starting the video playback for the end user, with assertions. |
Running HLS vs. RTMP in BlazeMeter
After creating your script in JMeter, take a moment to think about your next steps. Do you need to scale up your test to 1000s or 1000000s of users? Do you need to run the test from all over the world to simulate real user traffics? Do you need to share your tests and results?
If so, just upload your script to BlazeMeter and run it. You will get KPIs and insights in minutes, as well as sharing and collaboration capabilities.
To learn more about performance testing video streaming, view this free webinar, Using JMeter for Performance Testing HLS Video Delivery. Request a demo to get a more in-depth explanation.