Jul. 10th, 2014

Load Testing Your Email Server: How to Send and Receive E-mails with JMeter

E-mail is one of the most commonly used communication channels in the modern world. Gmail, Hotmail and Yahoo Mail have over 1 billion registered users. Over 300 billion email messages are sent daily and 3.5+ million emails are sent every second.

 

Having your “own” email server is the de facto standard for most businesses. And the majority of ECM, CMS and other enterprise solutions have inbound and outbound email integrations - no wonder email server performance is such a hot topic!

 

Using JMeter? Then you’re probably familiar with the following scenarios:

 

1. Emails need to be sent to testers/developers/managers when the test is ended, an error has occurred, a critical threshold has been exceeded, etc. etc.

 

2. Test situations demand that email messages are read -  to activate registered users via email confirmations,  to check activity notifications, reset passwords and so on.

 

In this post, I’m going to cover everything you need to know about sending and receiving emails through Apache JMeter - and include end-to-end demos of email server performance testing.

 

All email related operations in Apache JMeter have JavaMail API under the hood. The folder you need here is: mail-{version}.jar in JMeter /lib. It’s included in the JMeter bundle - but double-check it’s there because JMeter won’t be able to send or receive emails without it.

How to Send Emails with JMeter - A Step by Step Guide

 

You can use JMeter’s SMTP Sampler to send email messages over SMTP and SMTPS protocols. JMeter itself is not a mail server but it can send email messages through a specified mail transfer agent.

 

To do this, you’ll need to provide the following parameters via the SMTP Sampler GUI:

 

  1. SMTP Server Host

  2. SMTP Server Port

  3. Sender address

  4. At least one recipient address

  5. The username and password combination (often required by not always)

 

All the other fields are optional - but you’ll probably also need to include the message subject and body.

 

In this demo, I’m going to send an email message via Google Mail’s SMTP server. You’ll need the following SMTP Sampler configuration for this:

 

  1. Server: smtp.googlemail.com

  2. Port: 587

  3. Address from: Any Gmail address. I’m going to use: blazemeter.test@gmail.com

  4. Address to: Any recipient address. Again, I’m going to use: blazemeter.test@gmail.com

  5. The Google Mail SMTP Server requires authentication, so tick the “Use Auth” box.
     - Username: blazemeter.test@gmail.co
     - Password: Enter the password. The password is displayed with asterisks in the JMeter GUI  but it’s stored unencrypted (as is) in the JMeter .jmx file

  6. Google Mail’s SMTP Server requires StartTLS - so click this radio button

  7. Message Settings: Leave this area as it is or fill it out in accordance with your test scenario.
    - Subject: You can put anything you want here. You can even leave it blank or send it without a subject by selecting: Suppress Subject Header. For this demo, I’m going to write: “Hello from JMeter SMTP Sampler”. If you want, you can also add the current value of the System.currentTimeMillis() Java function to the message subject by selecting: Include timestamp in subject
    - Message: You will need to include the message body. This section accepts plain text and JMeter Variables, Functions, Properties, and more. For this example, I entered the following text:  “Message from JMeter thread # ${__threadNum}”. As I mentioned in a previous blog post, Using JMeter Functions - Part II, the  __threadNum function represents the number of the thread currently being executed.

 

A few words on the remaining configuration options of the SMTP Sampler:

 

Security Settings

 

  1. Use no security features - no security protocol will be applied, all credentials and messages will be sent in plain text.
  2. Use SSL - all communication with the SMTP server will be done via the SSL protocol
  3. Use StartTLS - this tells JMeter to try to use the TLS protocol
  4. Enforce StartTLS - this tells JMeter that the TLS protocol must be used. If the SMTP server fails to respond, the connection will be terminated and the sampler will fail.

  5. Send plain body -  by default, JMeter sends multipart/mixed type emails. This box allows you to change them to plain text only.

  6. Attach files - an attachment (or multiple attachments) can be added here. If you have multiple attachments,  just use the “Browse” button as many times as you need. If your file paths are dynamic, separate individual files with the semicolon (;) sign.

  7. Send .eml - an .eml file can substitute the subject, message and attachments

  8. Calculate message size - JMeter will calculate the message size

  9. Enable debug logging - set mail.debug property to “true” (equivalent of Session.setDebug(true)

 

All done? Great, then you’re good to go.

 

This is how my SMTP Sampler GUI looked after I entered all my details:

 

 

 

 

I decided to kick off this sampler with five threads.

 

I went to the View Results Tree Listener  and saw that all five requests had succeeded:

 

 

 

All five messages were in my Gmail Web UI and thread numbers are displayed in the body preview.

 

Reading Mails with JMeter

 

JMeter is able to read email messages via the Mail Reader Sampler - from the POP3 and IMAP protocols.

 

In general, the GUI is pretty user-friendly - minimalistic and self-explanatory. But there are a few areas that require further explanations.  

 

The Mail Reader Sampler offers the following configuration options:

 

  1. Protocol - this defines the mail reading protocol to use. The default is “pop3”. If you’re using any non-plaintext Security features (SSL/TLS), the protocol should be “pop3s” for POP3 or “imaps” for IMAP.  If not, the Sampler will fail, stating something like “EOF on socket” which, let’s face it, isn’t that informative unless you decide to debug the Mail Reader sampler

  2. Server host - the DNS name or IP address of the POP3/IMAP server

  3. Server port - the port the POP3/IMAP server is listening to. Standard ports will be figured out by JMeter from the protocol

  4. Username - the username for POP3/IMAP server

  5. Password - the password associated with the username for the POP3/IMAP server

  6. Folder - a folder can be specified for the IMAP protocol

  7. Number of messages to retrieve - this defines the amount of messages to fetch

  8. Delete messages from the server - specify whether JMeter should remove messages from the server after retrieval

  9. Store the message using MIME - this controls whether the entire message will be stored in the Response Data (including Headers and Signature). Check this box if you plan to use SMIME Assertion against the Mail Reader Sampler

  10. Security Settings - these are exactly the same as for the SMTP Sampler

 

Ok, enough theory - let’s get going! In this demo, we shall read what we previously sent through the SMTP Sampler using Google Mail POP3.

 

The Mail Reader Sampler needs to be configured as follows:

 

  1. Protocol: pop3s

  2. Server Host: pop.googlemail.com

  3. Server Port: 995

  4. Username: blazemeter.test@gmail.com

  5. Password: associated password

  6. Number of messages to retrieve: 5

  7. Security settings: Use SSL

 

 

 

Run it with one thread.

 

As you can see in the View Results Tree Listener Mail Reader Sampler, it has fetched five messages.

 

 


Testing Your Mail Server Performance

 

One of the easiest ways to test your mail server is with BlazeMeter.

 

Blazemeter fully supports mail-related samplers and protocols...with just one small limitation: you’ll have to choose JMeter 2.4 or above from the JMeter Version combo. This is because JMeter doesn’t have the SMTP sampler and support of all JavaMail’s protocols until version 2.4.

 

 

 

JMeter engines of versions starting from 2.4 at Blazemeter can be used for inbound/outbound email routines testing without any limitations - providing the same metrics level as for any other Sampler type.

 

By looking at the Load Report for the SMTP Sampler followed by the Mail Reader Sampler, you can see that the mail server’s test response time surges as the load increases and also decreases as the load does.

 

 

 

Errors that occur during the performance session are reported on the “Errors” tab of the Load Report UI

 

 

 

Just one final note: please don’t use BlazeMeter to test the performance of any public email service without asking explicit permission from owners. If you don’t do this, your associated accounts and IP addresses might get blacklisted.

 

Interested in writing for our Blog? Send us a pitch!