How to Test an Email

What can you test in your email?

Testing the subject line

Include tasks that measure how clear and useful contributors found your email

Creating an email test in the UserTesting Platform

Testing an email using an unmoderated test

Creating a test using the Templates Gallery

Concise, well-constructed emails are key to communicating effectively with your customers, and can play a pivotal role reinforcing brand recognition, and driving customer engagement with and adoption of the products and services you offer.

Before distributing it to a large audience, you’ll want to test your email and its content to ensure they fulfill your goals. Through tasks you present in a UserTesting test, you can do just that by testing specific aspects of the email, and whether contributors find the email useful and easy to understand.

Before diving in to the steps for creating an email test in the UserTesting Platform—including how to get the email in front of contributors so that they can test the email—let's take a closer look as to what aspects and attributes of the email you can test and the types of tasks/questions you can deploy to measure how well your email performs with contributors. 

What can you test in your email?

Testing the subject line

Before users can be motivated to read and act on the content of your email, they must first be motivated to open it. One task directly tests whether contributors find the subject line compelling is to ask:

• "Would you open the email based on the subject line? Explain your answer."

Or imagine a situation where you've come up with multiple subject lines and you want contributors to choose the one they prefer. In that case, run a preference test, asking contributors to compare two subject lines: 

• “Which subject line do you prefer?” 

(Depending on the contributor feedback—if, for example, contributors are split on which of the two options they prefer—you may run more preference tests with additional subject lines.)

When contributors have selected their preference, a useful follow-up is to ask:

• “Based on your previous response, would you open an email which features that subject line? Explain your answer.” 

Include tasks that measure how clear and useful contributors found your email

When testing your email through the UserTesting Platform, word tasks with an eye towards learning how simple and clear contributors find your email and whether you were able to convey the message you intended. Rating scale questions with instructions for contributors to explain aloud their rating are a common strategy:

• “How CLEAR would you say the messages are to understand for you personally? [11-point Rating scale: 'Not clear at all' to 'Very clear']. Explain your answer."
• “After reading this email, do you feel you have a basic understanding [insert email topic]? [11-point Rating scale: 'Do not understand at all' to 'Very clear understanding']. Explain your answer.”

Here's a sample of task questions which address how useful contributors find the email. Again, providing a rating scale task is a sound way to gauge how well the email is performing:

• “How helpful did you find the content of this email? Please explain aloud.” [11-point Rating scale: “Not at all helpful” to “Very helpful”]
• “If you were a new [company name] customer and received this email, how HELPFUL would you say the content is for you personally? Please explain aloud.” 
• “Do you feel this email includes useful information for a new [customer name] customer? Explain your answer.”
• “Do you feel this email includes the right kind of information for a customer to [do what the email explains]? Explain your answer.”

Still other tasks can probe a contributor’s general feeling toward the email or how the email can be improved:

• Do you have any comments about the email in general?
• If you had a magic wand, how would you improve this?

Creating an email test in the UserTesting Platform

There are two ways to create an email test through the UserTesting Platform:

• Create an unmoderated test from scratch. 
• Use the Email engagement test template from the Platform’s Templates Gallery.

Testing an email using an unmoderated test

If you have already written up tasks and questions to test your email, you can create a new unmoderated test from scratch. Go to the Platform Dashboard and, in the top right corner, click Create an unmoderated test.

Choose the device you're testing (website, app), and then, on the test-builder page, provide a starting URL and enter a scenario for your email test.

Creating tasks: use the Image asset to display the email to your test contributors

After choosing the device you are testing (website, app), and having provided a starting URL and a scenario for the test, add the tasks you wrote to test your email. For the first task, you'll want produce use the Image asset and drag and drop an image of the email into the task field. This approach allows the contributor to see both the content and the design of the email.

As seen in the instruction text in the image below, the contributors will see the email in the test once the image you earlier dragged and dropped finishes downloading. Language instructing the contributors to move on to the next step in the test will automatically populate in the task.

Subsequent tasks typically use rating scale questions to address key questions such as how clear contributors find the email, and whether they find it useful:

Creating a test using the Templates Gallery

From the same New drop-down menu accessed earlier, select the Create a test from a template option:

You’ll be taken to the Templates Gallery. Scroll down to find the Email engagement template tile. The “TEST PLAN” label at the top of the tile indicates that the template has a test plan, with tasks, already prepared for you to use.

Hovering your cursor over the tile triggers a blue Use this template call-to-action.

Clicking the call-to-action opens the template’s test-plan window, which describes what you can test (prototype, website) and whom you can invite as test contributors (from the UserTesting Contributor Network or your own Invite network).

To preview the test plan, in the lower left-hand corner, click the See template preview link

Among other things, the preview shows you each of the 11 tasks that UserTesting provides you to use to test the email. (Scroll to see all the tasks or click Show all.)

If instead of previewing the template, you click the Next button, you are taken to the Platform page where you can create your contributor audience and your test plan. In the same test-plan builder section as used when creating an email test from scratch, you'll see the 11 template tasks prepopulated into the Test Plan section:

Please provide any feedback you have on this article. Your feedback will be used to improve the article and should take no more than 5 minutes to complete. Article evaluations will remain completely confidential unless you request a follow-up.