Testing for Comprehension

Introduction

Building Your Test Plan

Templates

Use Cases

Sample Test Plan

Introduction

Streaming giant Netflix has a documentary series called Pepsi, Where’s My Jet?, that recounts the true story of John Leonard and his fight with Pepsi in 1996. Soft-drink company Pepsi had aired a commercial in which they stated, jokingly- if a customer collected enough "Pepsi points," through Pepsi purchases, the customer could then trade those points back to the company in exchange for a Harrier fighter jet plane. John Leonard's comprehension of the ad's messaging led him to acquire the number of points necessary, and eventually, this led to a long legal battle with Pepsi when they refused to give him the plane. 

Suffice it to say, your customers’ comprehension and understanding of your messaging is very, very important. It could be the message around your branding that your customer sees through an ad image, or it could be as specific as your data privacy policy, or even financial information. It becomes crucial that this information is as clear as can be, for everyone's sake.

Building Your Test Plan

When building your test plan, you’ll want to keep in mind that your goal is to observe how a potential customer interacts with your content, and whether or not they fully understand the message they are meant to comprehend. First, present your content, and second, find out what they understood about this content and if they grasped your intended meaning- without your help. Below are some suggested ways to achieve this, based on the kind of task you create within your test (helpful for both unmoderated testing or Live Conversations):

• Task: Ask your test contributor to view a piece of content. Ask them to think out loud as they view this content, describing their process and opinions as they go.
• Verbal response: Ask test contributors to describe what they just saw, or read. Do they grasp the intended meaning? When analyzing the results, watch the video of their responses and look for trends in their answers.
• Written response: Ask test contributors to write a 1-2 sentence description of what they just viewed. When reviewing results, look for common topics, words, and phrases.
• Multiple choice: Ask test contributors to select an answer choice that aligns with their understanding of the concept you are testing. When viewing results, determine how many contributors selected the answer you were hoping they would. Be sure to watch the videos for those that didn’t select the “right” answer, to find out why they chose another meaning.
• Rating scale: Ask test contributors about their perceptions of how understandable the content was. When looking at results, review the average rating, and see if it aligns with your goal. If not, it is a good idea to check the verbal responses in the first task and question.

Templates

We have Templates (pre-written test plans) for your convenience, and to address your specific needs around the kind of testing you want to do. We have many templates related to messaging for comprehension, including:

• Messaging Comprehension
• Healthcare Messaging Comprehension
• Data Privacy Comprehension
• Racial Equality: Response to Messaging

While you are testing for comprehension, you can also ask for feedback that gives you insight into a customer’s comfort level with your content, or their confusion, challenges, trust, clarity, etc.

Use Cases

The following are just a few examples of organizational use cases, where comprehension testing becomes crucial to scaling upward for their business:

• A company that is branching out into a new geographical region, state, or country: for example, let’s say a European company that decides to enter the US market. They will want to make sure to test their messaging comprehension to ensure that potential US customers understand their branding and products, and to account for any cultural differences. To read about this real-life use case of a company that worked with UserTesting for comprehension testing, please see this article.
• Organizational scaling and expansion: if a company experiences a period of growth and will be potentially rolling out new paperwork, either internally or customer-facing, comprehension testing is crucial to ensure policies and legalities are followed.
• New product releases: if an organization is anticipating the release of a new product, or new version of a product, etc., it is critical that customers understand what they are purchasing.
• Gaming/entertainment: in the case of a gaming company for example, they might be introducing new characters, new landscapes, etc. It is important for the players of the game that characters, plots, and entire worlds are clear. This is true for any artistic practice as business also, such as film studios, fashion houses, etc.

Sample Test Plan

The following test plan, "Data Privacy: Policy Comprehension" can be found within our collection of Templates. This test plan concerns the kind of comprehension testing you might wish to do with a written policy for customers.

• Task: Please start by finding the privacy policy. Speak aloud as you do so. Once you find it or once 5 minutes have passed, move on to the next task.
• Task: Take as much time to review the privacy policy as you usually would. Please speak aloud about what stands out to you as you do so. Once you have reviewed the policy, move on to the next task.
• Rating scale [Strongly disagree - Strongly agree]: Please rate your level of agreement with the following statement: “It is clear what data is being collected and how the company will use it.” Please explain your rating.
• Rating scale [Strongly disagree - Strongly agree]: Please rate your level of agreement with the following statement: “The language used in the privacy policy is easy to understand.” Please explain your rating.
• Verbal response: What, if anything, makes it **easy** to understand the privacy policy?
• Verbal response: What, if anything, makes it **difficult** to understand the privacy policy?
• Rating scale [Strongly disagree - Strongly agree]: Please rate your level of agreement with the following statement: “I trust this company to handle my data responsibly.” Please explain your rating.
• Written response: What, if any, concerns do you have about how this company handles your data? Summarize your answer in the textbook then explain in detail aloud.
   

Please provide any feedback you have on this article. Your feedback will be used to improve the article and should take no more than 5 minutes to complete. Article evaluations will remain completely confidential unless you request a follow-up.