Preference testing

On this page:

• About preference testing
• When to use preference testing
• Considerations when conducting a preference test
• Setting up a preference test

About preference testing

Preference testing is used to have contributors compare two or more designs and choose their preference. A preference test can be used to compare:

• Product designs
• Website navigation
• Landing pages
• Images
• Advertisements
• Messaging

When to use preference testing

• Throughout the creative process—no matter what type of asset you're testing.
• For product designs and prototypes
• It's especially useful in the early stages of development when understanding the contributors' impressions of an overall design is of greater importance than whether the product or service functions properly. 

Here are some examples of research questions that often guide preference tests:

• Which product design/advertisement/message did contributors prefer?
• Which design did contributors find the most visually appealing? Why?
• Which messaging did contributors say was the most compelling? 

Considerations when conducting a preference test

Before setting up your test, you’ll want to consider a couple of things:

Consider the number of test assets

• Best practice is to keep the number of assets to no more than three.
• Testing more than three will make for a longer test and risk fatiguing contributors.
• More than three can be too time-consuming for researchers, too: testing four assets, for example, means you might have to run as many as 16 versions to account for order bias. 

Test like assets

• When choosing the assets to test, be sure to compare apples to apples—select two assets that are of the same type, such as an image pitted against another image, or a video against another video.
• You don’t want to test a video against a still image since contributors may select the option based on which medium, rather than which content, they prefer.

Consider the number of contributors

• If your goal is to distribute the test equally between two contributor groups, select an even number of contributors.
• If you're trying to choose a “winner” between the two assets, then go with an odd number of contributors.

Set up a preference test

1. To run a preference test, use the UserTesting platform’s Balanced comparison feature on the Test Plan Builder page when setting up your preference test.
2. Add preliminary tasks, such as warm-up questions or questions that validate your screener questions by eliciting details about a contributor's background.
3. Have contributors interact with the first asset, such as questions that ask about the functionality, clarity, or usability of the assets being compared.
4. Ask the same question for the second asset. Word this question exactly as you did for the first asset. Doing so helps ensure that you produce the most accurate comparison. 
5. Have contributors compare the assets/experiences. After they've seen the assets, contributors are ready to choose which asset or experience they prefer. 
   Note: You can add an image to remind the contributors of the options they're comparing. 

Related content