What is Preference Testing?

When to Use Preference Testing

Preference testing is used to have contributors compare two or more designs and choose their preference. A preference test can be used to compare:

• Product designs
• Website navigation
• Landing pages
• Images 
• Advertisements
• Messaging

You can test, for example, a website landing page against a competitor’s landing page. Or you can test whether contributors prefer a new version of an app feature to the one you already have in place. These are but a couple of scenarios in which a preference test can well serve your testing goal.

Preference testing can be conducted throughout the creative process, no matter what type of asset you’re testing. For product designs and prototypes, such testing is especially useful in the early stages of development, when understanding the contributors’ impressions of an overall design is of greater importance than whether the product or service functions properly. 

Here are examples of research questions that often guide preference tests:

• Which product design/advertisement/message did contributors prefer?
• Which design did contributors find the most visually appealing? Why?
• Which messaging did contributors say was the most compelling? 

Considerations When Conducting a Preference Test

Before setting up your test, you’ll want to consider a couple of things:

• Consider the number of test assets: Best practice is to keep the number of assets to no more than three. Testing more than that will make for a longer test and risk fatiguing contributors. And it’s time-consuming for researchers too: testing four assets, for example, means you might have to run as many as 16 versions to account for order bias. 
• Test like assets: When choosing the assets to test, be sure to compare apples to apples—select two assets that are of the same type, such as an image pitted against another image, a video against another video. You don’t want to test a video against a still image since contributors may select the option based on which medium, rather than which content, they prefer.
• Consider the number of contributors: Let’s say you’re running a preference test featuring two assets/options. If your goal is to distribute the test equally between two contributor groups, select an even number of contributors. But if you’re aim is to choose a “winner” between the two assets, then go with an odd number of contributors.

Setting up a Preference Test

To run a preference test, use the UserTesting Platform’s Balanced comparison feature on the Test Plan Builder page when setting up your preference test.

Activating this tool helps to minimize the influence of order bias—the order in which things appear can influence how contributors perceive their value. (Some people’s inclination is to favor the second option of two things they are shown, thinking that the most recent thing they see is newer or better). If contributors are choosing between two images, for example, the Balanced comparison tool will counterbalance the test for you by presenting Image A first to half of the contributors and Image B first to the other half.

After activating the Balanced comparison tool, and having decided on which assets to test and the number of contributors to test them, the remainder of your preference-test planning should follow this flow:  

• Add preliminary tasks: Most tests start with such tasks, which appear before a contributor explores the assets or options being compared. Such tasks can both serve as warm-up questions and validate your screener questions by eliciting details about a contributor’s background. 
• Have contributors interact with the first asset: Here, you will ask questions about the functionality, clarity, or usability of the assets being compared. (Which area you focus on can vary based on the type of asset being tested.) 
• Ask the same question for the second asset: Word the question for the second asset exactly as you did the first. Doing so helps ensure that you produce the most accurate comparison.
• Have contributors compare the assets/experiences. Once they’ve seen the assets, contributors are ready to choose which asset or experience they prefer. If necessary, add an image to remind the contributors of the options they are comparing.

Please provide any feedback you have on this article. Your feedback will be used to improve the article and should take no more than 5 minutes to complete. Article evaluations will remain completely confidential unless you request a follow-up.