Using the Balanced Comparison Feature

Balanced Comparison is available on the following subscriptions:

Many UserTesting customers create tests that require a test contributor to compare two designs, such as: two different web pages, two prototypes, or even two marketing emails. The order in which those designs are presented can sometimes influence a contributor’s reactions, creating a subtle bias within a test. The Balanced Comparison feature is designed to reduce the risk of that bias.

When you use Balanced Comparison, you create two sections in your study, Part A and Part B. The order in which contributors see the two parts is automatically alternated: the first contributor will see option A first and B second, the next will see option B first and option A second, and so on. This balances any biases created by the order in which the two alternates are presented. A recommended sample size is 12.

Note: Balanced Comparison is only available on the Advanced and Ultimate edition of the Flex plan and on the Professional and Premium editions of the Seat-based plan.

How It Works

Create a new test and select your contributors. You can use Balanced Comparison to test any website, prototype, or app. You can also go to the Template gallery for further guidance.

Select your tasks and questions

To use Balanced Comparison, begin building your test plan and toggle Balanced Comparison from off to on.

You'll see a new part of your study labeled Group A and Group B. Drag tasks and questions into these areas normally. You can also put questions and tasks before and after the balanced part of the study.  

Note: When comparing two different URLs we recommend beginning on a blank page, then placing the first URL in section A and the second URL in section B.

Launch your test

When you're done, launch your test as you would normally. When contributors take the test, the first contributor will see the balanced section in A/B order. The next contributor will see it in B/A order. They'll continue to alternate until the test is completely filled, or you turn it off. This way you'll receive a balance of A/B and B/A responses even if you stop the study early.

Review your videos

Completed tests will appear on your dashboard as they would normally.

You can see which version a test contributor saw by selecting Metrics from your test dashboard:

The order of tasks in an individual study is displayed in the Tasks tab in the video player.

Please provide any feedback you have on this article. Your feedback will be used to improve the article, and when you submit your survey you'll be entered into a drawing for a $50 Amazon gift card. This survey should take 5 minutes to complete. Article evaluations will remain completely confidential.