Many UserTesting customers create tests that compare two alternative designs. For example, two different web pages, prototypes, or even marketing emails. The order in which those alternatives are presented can sometimes influence a participant’s reactions, creating a subtle bias within a test. The Balanced Comparison feature is designed to reduce the risk of that bias.
When you use Balanced Comparison, you create two sections in your study, part A and Part B. The order in which participants see the two parts is automatically alternated: the first participant will see option A first and B second, the next will see option B first and option A second, and so on. This balances any biases created by the order in which the two alternates are presented.
How it works
Create a new test and select your participants. You can use Balanced Comparison to test any website, prototype, or Android app. (The feature is currently unavailable for iOS app tests due to restrictions by Apple.) You can also use our Balanced Comparison test template.
Select your tasks and questions
To use Balanced Comparison, go to the question selector and click Balanced Comparison.
You'll see a new part of your study labeled Group A and Group B. Drag tasks and questions into these areas normally. You can also put questions and tasks before and after the balanced part of the study.
Note: When comparing two different URLs we recommend beginning on a blank page, then placing the first URL in section A and the second URL in section B.
Launch your test
When you're done, launch your test as you would normally. When participants take the test, the first participant will see the balanced section in A/B order. The next participant will see it in B/A order. They'll continue to alternate until the test is completely filled, or you turn it off. This way you'll receive a balance of A/B and B/A responses even if you stop the study early.
Review your videos
Completed tests will appear on your dashboard as they would normally.
You can see which version a test participant saw by selecting Metrics from your study dashboard:
You can also see this information by checking the participant's info in the video player:
You can view various individual metrics in the drop down menu next to the video player.
The order of tasks in an individual study is displayed in the Tasks tab in the video player.
Tell us what you think!
We’d love to get your feedback on balanced comparison. Tell us what you like and what we could do better, by e-mailing firstname.lastname@example.org.