Balanced comparison for the New experience

On this page:

• About the Balanced comparison feature
• Add Balanced comparison to your survey
• Add Balanced comparison to your think-out-loud test
• Balanced comparison results
• Balanced comparison use cases

About the Balanced comparison feature

• Balanced Comparison is available for Surveys, Interaction tests, and Think-out-loud tests. 
• Can be used to test a website, app, or prototype.
• When you set up a Balanced Comparison study, you create 2 to 4 groups, for example, Group A and Group B, and choose a randomization method that controls how participants are assigned to those groups.

Randomization methods

Random order (within-participant)

• Every participant sees all groups
• The order is rotated across participants to reduce sequence bias
• Best for direct comparisons where you want each person to evaluate all options
• Requires fewer participants, since each person contributes data across all groups

Split test (between-participant)

• Each participant sees only one group, assigned evenly
• Prevents exposure to multiple versions from influencing responses
• Best for first impressions, preference tests, sensitive content, or A/B experiments
• Requires more participants, since each person contributes data for only one group

Note: You choose your randomization method when building your test. The mode is locked once you launch and cannot be changed.

 

Add Balanced comparison to your survey

1. In the Test builder, select the + Add button.
2. Under the Groups and randomize section, select Balanced Comparison.
   
    
3. Choose your group: Image or Question page.
   
   
   Note: Balanced comparison will be in an empty state, and launching is disabled until a group is chosen and task information is completed.
    
4. Add two to four groups for participants to see.
5. Select your randomization method: Random order or Split test.
   
6. Complete your task information and launch.
   Note: Balanced Comparison will be in an empty state, and launching is disabled until a group type is chosen, task information is completed, and a randomization method is selected.
   
   Important: You must have at least two groups.

Add Balanced comparison to your think-out-loud test

1. In the Test builder, select the + Add button.
2. Under the Groups and randomize section, select Balanced Comparison.
   
    
3. Choose your group: Navigation task, Figma task, Image, or Question page
   
   
   Note: Balanced comparison will be in an empty state. Launching is disabled until a group is chosen and task information is completed.
    
4. Add two to four groups for participants to see.
   
   
   Important: You must have at least two groups.
    
5. Select your randomization method: Random order or Split test.
   
6. Complete your task information and launch.

Notes: 

• Balanced Comparison will be in an empty state and launching is disabled until a group type is chosen, task information is completed, and a randomization method is selected. 
• You cannot change the comparison method after launch:
  

Balanced comparison results

How your results appear depends on the randomization method you chose.

Random order results

When using Random order, results are organized by sequence. The specific order in which groups were shown to each participant.

• Sequence cards appear in the Overview section
• Results appear in a Sequence distribution, showing how participants were spread across different sequences
• Balanced Comparison uses a structured rotation (sometimes called a Latin square) to ensure each group appears in every position an equal number of times — this is designed to reduce order bias while keeping studies efficient
   

Sequences also appear in response tables for:

• Written and verbal responses
• Matrix questions
• NPS (written responses)
• Navigation or Figma task instructions

How videos work with Random order:

• Each participant's video follows the sequence assigned to them, so tasks appear in the same order shown in the sequence table. 
• For example, if you have four groups with one task each and a participant's sequence is BCAD, the task from Group A is third in their video. 
• When you open that task, the video starts at that point rather than at the beginning.

Split test results

When using Split test, results are organized by group, showing what percentage of participants were assigned to each group.

• The Sequence distribution card shows participant distribution across groups rather than sequences
• Use Group level filtering to view results for a specific group
• The Participant group column in exports shows which group each participant was assigned to

You can also filter based on Sequence directly from main page: 

How videos work with Split test: Each participant's video contains only the tasks for the group they were assigned to.

Balanced comparison use cases

Related content