Balanced Comparison with multiple concepts

The Balanced Comparison feature is currently set up to support studies with two concepts. Here's a strategy for how you can set up Balanced Comparison to test more than two concepts

 

This article applies to: UserTesting 

 

On this page:

• Balanced Comparison with three concepts
• Balancing contributors across two studies with three concepts
• Balanced Comparison with four concepts
• Balanced Comparison with five concepts
• Balance contributors across three studies with five concepts
• Balanced Comparison with six concepts

 

---

 

Balanced Comparison with three concepts

Here's how to set up a Balanced Comparison with three concepts.

 

3 Concept setup v1a

1. Set up a website study.
2. Add a Balanced Comparison task group.
3. Add the 1st concept in Group A Tasks.
4. Add the 2nd concept in Group B Tasks.
5. Add a Custom task group after the Balanced Comparison and add the 3rd concept to this task group.
   Note: Task groups are limited to five tasks in total. To ensure balance across all concepts, each group should have no more than five tasks.

First study (3 Concept setup v1a)

 

3 Concept setup v1b

1. Create a duplicate of the 3 Concept setup v1a test.
2. Move the Concept 3 Task group above the Balanced Comparison. 
   Note: The contributors should be balanced across the studies.
   

Second study (3 Concept setup v1b)

 

Explanation:

• Two studies will need to be run for 3 concepts (let's call them Concepts 1, 2, and 3).
• The first study (3 Concept Setup v1a) will have Concept 1 in Group A tasks, Concept 2 in Group B Tasks, and Concept 3 as a Custom task group after the Balanced Comparison. 
• The second study (3 Concept Setup v1b) will have Concept 3 preceding the Balanced Comparison in a Custom task group, with Concepts 1 and 2 remaining in Group A and B tasks, respectively.

 

This is how contributors will be shown your concepts:

Study 1 - Half the contributors will see Concepts 1, 2, and 3, and the other half will see Concepts 2, 1, and 3.

Study 2 - Half the contributors will be shown in Concepts 3, 1, and 2, and the other half in Concepts 3, 2, and 1.

 

 

Balancing contributors across two studies with three concepts

• To remove bias, we suggest balancing the number of contributors between the three concepts so that an equal number of contributors will see each concept first.
• Since Study 2 will always show the 3rd concept first, it will only need half as many contributors as Study 1.
• Example:
  • If you want results from 6 contributors in total (2 for each concept first), then please launch Study 1 with 4 contributors and Study 2 with 2 contributors.
  • If you want results from 12 contributors (4 contributors see each concept first), we suggest having 8 contributors take Study 1, and 4 contributors take Study 2.
  • This way, the same number of contributors sees each concept first.

6 Contributors (2 for each concept): Study 1 = 4 contributors Study 2 = 2 contributors

12 Contributors (4 for each concept): Study 1 = 8 contributors Study 2 = 4 contributors

 

 

Balanced Comparison with four concepts

Here's how to set up a Balanced Comparison with four concepts.

 

4 Concept setup v1a

1. Set up a website study.
2. Add the Balanced Comparison task group.
3. Add the 1st and 2nd concept in Group A Tasks.
4. Add the 3rd and 4th concept in Group B Tasks.
   Note: Task groups are limited to five tasks in total. To ensure balance across all concepts, each group should have no more than five tasks.

First study (4 Concept setup v1a)

 

4 Concept setup v1b

1. Create a duplicate of the 4 Concept setup v1a test.
2. Switch the tasks for the 1st and 2nd concept around, so the 2nd concept is first.
3. Switch the tasks for the 3rd and 4th concept around, so the 4th concept is first.
   Note: An equal number of contributors should take each study.

Second study (4 Concept Setup v1b)

 

Explanation:

• For 4 concepts (let's call them Concepts 1, 2, 3, and 4), two studies will need to be run.
• The first study will have Concepts 1 and 2 in Group A tasks, and Concepts 3 and 4 in Group B tasks.
• The second study will have Concepts 2 and 1 in Group A tasks and Concepts 4 and 3 in Group B tasks.

 

This is how contributors will be shown your concepts:

Study 1 - Half the contributors will see 1, 2, 3, and 4, and the other half will see 3, 4, 1, and 2.

Study 2 - Half the contributors will see 2, 1, 4, and 3, and the other half will see 4, 3, 2, and 1.

 

Note: An equal number of contributors should take each study.

 

 

Balanced Comparison with five concepts

Here's how to set up a Balanced Comparison with four concepts.

 

5 Concept setup v1a

1. Set up a website study.
2. Add a Balanced Comparison task group.
3. Add the 1st and 2nd concept in Group A Tasks.
4. Add the 3rd and 4th concept in Group B Tasks.
5. Add a Custom task group after the Balanced Comparison and add the 5th concept to this task group.
   Note: Task groups are limited to five tasks in total. To ensure balance across all concepts, each group should have no more than five tasks.

First study (5 Concept setup v1a)

 

5 Concept Setup v1b

1. Create a duplicate of the 5 Concept setup v1a test.
2. Switch the tasks for the 1st and 2nd concept around, so the 2nd concept is first.
3. Switch the tasks for the 3rd and 4th concept around, so the 4th concept is first.

Second study (5 Concept setup v1b)

 

5 Concept setup v1c

1. Create a duplicate of the 5 Concept setup v1a test.
2.  Move the Concept 5 task group above the Balanced Comparison. 
   Note: The contributors should be balanced across the studies.
   

Third study (5 Concept v1c)

 

Explanation:

• For 5 concepts (let's call them Concepts 1, 2, 3, 4, and 5), three studies will need to be run.
• The first study will have Concepts 1 and 2 in Group A Tasks, Concepts 3 and 4 in Group B Tasks, and Concept 5 in the subsequent custom task group.
• The second study will have Concepts 2 and 1 in Group A Tasks, Concepts 4 and 3 in Group B Tasks, and Concept 5 in the subsequent custom task groups.
• The third study will have concept 5 in the custom task group preceding the other 4 concepts in Groups A and B.
  
  

 

This is how contributors will be shown your concepts:

Study 1 - Half the contributors will see 1, 2, 3, 4, and 5, and the other half will see 3, 4, 1, 2, and 5.

Study 2 - Half the contributors will see 2, 1, 4, 3, and 5, and the other half will see 4, 3, 2, 1, and 5.

Study 3 - Half the contributors will see 5, 1, 2, 3, and 4, and the other half will see 5, 3, 4, 1, and 2.

 

 

Balance contributors across three studies with five concepts

• To remove bias, we suggest balancing the number of contributors between the five concepts so that an equal number of contributors will see each concept first.
• Since the third study always shows the 5th concept first, then it only needs half as many contributors as the first and second study.
• If you want results from 10 contributors total (2 for each concept), then please ensure that Study 1 and 2 have 4 contributors each, while Study 3 should have 2 contributors.
• If you want results from 20 (4 for each concept) contributors total, then please ensure that 8 contributors take Study 1 and 2, and 4 take Study 3.
• This way, the same number of contributors sees each concept first.

10 Contributors (2 for each concept): Study 1 = 4 contributors Study 2 = 4 contributors Study 3 = 2 contributors

20 Contributors (4 for each concept): Study 1 = 8 contributors Study 2 = 8 contributors Study 3 = 4 contributors

 

 

Balanced Comparison with six concepts

Here's how to set up a Balanced Comparison with six concepts.

 

6 Concept setup v1a

1. Set up a website study.
2. Add a Balanced Comparison task group.
3. Add the 1st, 2nd and 3rd concept in Group A Tasks.
4. Add the 4th, 5th and 6th concept in Group B Tasks.
   Note: Task groups are limited to five tasks in total. To ensure balance across all concepts, each group should have no more than five tasks.

First study (6 Concept setup v1a)

 

6 Concept Setup v1b

1. Create a duplicate of the 6 Concept setup v1a test.
2. Switch the tasks for the 1st concept to the bottom of Group A Tasks so that the order is concepts 2 3 1.
3. Switch the tasks for the 4th concept to the bottom of Group B Tasks so that the order is concepts 5 6 4.

Second study (6 Concept setup v1b)

 

6 Concept setup v1c

1. Create a duplicate of the 6 Concept setup v1b test.
2. Switch the tasks for the 2nd concept to the bottom of Group A Tasks, so that the order is concepts 3, 1, 2.
3. Switch the tasks for the 5th concept to the bottom of Group B Tasks, so that the order is concepts 6, 4, 5.
   Note: An equal number of contributors should take each study.

Third study (6 Concept Setup v1c)

 

Explanation:

• For 6 concepts (let's call them Concepts 1, 2, 3, 4, 5, and 6), three studies will need to be run.
• The first study will have Concepts 1, 2, and 3 in Group A Tasks, and Concepts 4 and 5, and Concept 6 in Group B Tasks.
• The second study will have Concepts 2, 3, and 1 in Group A Tasks, Concepts 5 and 6, and Concept 4 in Group B Tasks.
• The third study should have Concepts 3, 1, and 2 in Group A Tasks, Concepts 6, 4, and 5 in Group B Tasks.

 

This is how contributors will be shown your concepts:

Study 1 - Half the contributors will see 1, 2, 3, 4, 5, and 6, and the other half will see 4, 5, 6, 1, 2, and 3.

Study 2 - Half the contributors will see 2, 3, 1, 5, 6, and 4, and the other half will see 5, 6, 4, 2, 3, and 1.

Study 3 - Half the contributors will see 3, 1, 2, 6, 4, and 5, and the other half will see 6, 4, 5, 3, 1, and 2.

 

 

Related content

Want to learn more? Check out these Knowledge Base articles...  Balanced Comparison Task groups and results	Interested in growing your skills? Check out our University courses... Using Balanced Comparison What sample size do I need?
Need hands-on training? Find a live, virtual class that fits your needs. Enroll in one of our UserTesting certification programs.	Can't find your answer? Check out our CommUnity forums. Reach out to Support or our Professional Services team.

 

Please provide any feedback you have on this article. Your feedback will be used to improve the article and should take no more than 5 minutes to complete. Article evaluations will remain completely confidential unless you request a follow-up.