A/B and multivariate testing

Learn how UserTesting complements A/B and multivariate testing with qualitative feedback to uncover the "why" behind the results.

This article applies to: UserTesting 

On this page:

• About A/B testing and multivariate testing 
• Comparing variations
• Enhancing A/B and multivariate testing with UserTesting insights
• A/B testing vs preference testing

 

---

 

About A/B and multivariate testing

• A/B and multivariate tests compare multiple versions of the same asset (e.g., landing pages, shopping carts, navigation menus) on live sites.
• These tests are most effective when only one element differs between versions, often referred to as "split testing."
• Researchers analyze which version performs better based on specific outcomes, such as conversions or user engagement.
• A/B testing provides quantitative insights, helping identify which element drives better results (e.g., more completed checkouts).

 

Comparing variations

A/B testing

• Users see one of two design versions (A or B), and conversion rates are compared to determine the better-performing version.

Example

• Version A: A checkout page with an "Add to cart" CTA.
• Version B: The same page with a "Buy now" button.

Multivariate testing:

• A variation of A/B testing that compares three or more design versions to evaluate performance.

Example

• Version A: A checkout page with an "Add to cart" CTA and a green button.
• Version B: A checkout page with a "Buy now" CTA and a green button.
• Version C: A checkout page with an "Add to cart" CTA and a blue button.
• Version D: A checkout page with a "Buy now" CTA and a blue button.

 

 

Enhancing A/B and multivariate testing with UserTesting insights

• Your A/B test tells you WHICH version has a higher conversion rate; testing through UserTesting can tell you WHY that version was more successful. 
• UserTesting refrains from conducting traditional A/B or multivariate testing, which is inherently more quantitative.

But we can help inform your current or planned tests in the following ways:

Generate ideas of what to test with an A/B or multivariate test

• Run a usability test through the UserTesting Platform to quickly collect contributor feedback and identify weaknesses or areas of confusion with your site.
• This feedback can point you to those specific things that would best be served by A/B or multivariate testing.

Illuminate why a design performed better than another during an A/B or multivariate test:

• An A/B or multivariate test will tell you which version of a site had a higher conversion or success rate.
• Pairing this numbers-driven data with the qualitative insights found in the test recordings leads to a fuller understanding of why one version outperformed another/the others. 

 

 

A/B testing vs preference testing

Because both forms of testing involve testing two or more options, it’s easy to think that A/B and preference testing are interchangeable. But there are some key differences:

A/B testing	Preference testing
Objective and numbers-driven (quantitative). Whether one version “performs” better is the key measurement. User preference is not a criterion.	Subjective and driven by user preference (qualitative). This test measures which of the two or more options a user prefers.
Half of the contributors see Design A, and the other half see Design B.	Contributors see both designs and evaluate which one they prefer.
Demands a larger number of test contributors. This ensures a statistically accurate and reliable sample.	Can be conducted with fewer contributors, but the results are not as statistically significant.
Typically conducted on a live site or finished product.	Can be conducted at any stage of product development. Often valuable at the beginning stages of the design process.

 

 

Related content

Want to learn more? Check out these Knowledge Base articles...  Preference testing Choosing a method for collecting feedback	Interested in growing your skills? Check out our University courses... Approaches to testing: Comparison testing How to use Balanced comparison
Need hands-on training? Find a live, virtual class that fits your needs. Enroll in one of our UserTesting certification programs.	Can't find your answer? Check out our CommUnity forums. Reach out to Support or our Professional Services team.