What Is a Benchmark Study?

At a Glance

A benchmark study measures and compares usability metrics against a baseline study. Such studies are typically run on a regular basis (monthly, quarterly, yearly) and evaluate how a user's experience with your product changes over time.

 

How many contributors?

Benchmark studies must have larger sample sizes, compared to basic qualitative (or formative) usability studies, in order to make claims about positive and negative changes.

The recommended sample size for benchmark and baseline studies depends on a handful of factors. For more details, see How do you decide how many users to have in a study.

A baseline study is the first in the series of benchmark studies. It is your starting point and what you will compare additional benchmark study results to.

Most benchmark studies focus on basic usability requirements, such as task success, errors, time on task, and subjective user ratings (satisfaction, ease of use, confidence in task completion).

Why are benchmarking studies helpful?

Benchmarking allows you to track your progress over time, thus giving you an indication of how successful site changes and revisions have been.

A benchmark study is considered a summative assessment, so greater attention is paid to methodology details, tasks, protocol, and analysis. For example, it’s important to choose tasks that can be reused over time so that a fair comparison can be made.

When should a baseline or benchmark study be conducted?

Baseline studies can be conducted at any time, but they are particularly valuable when they are started before any redesign efforts, in order to show the benefit of your redesign.

For example, you can run a baseline study on an existing site or product, and then run additional benchmarking studies on a regular basis as the new design becomes available. This will allow you to track changes in usability metrics over time, as well as compare usability metrics from the old design to the new design.

Contact your Customer Success Manager to plan a baseline or benchmark study.

 

Learn More

Need more information? Read these related articles.

Want to learn more about this topic? Check out our University courses.

Was this article helpful?
33 out of 41 found this helpful