What is a benchmark study?
A benchmark study measures and compares usability metrics against a baseline study. Benchmark studies are typically run on a regular basis (monthly, quarterly, yearly, etc.) to evaluate how your product’s experience has changed over time. These studies must have larger sample sizes, as compared to a basic qualitative (or formative) usability study, to make claims about positive and negative changes.
The recommended sample size for benchmark and baseline studies depends on a handful of factors. For more details, see How do you decide how many users to have in a study.
A baseline study is the first in the series of benchmark studies. It is your starting point and what you will compare additional benchmark study results to.
Most benchmark studies focus on basic usability requirements, such as task success, errors, time on task, and subjective user ratings (satisfaction, ease of use, confidence in task completion).
Why are benchmarking studies helpful?
Benchmarking allows you to track your progress over time, thus giving you an indication of how successful site changes and revisions have been.
A benchmark study is considered a summative assessment, so there is greater attention to methodology details, tasks, protocol, and analysis. For example, it’s important to choose tasks that can be reused over time so a fair comparison can be made.
When should a baseline or benchmark study be conducted?
Baseline studies can be conducted at any time, but they are particularly valuable when they are started before any redesign efforts, in order to show the benefit of your redesign.
For example, you can run a baseline study on an existing site or product, and then run additional benchmarking studies on a regular basis as the new design becomes available. This will allow you to track changes in usability metrics over time, as well as compare usability metrics from the old design to the new design.
Contact your Customer Success Manager to plan a baseline or benchmark study.