At a Glance
The wording of the tasks and questions in your test can inadvertently introduce bias that can affect the feedback you receive and lead to inaccurate results and conclusions.
Generally speaking, contributors will want to succeed when taking your tests, to answer all the questions and complete all the tasks that make up your test. Since you know much more about what is going to happen during the session than does the contributor, they will therefore try to pick up from you any cues that help them achieve this objective.
These cues can be anything from tone of voice to body language to facial expressions. But such tells or indicators can influence how the contributors answer questions or perform tasks in such a way that the resulting findings are compromised and made unreliable. With a clearer understanding of how such unintentional elements can bias your test, you'll to during a session avoid frowning, gasping, or furrowing your brow. Rather, you want to maintain an interested but largely noncommital expression, and a neutral posture.
One of the benefits of unmoderated tests is that the absence of a moderator all but eliminates the likelihood that verbal and nonverbal signals will unduly influence the contributors when taking the test.
However, the way you word your tasks can introduce bias. There are two primary aspects of tasks that can introduce bias:
- Being overly directive
- Failing to use neutral language
Avoid being overly directive
Most bias that surfaces in task wording is due to overly explicit instructions about how to complete a task. When writing activities for your test, provide a goal to be accomplished, but don’t give away the answer or have respondents simply follow your instructions.
|Poor example:||Better example:|
|"Go to the filters and select on size XL, then sort by price to find the most inexpensive sweater that is a size XL."||
"Find a sweater under $100 in size XL."
"If you haven’t already done so, organize the list of sweaters to display from highest price to lowest price."
|The example above is too explicit. It tests whether someone can follow your directions, rather than if they can use the design.||This better example sees whether the respondents even try to use the sort and filter controls.|
You can always follow up on goal-directed tasks with more directive tasks. For example, the follow-up task in the “Better example” is more directive, with the intent of exercising the sort controls. Notice that it still does not use the word “sort,” because that matches the label of the control on the page.
Use neutral language
Asking "leading" questions, in which you inadvertently point the contributor to the correct answer, is another bias-inducing practice you want to avoid.
Here are some examples of leading questions:
- “How much better is the new version than the original homepage?”
- “Was it hard to find the Preferences page?”
- “What would you improve about this page?”
Note that even subtle wording can bias the respondent. These questions assume the following
- The new version is better.
- Finding the Preferences page is hard.
- The page needs improvement.
Use balanced language to address this issue. In so doing, your questions will elicit more accurate feedback.
- “How does the new version of the homepage compare to the original?”
- “How difficult or easy was it to find the Preferences page?”
- “What, if anything, would you improve about this page?”
Need more information? Read these related articles.
Want to learn more about this topic? Check out our University courses.