Avoiding Bias When Collecting Feedback

Overview

Generally speaking, contributors will want to succeed when taking your tests, to answer all the questions and complete all the tasks that make up your test. Since you know much more about what is going to happen during the session than does the contributor, they will therefore try to pick up from you any cues that help them achieve this objective.

These cues can be anything from tone of voice to body language to facial expressions. But such tells or indicators can influence how the contributors answer questions or perform tasks in such a way that the resulting findings are compromised and made unreliable. With a clearer understanding of how such unintentional elements can bias your test, you'll to during a session avoid frowning, gasping, or furrowing your brow. Rather, you want to maintain an interested but largely noncommital expression, and a neutral posture.

One of the benefits of unmoderated tests is that the absence of a moderator all but eliminates the likelihood that verbal and nonverbal signals will unduly influence the contributors when taking the test.

However, the way you word your tasks can introduce bias. There are two primary aspects of tasks that can introduce bias:

• Being overly directive
• Failing to use neutral language

Avoid being overly directive

Most bias that surfaces in task wording is due to overly explicit instructions about how to complete a task. When writing activities for your test, provide a goal to be accomplished, but don’t give away the answer or have respondents simply follow your instructions.

You can always follow up on goal-directed tasks with more directive tasks. For example, the follow-up task in the “Better example” is more directive, with the intent of exercising the sort controls. Notice that it still does not use the word “sort,” because that matches the label of the control on the page.

Use neutral language

Asking "leading" questions, in which you inadvertently point the contributor to the correct answer, is another bias-inducing practice you want to avoid. 

Here are some examples of leading questions:

• “How much better is the new version than the original homepage?”
• “Was it hard to find the Preferences page?”
• “What would you improve about this page?”

Note that even subtle wording can bias the respondent. These questions assume the following

• The new version is better.
• Finding the Preferences page is hard.
• The page needs improvement.

Use balanced language to address this issue. In so doing, your questions will elicit more accurate feedback.

• “How does the new version of the homepage compare to the original?”
• “How difficult or easy was it to find the Preferences page?”
• “What, if anything, would you improve about this page?”

Please provide any feedback you have on this article. Your feedback will be used to improve the article and should take no more than 5 minutes to complete. Article evaluations will remain completely confidential unless you request a follow-up.