Avoiding Bias When Collecting Feedback

At a Glance

The wording of the tasks and questions in your test can inadvertently introduce bias that affects the feedback you receive. This bias can change that feedback, and result in insights that are not accurate.

 

Overview

People generally want to be successful and do the right thing. Therefore, they will want to successfully complete the tasks in your test and they will notice any cues you provide about how to be successful.

For example, when you are leading a live session, you know much more about what is going to happen during the session than the person you are talking to. They will look to you for cues and feedback, such as your body language, smiles, and frowns. Avoid frowning, gasping, or furrowing your brow. Maintain a smile, an interested expression, and a neutral posture.

One of the benefits of unmoderated tests is that you can reduce the chance of influencing the people completing the test. However, the way you word your tasks can introduce bias.

There are two primary aspects of tasks that can introduce bias:

  • Being overly directive
  • Failing to use neutral language

 

Avoid being overly directive

Most bias that surfaces in task wording is due to overly explicit instructions about how to complete a task. When writing activities for your test, provide a goal to be accomplished, but don’t give away the answer or have respondents simply follow your instructions.

Poor example: Better example:
“Go to the filters and select on size XL, then sort by price to find the most inexpensive sweater that is a size XL.”

"Find a sweater under $100 in size XL."

—THEN—

"If you haven’t already done so, organize the list of sweaters to display from highest price to lowest price."

This example is too explicit. It tests whether someone can follow your directions, rather than if they can use the design. This better example sees whether the respondents even try to use the sort and filter controls.

 

You can always follow up on goal-directed tasks with more directive tasks. For example, the follow-up task in the “Better example” is more directive, with the intent of exercising the sort controls. Notice that it still does not use the word “sort,” because that matches the label of the control on the page.

 

Use neutral language

When posing questions, you want to avoid biasing the respondent towards a particular answer.

Here are some examples of leading questions:

  • “How much better is the new version than the original homepage?”
  • “Was it hard to find the Preferences page?”
  • “What would you improve about this page?”

Note that even subtle wording can bias the respondent. These questions assume the new version is better, finding the Preferences page is hard, and that the page needs improvement.

Use balanced language to elicit more accurate feedback

  • “Compare the new version of the homepage to the original.”
  • “How difficult or easy was it to find the Preferences page?”
  • “What, if anything, would you improve about this page?”

 

Learn More

Need more information? Read these related articles.

Want to learn more about this topic? Check out our University courses.

Was this article helpful?
0 out of 0 found this helpful