Gathering Factual Responses vs. Subjective Data

At a Glance

Asking the right questions in the right way is necessary to gain accurate insight from your user tests. This article has effective tips on how you can you gather factual and subjective data that you can trust.


When creating a user test, you need to ask the right question in the right way. You want results you can trust, on which you can confidently base decisions.

It’s important to structure questions logically and strategically. Doing so will maximize the impact of your user tests and generate insights that will really help move your project forward.

Here are some tips to help you ask questions in a way that produces a range of meaningful feedback and actionable data: 

Tips for gathering factual responses

1. Don’t use industry jargon

Terms like “sub-navigation” don’t make sense to the average user. Define key terms or concepts in the questions themselves (unless the goal of your study is to see if they understand these terms/concepts).

2. Define the timeline

If you are asking questions related to frequency—how often a user visits a particular site—make sure you define the timeline clearly. Always put the timeline at the beginning of the sentence.

Needs improvement: How often do you visit Amazon.com?

Improved: How often did you visit Amazon.com in the past six months? Best: In the past six months, how often did you visit Amazon.com?

3. Break up complicated questions

Don’t try to pack all the concepts into one question. Asking direct, single-focused questions will make your analysis more efficient.

4. Ask specific questions to get specific answers

If your respondent can give you the answer, “It depends,” then you probably need to make your question more specific.

5. Ask about first-hand experiences

Ask about what people have actually done, not what they will do or would do. An example might be: “Describe your most recent experience [depositing a check online; ordering groceries to be delivered;insert the activity/experience you are seeking to understand ].” 

Try your best to avoid hypotheticals and hearsay.

Tips for gathering subjective data

1. Leave breadcrumbs

Remind them where they should be on the site. To ensure they’re where you want them to be, provide a URL for them to click so they are looking at the right part of the site or app.

2. Don’t make the user feel guilty or dumb

When asking contributors after the test to evaluate their experience, phrase questions so that the burden of any struggles they may have had are placed squarely on the website, product, or app being test—and not on the contributor: 

Bad question example: “I was very lost and confused” (agree/disagree)
Good example: “The site caused me to feel lost and confused” (agree/disagree)

3. Make sure your rating scale questions aren’t skewed

Be fair, realistic, and consistent with the two ends of a rating spectrum.

Bad example: “After going through the checkout process, to what extent do you trust or distrust this company?” (I distrust it just a tad ←→ I trust it with my life) Good example: “After going through the checkout process, to what extent do you trust or distrust this company?” (I strongly distrust this company ←→ I strongly trust this company)

4. Subjective states are relative

“Happy” in one context can mean something very different than “happy” in another context. Be clear about whether “happy” means the opposite of neutral or the opposite of unhappy.

  • Option 1: Not Happy / Happy (Happy = Opposite of not happy)
  • Option 2: Unhappy / Neutral / Happy (Happy = The best!)
  • Option 3: Very Unhappy / Unhappy / Happy / Very Happy (Happy = Just above neutral)

5. Ask many specific questions, rather than a few all-encompassing questions

When you ask questions about vague or complex concepts, users often don’t know how to answer them. Break concepts up when asking the questions and put them back together when analyzing the results.

Learn More

Need more information? Read these related resources.

Want to learn more about this topic? Check out these University courses: 

 

Please provide any feedback you have on this article. Your feedback will be used to improve the article, and when you submit your survey you'll be entered into a drawing for a $50 Amazon gift card. This survey should take 5 minutes to complete. Article evaluations will remain completely confidential.

Was this article helpful?
10 out of 11 found this helpful