Structure questions for factual responses and subjective data

This article applies to: UserTesting 

On this page:

• Tips for gathering factual responses
• Tips for gathering subjective data

Tips for gathering factual responses

1. Don't use industry jargon

• Terms like “sub-navigation” don’t make sense to the average user.
• Define key terms or concepts in the questions themselves (unless the goal of your study is to see if they understand these terms/concepts).

2. Define the timeline

• If you are asking questions related to frequency—how often a user visits a particular site—make sure you define the timeline clearly.

• Always put the timeline at the beginning of the sentence.

  • Needs improvement: How often do you visit Amazon.com?

  • Improved: How often did you visit Amazon.com in the past six months?

  • Best: In the past six months, how often did you visit Amazon.com?

3. Break up complicated questions

• Don’t try to pack all the concepts into one question.
• Asking direct, single-focused questions will make your analysis more efficient.

4. Ask specific questions to get specific answers

• If your respondent can give you the answer, “It depends,” then you probably need to make your question more specific.

5. Ask about first-hand experiences

• Ask about what people have actually done, not what they will do or would do.

• An example might be: “Describe your most recent experience [depositing a check online; ordering groceries to be delivered; insert the activity/experience you are seeking to understand].”

• Try your best to avoid hypotheticals and hearsay.

Tips for gathering subjective data

1. Leave breadcrumbs

• Remind them where they should be on the site.
• To ensure they’re where you want them to be, provide a URL for them to click so they can see the right part of the site or app.

2. Don't make the user feel guilty or dumb

• When asking participants after the test to evaluate their experience, phrase questions so that the burden of any struggles they may have had are placed squarely on the website, product, or app being test—and not on the participant:

  • Bad question example: “I was very lost and confused” (agree/disagree)

  • Good example: “The site caused me to feel lost and confused” (agree/disagree)

3. Make sure your rating scale questions aren't skewed

• Be fair, realistic, and consistent with the two ends of a rating spectrum.

  • Bad example: “After going through the checkout process, to what extent do you trust or distrust this company?” (I distrust it just a tad ←→ I trust it with my life)

  • Good example: “After going through the checkout process, to what extent do you trust or distrust this company?” (I strongly distrust this company ←→ I strongly trust this company)

4. Subjective states are relative

• “Happy” in one context can mean something very different than “happy” in another context.

• Be clear about whether “happy” means the opposite of neutral or the opposite of unhappy.

  • Option 1: Not Happy / Happy (Happy = Opposite of not happy)

  • Option 2: Unhappy / Neutral / Happy (Happy = The best!)

  • Option 3: Very Unhappy / Unhappy / Happy / Very Happy (Happy = Just above neutral)

5. Ask many specific questions rather than a few all-encompassing questions

• When you ask questions about vague or complex concepts, users often don’t know how to answer them.
• Break concepts up when asking the questions and put them back together when analyzing the results.

Related content