Using Rating Scale Questions

When asking contributors to rate an experience or align their feedback with a value, you want to ensure that your rating scale offers the right granularity in response options. This ensures that contributors can select the option that best matches their opinion, resulting in more statistically accurate feedback for you.

How It Works

When building your test plan, you can select a Rating scale task under Tasks and Questions.

When creating questions using a Rating Scale, you have a number of options. You can select from a 7, 9, or 11-point scale, in addition to the default 5-point scale. You can choose from a menu of pre-written endpoints, or add your own custom endpoints.

When viewing your test within the Metrics page, you’ll see the link to the video player. To view the contributor associated with each video link, just hover your cursor over the link and the contributor name will display as a pop-up.

Here are some common examples of rating scale questions:

• How difficult (1) or easy (5) was it to log in to the app using Facebook? (1=Very difficult, 5=Very easy)
• How unlikely (1) or likely (5) would you be to recommend this game to a friend? (1=Not at all likely, 5=Extremely likely)
• How cluttered (1) or organized (5) do you consider the home page? (1=Cluttered, 5=Organized)
• How untrustworthy (1) or trustworthy (5) do you consider the site? Please explain your answer aloud. (1=Not at all trustworthy, 5=Entirely trustworthy)
• How disinterested (1) or interested (5) are you in purchasing boosters for this game? (1=Not at all interested, 5=Extremely interested)
• Please rate your agreement with the following statement: “I understand who this site is for.” (1=Strongly disagree, 5=Strongly agree)

Tips to keep in mind when writing rating scale questions:

1. Clarify the endpoints (the labels for 1 and 5, 7, 9, or 11, depending on which point-scale you selected) of your scale

Rating scales are highly adaptable and can be used to measure many different things, but only if the people taking your test understand what you’re asking them to measure!

Use the dropdown in the question to select pre-written endpoint levels, or create your own.

2. Include both ends of the scale in your question in addition to the endpoint fields

People are naturally eager to please, so if you ask contributors how easy it was to accomplish something, their inclination is to say it was easy—even if they struggled.

To avoid introducing bias into the response, it’s best to mention both ends of the spectrum as you ask your question (in this example, how difficult or easy). Including the both sides of the scale in the body of your question also reinforces the endpoints you’ve defined and lowers the chance that a contributor will misread your question.

3. 1 should always be the “pain” point of the scale, and 5, 7, 9, or 11 (whichever point-scale you selected) should always be the “positive” point

Because we tend to think of the highest number as the positive point in a rating scale, we can easily overlook a scale that has been labeled the opposite way.

The UserTesting Research Team has repeatedly observed contributors providing a positive verbal response (i.e. “It was great!”), but clicking the “5 = Low Quality” radio button, despite the negative endpoint label.

Pro tip: If you set up a rating scale question and realize that the two endpoints are neutral and/or the positive point is right in the middle, consider turning it into a multiple-choice question instead.

4. Measure one element or action at a time

Write individual rating scale questions for each task that your contributors perform. If you’re worried about contributors mistaking the second question as a duplicate, you can always capitalize the task you are asking contributors to rate, for additional emphasis.

Want to try out Rating Scale questions for yourself?

Now you’re all set to start writing some stellar Rating Scale questions and getting some solid quantitative data with your next study!

For more information, visit the original blog post Using Rating Scale Questions in Your UX Research.