Quick Answer: Compare two websites

What are Quick Answers?

Quick Answers are insights into a problem you encounter or question you have during the development process. You can read more about Quick Answers in the Product Insight: Getting Started.

“Compare two websites" is one of six types of Quick Answers. Each Quick Answer provides a unique insight or answers a question you have.

For “Compare two websites" we want to see which of two websites participants prefer using to complete specific task.

When to use “Compare two websites"

Use “Compare two websites" in the Discover and Iterate phases of the development process to:

  • Compare websites’ strengths, weaknesses, and ease of use.
  • Test your website against a competitor’s.

Set up “Compare two websites"

1. From the dashboard click Get Quick Answers, then click “Compare two websites."

Websites2.png

2. Describe the task you want participants to do on both sites. Start your description with a verb, but don’t include exact steps, such as “Click on this button. Then this link. Then go….”

The description is added to a pre-written sentence, so it’s a good practice to use a lowercase letter for the first word and not end with a period or other punctuation. Quick Answers converts your description into the test questions, so the wording you enter is what people see.

Some examples include:

  • sign up for the newsletter becomes Take as much time as you need to sign up for the newsletter.
  • find out when to report for jury duty becomes Take as much time as you need to find out when to report for jury duty.
  • change your profile picture becomes Take as much time as you need to change your profile picture.

Enter the links and generic names for both websites.

TwoWebsites.png

Then name your test so it’s easy to recognize from other tests on the dashboard. 

 At this point, you'll need to determine whether you'd like to test with participants from the UserTesting Panel, or your own customers, via My Recruit. 

For this example, we'll select From the UserTesting panel.

 

3. Click Next and do any of the following:

Websites4.png

  • Accept defaults: The default is to use any member of the UserTesting Panel. For many tests, you may find that works well enough. If that’s the case, click Done at the bottom of the page.
  • Number of participants: This is always set to five.
  • Device: Select the type of device you want your participants to use for the test. If you need to compare websites as they appear on a smartphone, select smartphone. If you want to compare websites on a tablet, select tablet.

    For more information on selecting a device and best practices, see:

    Mobile Testing Capabilities
    Mobile Recorder Best Practices

  • Demographic Filters: To refine participant selection with demographic filters, select and configure any filters you want to use from the options on the right. If you decide not to use a filter after selecting it, click the Remove link in the upper-right corner or deselect it from the list on the right.

    For more help with demographic filters, see:

    Choose your target audience from the UserTesting panel.
  • Screener Questions: Screener questions are multiple choice questions participants must answer correctly before they're able to accept a test. This is an optional feature you can select if the demographic filters above do not cover your requirements. In most cases, you can leave Screener Questions blank.

    To refine participant selection with screener questions, do any of the following:

    • To add a screener question you saved from a previous test, click Saved Screener Questions on the right. If you haven’t saved any screener questions yet, this is empty.
    • To create a screener question, click the Screener Question button on the right.
    • To add more screener questions, click the Screener Question button or the “Add another screener question” link below the last screener question.

      Note: After you create a screener question, you can click the “Add to saved screener,” “Duplicate,” or “Delete” links. To reorder screener questions, drag the question to where you want it to be using the Move icon.
    • To preview any screener questions, click Preview Screener.


You can remove any screener questions you add by clicking Delete at the top right of each question’s dialog.

For more help creating screener questions, see:

Using Screener Questions to find your ideal participants
Managing screeners
Screener question best practices

4. Click Done. 

Websites5.png

  • To see a list of all the questions the test asks participants, click the link under Tasks and Questions. To see what your test looks like to participants, click Preview Test to the right of the Test Plan Summary, then click Preview Test. When you finish the preview, click the Close button.

Note If you realize you need to add or remove a participant, or you find something you need to change in your test plan—for example, you used the wrong URL or you find a typo, do either of the following:

  • Click Participants to add or remove test participants.
  • Click Edit Details in the upper right. When you are done editing the test, click "Save and Go to Order Page."

5. Click Launch Test to send the test to the UserTesting Panel.

 

Understand  “Compare two websites" results

As participants finish, the software uploads the recorded video to our system and notifies you by email. To configure notifications, see Email notification preferences.

Open the email and click the “account dashboard” link in the order summary to go to the dashboard.


For each test, the dashboard displays the type of test—using the same icon from each Quick Answer button—the test name, the day you created it, and how many participants have and have not taken the test.

Websites6.png

After all participants finish, a summary appears in the Result column.

Valuable.png

 Click a test title on the dashboard to go to the Results tab on the Results page.

Results Tab

The Results tab is the sweet spot for Quick Answers. It summarizes the answer to your question: Which website do people prefer?

 ResultsTab.png

We get this information from Question 7 below. We ask participants, “Which website do you prefer? Explain your answer.” [Name for Website A] or [Name for Website B]. 

Then we count how many of the five participants preferred Website A and how many preferred Website B.

 

Watch key moments 

The summary video shows all five participants describing which website they prefer.

Websites8.png

We get this information from Question 5 below. We tell participants: “Take as much time as you need to [activity description] and we record their actions and comments as they do the task.

 

Note: Until all participants finish, the graph shows a running total as participants complete the test and displays the number of participants yet to finish, but there is no key moments video until all participants finish.

 

 

Sessions

Click the Sessions tab to watch each participant’s complete video as they answer all the questions in order. Sessions are available as each participant finishes. There is a text field on the right for you to add a summary of the video. The Summary field is also useful for adding timecodes for moments of interest, such as a participant expressing delight with a task, or complaining about a difficult interface.

PISessionsTab.png

Metrics

Click the Metrics tab to watch participants’ videos broken down by question so you can easily skim the results. Questions have either written responses or video response.

Metrics is a powerful way to understand the details of your participants’ experiences. After the Results tab, you are likely to find this tab contains the most useful information because it lets you focus on each test question and each participant’s answer.

Websites12.png

Details

Click the Details tab to see the demographics breakdown and an overview of the test plan tasks and questions.

DetailsTab.png

 

How we get Quick Answers

After you click Launch Test, we send the test to people in our UserTesting Panel who match the demographics you chose. These people then answer any screener questions you added in order to qualify to take the test.

If you’d like to see what answering screener questions looks like from the participants’ view, watch this video.

Here’s what we ask participants based on the questions you answered when you created the test:

Introduction

You’ll be performing the same activity on 2 different websites and then asked to rate your experience. Please remember to clearly explain your thoughts and actions out loud. Don’t pretend to be someone else; we care about your needs and thoughts.

  1. [Launch URL][Label A]: [URL Website A provided]

  2. [Verbal] Take as much time as you need to [activity description].

  3. [Rating] Overall, this activity was …? Explain your answer. 1=Very difficult…5=Very easy

  4. [Launch URL][Label B]: [URL Website B provided]

  5. [Verbal] Take as much time as you need to [activity description].

    Note: The answer to this question is where we get the highlights for the video on the Results tab.

  6. [Rating] Overall, this activity was …? Explain your answer. 1=Very difficult…5=Very easy

  7. [Multiple choice] Which website do you prefer? Explain your answer. {Results Graph]
     
    [Label Website A]
    [Label Website B]

    Note: The answer to this question is where we get the data for the graph on the Results tab.

  8. [Verbal] What, if anything, do you think [Label Website A] does better than [Label Website B] to help you [activity description]?

  9. [Written] Summarize your previous answer in a few words. What do you think [Label Website A] does better to help you [activity description]?

  10. [Verbal] What, if anything, do you think [Label Website B] does better than [Label Website A] to help you [activity description]?

  11. [Written] Summarize your previous answer in a few words. What do you think [Label Website B] does better to help you [activity description]?

 

As participants finish, the software uploads the recorded video to our system and notifies you by email that the videos are ready to watch. Watch each participant’s entire video on the Session tab or wait for the fifth video session to be uploaded, at which point the highlight reel is available.


See also

New to Product Insight?
Product Insight overview
Quick Answer: Test website usability
Quick Answer: Choose between two options
Quick Answer: Validate a prototype
Quick Answer: Validate an Idea
Quick Answer: Discover needs and frustrations
Product Insight: Live Conversation

 

 

 

 

Was this article helpful?
1 out of 2 found this helpful