Card Sorting with OptimalSort and UserTesting

This article explains how to use Optimal Workshop’s card sorting tool, OptimalSort, in conjunction with the UserTesting platform. To learn about card sorting, including why it’s useful and when you should do it, read this article here. 

 

Why should you use OptimalSort and UserTesting together?

OptimalSort is a powerful unmoderated research tool. UserTesting offers the ability to see, hear and talk to users as they engage with products in real-time. Used together, OptimalSort and UserTesting can deliver both qualitative and quantitative insights at an affordable price.

  • OptimalSort will help you understand how people think your content should be grouped. You can use the results of your OptimalSort studies to quickly spot popular groupings and categories, and use this information to build a better website.

  • UserTesting will give you behavioral insights from watching and listening to participants complete your card sort, and answers to questions such as “Did people find any task particularly confusing?” and “Why did people group the cards one way and not another?”

How to set up an OptimalSort study in the UserTesting platform

Part 1 - Set up your card sort in Optimal Sort

1. Create your cards

Create cards that represent the information you want feedback on, such as content topics, page names, or category names on your website. You can choose to add descriptions and images, or even just images on their own (for an eCommerce website, for example).

You will also select either an open, closed, or hybrid option, depending on what you want to find out:

  • Open card sort: participants sort items into categories that make sense to them, and then label the categories
  • Closed card sort: participants sort items into categories you give them
  • Hybrid card sort: participants sort items into categories you give them and can also create their own

OS1.png

2. Adjust Participant Identifier on the Questionnaire tab

Instead of asking participants for an email address, you should ask users to type in their UserTesting username. This will help you easily match their OptimalSort results with their UserTesting recording when you analyze your data.

You might want to invite participants outside of the UserTesting platform to gather more quantitative data. These participants won’t have a username. They’ll still need to enter something into the identifier box, so you should tell these participants to enter something simple, like 123.

OS2.png

3. Edit the welcome and thank you messages in the Messages tab.

Participants see the ‘Welcome’ message when they click on the survey link you give them. The default message welcomes participants to the card sort, gives an estimated time frame and a short explanation of what the card sort is for.

Here’s an example message that you can tailor based on the particulars of your card sort:

“Welcome to this OptimalSort study, and thank you for agreeing to participate. This activity shouldn’t take longer than 10 to 15 minutes to complete. Your response will help us to organize the content on our website.”

 

OS3.png

Participants see the ‘Thank You’ message when they click ‘Finish’. The default message ends with the sentence, ‘You may now close this window or navigate to another web page.’ If you are recruiting only UserTesting participants, use the following message to direct participants back to their task box:

‘Thanks for completing our study. To continue, please return to the UserTesting task box in the top right corner of the screen.’

OS4.png

If you plan to recruit other participants (not via UserTesting) to get more data, add a note for those participants as well, such as “If you are not a UserTesting participant, you can now close this window or navigate to another webpage.”

4. Write pre or post card sort questions

You can also ask your participants to answer pre- and post-card sort questions to give you more qualitative insights. This will enable you to segment your results based on their answers.

If you plan to recruit UserTesting participants only, you could choose to only ask these questions in the UserTesting platform. If you plan to recruit participants outside the UserTesting panel as well, then asking the questions on both platforms will mean all your participants get a chance to answer them.

You might want to ask questions like:

  • How easy or difficult did you find this card sort?
  • Which cards, if any, were most difficult to sort? Why do you think that is?
  • Which cards, if any, were difficult to understand?
  • [for open sorts only] What groups, if any, were difficult to name? What groups, if any, were easy to name?
  • In general, what are your thoughts on the groupings (and labels, if applicable), that you created?

OS5.png

5. Preview your survey before launch to ensure accuracy

The Preview button is your best friend when it comes to getting your card sort right. Preview enables you to see your card sort from your participants’ point of view, and it makes spotting errors or inappropriate tasks much easier.

When you preview your survey, check that: 

  • Participants are required to give their UserTesting username (or another participant identifier if they aren’t UserTesting participants)
  • The 'Welcome' and 'Thank you' messages are suitable and error-free
  • Your cards contain the right content and are error-free
  • Your pre or post-survey questions are clear.

OS6.png 

6. Launch your survey and copy the link

Before you can finalize your UserTesting survey, you’ll need to launch your card sort and copy the link into your UserTesting invitation. You’ll get a link to your card sort when you launch it, and you can also find it on the ‘Setup’ tab.

 

OS7.png

Part 2 - Set up your UserTesting card sort with your survey link

1. Create a new unmoderated test in UserTesting

NewUnmod.jpg

 

 Next, click 'Test a website'. 

TestAWebsite.png

Then, you'll have the option of testing with the UserTesting Panel or your own customers. In this example, we'll be using the UserTesting Panel.

PanelMyRecruit.png

2. Select your sample size and target audience in UserTesting

Usually, when running a card sort, it’s best to opt for a larger sample size so you can get solid quantitative data. However, we recommend only asking a portion of your participants to record themselves via UserTesting as they complete the activity in OptimalSort. 

Typically, we find that 8-10 users will give you the qualitative insight you need.

TargetAudience.png

 

3. Choose a starting URL (not the OptimalSort link) and provide some context

In the ‘Starting Instructions’ portion of UserTesting, choose a URL to send users to, such as www.google.com. Don’t use the OptimalSort link just yet. You want to provide more context and instruction before users get to the card sorting activity.

You’ll also want to provide some context in the ‘Introduction’ field. We’ve had success with the following:

‘Please note: In this test, you’ll be asked to use OptimalSort. OptimalSort is an online tool that helps companies determine how they should structure and organize items’.

Card_Sorting1.png

4. Add the OptimalSort link and descriptive instructions to the first task

The descriptive instructions provide details about the card sort activity and how the participant should ‘end’ the test in both UserTesting and OptimalSort. The UserTesting Research Team recommends the following text for the first task for additional clarity before sending participants to the OptimalSort activity.

'Please visit the following link to access OptimalSort: [URL]

Follow the OptimalSort instructions, speaking your thoughts aloud as much as possible. When you finish sorting the cards, **please click “FINISH” in the upper right corner of the page to save your selections, answer any follow-up questions in OptimalSort, and click “Continue".**'

Then click NEXT in the upper right of this UserTesting task box.

Card_Sorting2.png

5. Include follow-up questions to get additional insight from users

Finally, the UserTesting platform allows you to ask follow-up questions to evaluate the participant’s experience as they completed the card sort activity. If you’re considering inviting participants outside of UserTesting as well, then asking these questions in OptimalSort will ensure you get answers from those participants as well.

You might ask people:

  • How easy or difficult did you find this card sort?
  • Which cards, if any, were most difficult to sort? Why do you think that is?
  • Which cards, if any, were difficult to understand?
  • [For open card sorts only] What groups, if any, were difficult to name? What groups, if any, were easy to name?
  • In general, what are your thoughts on the groupings (and labels, if applicable), that you created?

Related Resources

 

Was this article helpful?
11 out of 15 found this helpful