Card Sorting with OptimalSort and UserTesting

Last Updated: Nov 30, 2016 11:37AM PST

This article will provide you with practical tips on how to combine OptimalSort and UserTesting, to get both quantitative and qualitative data for card sorts. Optimal Sort and Tree Jack are part of Optimal Workshop's product suite.

Why use both OptimalSort and UserTesting

When you invite participants to complete your OptimalSort card sort via the UserTesting platform, you’ll get two valuable types of data for making informed design decisions:

1. With OptimalSort , you’ll get in-depth data visualizations (such as dendrograms and similarity matrices) for quickly spotting the most popular groupings and category labels, and for sharing your results with stakeholders.

2. With UserTesting, you’ll get behavioural insights from watching and listening to participants complete the card sort, and answers to questions such as “Why did people group items together?” and “What items were difficult (or easy) to group or name?”

Here’s a simple process to follow for setting up a card sort using OptimalSort and UserTesting.

Part 1. Set up your card sort in OptimalSort

1. Create your cards

Create cards that represent the different information, such as content topics, page names, and category names, on your website. You can choose to add descriptions and images, or even images on their own (for an ecommerce website, for example).

You will also select either an open, closed, or hybrid option, depending on what you want to find out:

  • An open card sort — participants sort items into categories that make sense to them, and then label the categories
  • A closed card sort — participants sort items into categories you give them
  • A hybrid card sort — participants sort items into categories you give them and can also create their own

 

2. Adjust Participant Identifier in the Questionnaire tab

Instead of asking participants for an email address, you should have users type in their UserTesting user name. This will help you easily match their OptimalSort results with their UserTesting recording when you analyze your data.

You might want to invite remote participants to supplement the quantitative data, and these participants won’t have a username. Participants must enter an identifier if you request it, so you could tell these participants to enter something simple (like 123).

3. Edit the welcome and thank you messages in the Messages tab

Participants see the “Welcome” message when they click on the survey link you gave them. The default message welcomes participants to the card sort, gives an estimated time frame, and a short explanation of what the card sort is for. You can tailor this message based on the particulars of your card sort:

“Welcome to this OptimalSort study, and thank you for agreeing to participate. This activity shouldn’t take longer than 10 to 15 minutes to complete. Your response will help us to organize the content on our website.”

Participants see the “Thank You” message when they click “Finish”. The default message ends with the sentence, “You may now close this window or navigate to another web page.” If you are recruiting only UserTesting participants, use this message to direct participants back to their task box:

“Thanks for completing our card sort. To continue, please return to the UserTesting task box in the top right corner of the screen.”

If you plan to recruit other participants (not via UserTesting) to get more data, add a note for those participants as well, such as “If you are not a UserTesting participant, you can now close this window or navigate to another webpage.”

4. Write pre or post card sort questions

You can ask participants to answer pre and post survey questions to give you more qualitative insights, and to enable you to segment your results based on their answers in your analysis.

If you plan to recruit UserTesting participants only, you could choose to only ask these questions in the UserTesting platform. If you plan to recruit participants outside the UserTesting panel as well, then asking the questions in both platforms will mean all your participants get a chance to answer them. You might want to ask questions like:

  • How easy or difficult did you find this card sort?
  • Which cards, if any, were most difficult to sort? Why do you think that is?
  • Which cards, if any, were difficult to understand?
  • [for open sorts only] What groups, if any, were difficult to name? What groups, if any, were easy to name?
  • In general, what are your thoughts on the groupings (and labels, if applicable), that you created?

 

5. Preview your survey before launch to ensure accuracy

The Preview button is your best friend when it comes to getting your card sort right. Preview enables you to see your card sort exactly as your participants will, and makes spotting spelling or content errors much easier. When you preview your survey, check that:

  • Participants are required to give their UserTesting username (or another participant identifier if they aren’t UserTesting participants)
  • The “Welcome and “Thank you” messages are suitable and error-free
  • Your cards contain the right content and are error-free
  • Your pre or post survey questions are clear.

 

6. Launch your survey and copy the link

Before you can finalize your UserTesting survey, you’ll need to launch your card sort and copy the link into your UserTesting invitation.

 

Part 2. Set up your UserTesting card sort with your survey link

1. Select your sample size in UserTesting

While you typically use larger sample sizes for card sorting to get solid quantitative data and reduce the margin of error, consider asking a only portion of your larger sample to record themselves via UserTesting as they complete the activity using OptimalSort. Typically, 8-10 users will give you the qualitative insight you need.

2. Choose a starting URL (not the Optimal Sort link -- yet!) and provide some context.

In the “Starting Instructions” portion of UserTesting, choose a URL to send users to, such as www.google.com. Don’t use the Optimal Sort link just yet. You want to provide more context and instruction before users get to the card sort activity.

You’ll also want to provide some context in the Introduction field. We’ve had success with the following:

Please note: In this test, you’ll be asked to use OptimalSort. OptimalSort is an online program that helps companies determine how they should structure and organize items.

3. Add the OptimalSort link and descriptive instructions to the first task

The descriptive instructions provide details about the card sort activity and how the tester should ‘end’ the test in both UserTesting and OptimalSort. The UserTesting Research Team recommends the following text for the first task for additional clarity before sending users to the OptimalSort activity.

Please visit the following link to access OptimalSort: [URL]

Follow the OptimalSort instructions, speaking your thoughts aloud as much as possible. When you finish sorting the cards, **please click “FINISH” in the upper right corner of the page to save your selections, answer any follow-up questions in OptimalSort, and click “Continue”.**

Then click NEXT in the upper right of this UserTesting task box.

4. Include follow-up questions to get additional insight from users.

Finally, the UserTesting platform allows you to ask follow-up questions to evaluate the user’s experience as they completed the card sort activity. If you’re considering inviting participants outside of UserTesting as well, then asking these questions in OptimalSort will ensure you get answers from those participants as well. You might ask people:

  • How easy or difficult did you find this card sort?
  • Which cards, if any, were most difficult to sort? Why do you think that is?
  • Which cards, if any, were difficult to understand?
  • [for open sorts only] What groups, if any, were difficult to name? What groups, if any, were easy to name?
  • In general, what are your thoughts on the groupings (and labels, if applicable), that you created?

Other Resources

entsupport@usertesting.com
http://assets1.desk.com/
false
desk
Loading
seconds ago
a minute ago
minutes ago
an hour ago
hours ago
a day ago
days ago
about
false
Invalid characters found
/customer/en/portal/articles/autocomplete