Card sorting with UserTesting

Overview

This article explains how to use UserTesting’s card sorting tool. We recommend most users start with the card sorting template. To learn about card sorting, including why it’s useful and when you should do it, read the article What is card sorting?

Why should you run your card sort via UserTesting?

Running your card sort via UserTesting can deliver both qualitative and quantitative insights.

  • The card sort results will help you understand how people think your content should be grouped. You can use the results of the card sort to quickly spot popular groupings and categories, and use this information to build a better website.

  • The recordings of the participant sessions will give you behavioral insights from watching and listening to participants complete the card sort, and answers to questions such as “Did people find any task particularly confusing?” and “Why did people group the cards one way and not another?”

  • Card sorting is included with our Ultimate Edition subscriptions. For customers on our Advanced, Pro, Premium Editions, card sorting can be added for an additional cost. Please contact your account team for more information.

Note: Card sorting is not optimized for mobile.

How to set up a card sort in the UserTesting platform.

1. Create a new unmoderated test in UserTesting.

One of the tasks you will be able to add to your test is a card sorting task. To start your test, select New > Create an unmoderated test:

Dropdown_Menu_New.png

Next, click 'Test a website'.

Test_website_cardsorting.jpg

You'll have three options for where your participants will come from; our participants (the UserTesting Panel), or your own participants (My Panel or My Recruit). Panel_Options.jpg

2. Select your sample size and target audience in UserTesting.

When running a card sort, it’s best to opt for a larger sample size so you can get solid data and see content groups. Send your card sort test to 30 participants. If your account limits you to 15 participants per test, launch one test with 15 participants, then create a copy of that test (Create similar test) and launch to 15 more participants. Alternately, you can keep all the results in one test by inviting 30 of your own My Recruit participants to take the test.

For more information about finding your ideal participants, see our other Knowledgebase articles about targeting your audience.

 

cardsort_audience_cardsorting.jpg

3. Choose a starting URL and provide some context.

In the Starting Instructions section of UserTesting, choose a URL to send participants to. If you don’t have any specific site you want your participants to start on, use the A blank page option. Note that this Starting URL will not be the link to your card sort—that will be added in the Test Plan section of your test because you want to provide more context and instruction before users get to the card sorting activity.

Also, provide some context in the Scenario field, such as:

“Please note: One of the activities in this test will be conducting a card sort. This activity helps companies determine how they should structure and organize items.”

Starting_instruction_cardsorting.jpg

4. Build out your test plan, adding a Card Sort at the appropriate point in the flow of the test.

You may want to ask some background questions before the card sort. For example, you may want participants to rate their familiarity with the topic that the card sort is focused on. Do not include too many tasks before the card sort because (1) you don’t want to inadvertently bias the participant’s thinking before they take the card sort and (2) you want to give the participant ample time to complete the card sorting task.

Card_sort_task_cardsorting.jpg

5. Set up the card sort activity.

In the card sort task, follow the link to the card sorting app: https://ia.usertesting.com/. This will open in a new browser tab.

Create a new card sort. You will also select either an open, closed, or hybrid option, depending on what you want to find out:

New_card_sort_options_cardsorting.jpg

  • Closed card sort: participants sort items into categories you give them
  • Open card sort: participants sort items into categories that make sense to them, and then label the categories
  • Hybrid card sort: participants sort items into categories you give them and can also create their own

Add_cards_cardsorting.jpgFill out the cards with the content you want participants to sort. Create cards that represent the information you want feedback on, such as content topics, page names, or category names on your website. 

Example_card_sort_cardsorting.jpg

If you are running a closed card sort or hybrid card sort, you will also add the category labels.

Categories_cardsorting.jpg

Click Publish to publish your card sort activity. ia.usertesting_05.png

 

ia.usertesting_06.png

Copy the link to your card sort activity. Card_sort_activity.jpg

Paste the link into the Card Sort task.

Card_sort_URL_Cardsorting.jpg

6. Include follow-up questions to get additional insight from users.

Finally, add tasks for any follow-up questions you want to ask to evaluate the participant’s experience as they completed the card sort activity..

You might ask people:

  • How easy or difficult did you find this card sort?
  • Which cards, if any, were most difficult to sort? Why do you think that is?
  • Which cards, if any, were difficult to understand?
  • [For open card sorts only] What groups, if any, were difficult to name? What groups, if any, were easy to name?
  • In general, what are your thoughts on the groupings (and labels, if applicable), that you created?

Results of your card sort

Once the participants complete your test, you can see the results in your Dashboard, just as any UserTesting test, and watch videos of participants completing the card sort.

For any test that includes a card sort task, at the top of the Sessions tab, you will also see a button to view the results of the card sort activity.

Results_Cardsorting.jpg

Clicking the button will open a new tab. You will be taken to the dashboard on the card sort app, from where you’ll be able to select the test to review.

www.usabilitest_3.png

  • On this page, you can view the participants. If you have participants who did not complete much of the card sort activity, you can exclude them here.
    Note that you may see more participants in the card sort than in your UserTesting test. This is because the UserTesting system will automatically replace any participants who abandon their test. So, if any participants start the card sort activity, but do not complete their UserTesting test, their data will be included in the card sort activity, but they will not be reflected in the list of completed sessions.

  • The results of the card sort include different views where you can see how participants grouped content and how often different cards were grouped together.
    • Raw numbers: This table represents how many cards were dropped into each category by all participants combined.CS_Raw_Numbers.png
    • Percentages: This table represents how many cards were dropped into each category above a threshold you can adjust. The default is a 5% threshold, which means that the results below that threshold have been omitted.CS_Percentages.png
    • Grouped percentages: This table represents how many cards were dropped into each category above a threshold and grouped based on the most cards associated with these categories. The default is a 5% threshold, which means that the results below that threshold have been omitted. You can adjust the threshold level.CS_Grouped_Percentages.png
    • Distance matrix: This table represents dissimilarities between cards in each pair. The lower the number, the more similar the two cards are, and the higher the number, the more dissimilar they are.CS_Distance_Matrix.png
    • MDS (Multidimensional Scaling): This diagram is an analysis and visual representation of the distance matrix. You can analyze any kind of similarity or dissimilarity matrix using multi-dimensional scaling. Plotting these data sets on a multi-dimensional scale allows for easier interpretation and comparison than a linear dataset permits.CS_MDS.png
    • Dendrogram: This diagram represents a Hierarchical Cluster Analysis, which groups the cards that were considered most similar by the participants and places them on branches that are close together. This is sometimes called a tree diagram. The longer the line (or branch) that connects to cards, the more dissimilar they are. Shorter lines indicate cards that are more similar to each other.CS_Dendrogram.png

Related Resources

Was this article helpful?
0 out of 4 found this helpful