Card Sorting with UserTesting

Flex_Add-on_Seat_Add-on.png

At a Glance

This article explains how to use UserTesting’s card sorting tool. We recommend most users start with the IA - Card Sorting template available in the Template gallery (under the Information architecture category). To learn about card sorting, including why it’s useful and when you should do it, read the article What is card sorting?

 

Why should you run your card sort via UserTesting?

How to set up a card sort in the UserTesting platform

Viewing card sort results on the Metrics tab

Sharing card sort results with team members/colleagues

 

Why should you run your card sort via UserTesting?

Running your card sort via UserTesting can deliver both qualitative and quantitative insights.

  • The card sort results will help you understand how people think your content should be grouped. You can use the card sort results to quickly spot popular groupings and categories and use this information to build a better website.
  • The recordings of the participant sessions will give you behavioral insights from watching and listening to participants complete the card sort and answers to questions such as “Did people find any task particularly confusing?” and “Why did people group the cards one way and not another?”
  • You can use card sorting in both unmoderated and moderated tests. Add a card sort as you build your unmoderated test plan, or share it with a participant during a moderated Live Conversation.
  • Card sorting is included with our Ultimate Edition subscriptions. For customers on our Advanced, Pro, Premium Editions, card sorting can be added for an additional cost. Please get in touch with your account team for more information.

Note: Card sorting is not optimized for mobile.

How to set up a card sort in the UserTesting platform

1. Create a new unmoderated test in UserTesting.

One of the tasks you will add to your test is a card sorting task. To start your test, select New > Create an unmoderated test:

create_an_unmoderated_test_copy.png

Next, click Website.

Test_website_cardsorting.jpg

You'll have three options for where your participants will come from our participants (the UserTesting Panel) or your participants (My Panel or My Recruit). Panel_Options.jpg

2. Select your sample size and target audience in UserTesting.

When running a card sort, it’s best to opt for a larger sample size so you can get reliable data and see content groups. Send your card sort test to 30 participants. If your account limits you to 15 participants per test, launch one test with 15 participants, create a copy of that test (Create a similar test), and launch to 15 more participants. Alternately, you can keep all the results in one test by inviting 30 of your own My Recruit participants to take the test.

For more information about finding your ideal participants, see our other Knowledgebase articles about targeting your audience.

cardsort_audience_cardsorting.jpg

3. Choose a starting URL and provide some context.

In the Starting Instructions section of UserTesting, choose a URL to send participants to. Use the A blank page option if you don’t have any specific site you want your participants to start on. Note that this Starting URL will not be the link to your card sort—that will be added in the Test Plan section of your test because you want to provide more context and instruction before users get to the card sorting activity.

Also, provide some context in the Scenario field, such as:

“Please note: One of the activities in this test will be conducting a card sort. This activity helps companies determine how they should structure and organize items.”

Starting_instruction_cardsorting.jpg

4. Build out your test plan, adding a card sort at the appropriate point in the test flow.

You may want to ask some background questions before the card sort. For example, you may want participants to rate their familiarity with the topic that the card sort is focused on. Do not include too many tasks before the card sort because (1) you don’t want to inadvertently bias the participant’s thinking before they take the card sort, and (2) you want to give the participant ample time to complete the card sorting task.

Card_sort_task_cardsorting.jpg

5. Set up the card sort activity.

In the card sort task, follow the link to the card sorting app: https://ia.usertesting.com/. This will open in a new browser tab.

Create a new card sort. You will also select either an open, closed, or hybrid option, depending on what you want to find out:

Card_sorting_2.png

  • Closed card sort: participants sort items into categories you give them
  • Open card sort: participants sort items into categories that make sense to them and then label the categories
  • Hybrid card sort: participants sort items into categories you give them and can also create their own

Create cards that represent the information you want feedback on, such as content topics, page names, or category names on your website.

Note: Creators can access, edit, and view results for their tests. They cannot view or access tests launched by others on the account. 

Label each card you want participants to sort. You will need at least two cards added to enable Publishing, and you can have as many cards as you need. To add more cards, click Add Card.

Card_sorting_1.pngFill

Note: For Open and Hybrid tests only, there exists in Advance Options a setting that allows participants to name groups after sorting their cards. Participants will see at the end of an open or hybrid test a message ("Please give a name to each of the groups you have created to complete the task.") reminding them not to leave the test before adding names to the groups.

You also have the option to add images to your cards. To add images, click Show/Hide Images input. The maximum number of images you can upload at one time is 20 images, 8MB total. If your image file size exceeds these limits, upload images in several smaller groups. Images must also be <2MB in size and be .jpg, .gif, or .png files. For image cards, you must label each image. Click Save Cards to upload.

Note: Your card sort activity can be a mix of images and no images.

Once you have uploaded your card sort images, you can preview the image by hovering over the image thumbnail on the card.

card_sorting_4.png

You will also add the category labels if you are running a closed card sort or hybrid card sort.

Categories_cardsorting.jpg

Click Publish to publish your card sort activity. 

Note: Admins can publish and unpublish tests created by others, just as the creator can. However, admins cannot edit tests created by others. Once a test is deployed, admins can also view and share results as necessary. ia.usertesting_05.png

 

ia.usertesting_06.png

Copy the link to your card sort activity.  Card_sort_activity.jpg

Paste the link into the Card Sort Task. The link can also be shared with a participant during a moderated Live Conversation.
Card_sort_URL_Cardsorting.jpg

Note: All participants upon completing the card sort test—whether that be a closed, open, or hybrid card sort—will see a "Move on to the next step" message, which is intended to make it super clear to the participants that just the IA test is complete and not the UT test.

6. Include follow-up questions to get additional insight from users.

Finally, add tasks for any follow-up questions you want to ask to evaluate the participant’s experience as they completed the card sort activity.

You might ask people:

  • How easy or difficult did you find this card sort?
  • Which cards, if any, were the most difficult to sort? Why do you think that is?
  • Which cards, if any, were difficult to understand?
  • [For open card sorts only] What groups, if any, were difficult to name? What groups, if any, were easy to name?
  • In general, what are your thoughts on the groupings (and labels, if applicable) that you created?

Viewing card sort results on the Metrics tab

While only the test creator and account admins can access the results of a card test at the ia.usertesting subdomain, all users on the account can see general results on the Metrics tab in the UserTesting platform.

Below is the Results Overview section of the Metrics tab, showing the card sort task results. Two features within the results table give users the options to...

  1. Sort the cards by Most Popular Category or by Order of Entry (the order in which the cards were entered).
  2. Switch between two views: Percentage (percentage of participants) and Count view (number of participants/”raw numbers”)

Metrics_tab_Card_sort.png

The results table with color-coded cells

The results table shows the cards sorted by most-popular category. The background colors of the cells differentiate higher and lower percentages of users who sorted the cards into each category.

Listed beneath the results table are...

  1. Videos of each contributor executing individual tasks within the card sort (see the Sessions tab for the full videos of individual contributors completing all the tasks.)
  2. A View Full Results link that takes admins and test creators to the ia.usertesting.com subdomain, which uses grouped percentages, multidimensional scaling, and other measurements to break down the results. It is also at this subdomain where admins and creators can share the complete results of the sort. (See the "Sharing card sort results with team members/colleagues" section of this article below.)

Videos_and_link.png

 

Clicking the View full results button will take you to the dashboard on the card sort app. From here, you’ll be able to select the test to review.

www.usabilitest_3.png

On this page, you can view the participants. If you have participants who did not complete much of the card sort activity, you can exclude them here.

Note that you may see more participants in the card sort than in your UserTesting test. The UserTesting system will automatically replace any participants who abandon their test. So, if any participants start the card sort activity but do not complete their UserTesting test, their data will be included in the card sort activity. Still, they will not be reflected in the list of completed sessions.

The card sort results include different views where you can see how participants grouped content and how often other cards were grouped.

    • Raw numbers: This table represents how many cards were dropped into each category by all participants combined. This view of the data is useful to export and manipulate externally - this is especially helpful in comparing results from multiple similar studies (i.e. the same card sort distributed to different audiences).CS_Raw_Numbers.png

Images are also visible when you review the results of your card sort. View images when you hover over the thumbnails on the raw data, percentage table, grouped percentages, and distance matrix tables. Images are not viewable on the MDS or dendrogram tabs.

card_sort_5.png

    • Percentages: This table represents how many cards were dropped into each category above a threshold you can adjust. The default is a 5% threshold, which means that the results below that threshold have been omitted. A higher percentage indicates the card was placed in that category by more participants. A lower percentage split across multiple categories indicates some ambiguity in terms of how to classify a particular card.CS_Percentages.png
    • Grouped percentages: This table represents how many cards were dropped into each category above a threshold and grouped based on the most cards associated with these categories. The default is a 5% threshold, which means that the results below that threshold have been omitted. You can adjust the threshold level. This visual grouping of cards makes it easier to understand which cards are related.
      • If no cards are highlighted for a particular category, that category may not be relevant or may need to be merged with another category.
      • If evenly split between multiple categories (i.e. same percentage) a card will be grouped with the first category on the list.CS_Grouped_Percentages.png
    • Distance matrix: This table captures the relationship between individual cards (without consideration for the specific categories chosen). The number for each cell is based on the number of participants that did not group the two cards in the same category. The lower the number, the more similar the two cards are, and the higher the number, the more dissimilar they are.CS_Distance_Matrix.png
    • MDS (Multidimensional Scaling): This diagram is an analysis and visual representation of the distance matrix. You can analyze any kind of similarity or dissimilarity matrix using multi-dimensional scaling. Plotting these data sets on a multi-dimensional scale allows for easier interpretation and comparison than a linear dataset permits. When reading an MDS map it represents the relative distances between cards - the scale of the axes has no impact on the relative distance between cards. Based on the data, similar cards will be clustered together. The proximity of the clusters to one another also indicate how similar (close) or dissimilar (far apart) they are from one anotherCS_MDS.png
    • Dendrogram: This diagram represents a Hierarchical Cluster Analysis, which groups the cards that were considered most similar by the participants and places them on branches that are close together. This is sometimes called a tree diagram. The longer the line (or branch) that connects to cards, the more dissimilar they are. Shorter lines indicate cards that are more similar to each other. The key to interpreting a dendrogram is to focus on where any two cards are joined together 
      • If the cards are joined together on the left of the dendrogram then those cards are more different - i.e. the relative distance between them on the MDS is large.
      • If the cards are joined together on the right of the dendrogram then the cards are more similar - i.e. the relative distance between them on the MDS is small.CS_Dendrogram.png

Sharing card sort results with team members/colleagues

To share the results of a card sort, open the Advanced Options drop-down menu under the Test Details section...

Advanced_Options.png

...and select the Shared button found at the bottom of the drop-down.

 

Shared.png

 

This will generate a SHARED link that appears at the bottom of Test Details...

Screen_Shot_2021-06-07_at_4.54.58_PM.png

A couple of things to note about this SHARED link:

  • It will be generated only when the Shared button has been activated.

  • It will return data only when participants have taken the test.
  • It will only show team members the Raw Numbers tab in the ia.usertesting subdomain. It will NOT show percentages, grouped percentages, distance matrix, MDS, and dendrogram results.\

The XLS or CSV file for each individual result can be downloaded from the card sort Results tab and shared with team members.

 

Learn More

Need more information? Read these related articles.

Want to learn more about this topic? Check out our University course.

Was this article helpful?
3 out of 16 found this helpful