Card sort tasks in the new experience

At a glance

Learn how to set up card sort tasks and analyze their results in UserTesting. 

Card sorts help you understand how people naturally group and label information, making them a powerful tool for designing intuitive navigation, site structure, and content organization. For more information around what card sorts are and when to use them, refer to the article "what is card sorting?

On this page:

• Availability
• Create a Card Sort
• Analyze a Card Sort
  • Dendrogram tab
  • Placement tab
  • Similarity tab (open and hybrid only)
  • Categories tab
  • Cards tab
  • Responses tab
• FAQs

Availability

New Card sort tasks (open, closed, and hybrid) are available in the new experience for the following test types on desktop:

• new Think-out-loud test (qualitative) 
• Interaction test (quantitative)
   

Create a Card Sort task

Follow these steps to add a Card Sort to your test:

1. Go to the Test Builder tab of your test.
2. Click on the Add button on the left-hand side menu.
3. Choose Card Sort.
4. Choose a Type
   • Closed – Participants sort the cards into pre-defined categories
   • Open – Participants create (and name) the categories
   • Hybrid – You set some pre-defined categories, but participants can also create their own
5. Configure the card sort task
   • The instructions are pre-filled, but are editable.
     Participants will also get instructions on how the card sort feature works prior to the task. See FAQ section for a screenshot.
   • Under Cards
     • Type in your cards one by one or
     • Paste all the cards into the field for the first card.
       Each item should be on a new line.
   • Under Categories (Closed and Hybrid only)
     • Type in your categories one by one or
     • Paste all the categories into the field for the one.
       Each item should be on a new line.

Unsure how many participants you need for a card sort? Refer to our sample size recommendations article. 

Analyze a Card Sort

The results section offers various tabs for card sort tasks. Each tab shows the results of your card sort from a different angle, giving you a fuller picture of how participants grouped the cards. 

The tabs available as well as what you can do on some of them depends on whether it's a closed or open/hybrid card sort. This is mostly because of the need for standardization (see below).

Standardization in open and hybrid card sorts

Standardization is needed due to categories created by participants. This is the case in:

• open card sorts, where all the categories are created by participants
• hybrid card sorts, where potentially some of the categories are created by participants 

For example, imagine that we have 4 participants who created a category with substantial overlap with regards to the cards included and:

• participant 1 named the category "vegetables" 
• participant 2 named the category "vegetables" 
• participant 3 named the category "vegetabless" (typo)
• participant 4 named the category "veggies"

These 4 categories will appear as separate categories in your results until you manually merge them into one standardized category, called for example “vegetables”. You can do this following the steps below.

How to standardize categories

1. Go to the Categories tab of your card sort task results.
2. Using the checkbox on the left, select the categories you'd like to merge together.
3. Scroll to the very bottom of the table and click Standardize.
   
4. In the modal that appears, you can:
   • Select/de-select categories
   • Name the resulting standardized category
     
5. Click Standardize categories. 
   Once the categories have been standardized, you will see the agreement percentage appear on the right. Unstandardized categories will have a dash (–) instead of a percentage.
   
   Note: Already standardized categories can get merged with other categories.

How to unstandardize categories

1. Go to the Categories tab of your card sort task results.
2. Using the checkbox on the left, select the category (or categories) you want to unstandardize.
3. Click the unstandardize option which will appear at the bottom of the table.
   
4. Confirm your action.
   The selected categories will go back to their original state.

Dendrogram tab

 Note: This tab isn't affected by standardization.

A dendrogram is an interactive visual representation in the form of a tree that shows how participants grouped items together (based on co-placement behavior, not based on categories). 
 

A dendrogram consists of various elements including:

• Clusters which are the groupings created by participants and are shown as light blue boxes. 
• Agreement line which indicates the percentage of participants (33% in the example below) that agree that the current clusters are the "right" clusters. You can slide it left and right (refer to the "Agreement and clusters" section below).

How to interpret a dendrogram

A dendrogram illustrates the strength of relationships between items, and between groups of items:

• The further left two branches merge, the more participants agreed that these cards belong together.
  In the example above, the mango, apple, banana, and strawberry branches merge on the very left because they were grouped together by all participants.
• The further right two branches merge, the fewer participants agreed that these cards belonged together.
  In the example above, the fruits and vegetables branches merge first, before they merge with the furniture branch on the very right.

Agreement and clusters

The number of clusters is not static and depends on:

• the placement of the agreement line (at 33% in the example above) or
• the number of clusters you want (dropdown at the bottom left)

As you slide the agreement line between 100% and 0%, the number of clusters changes:

• Sliding it left gives you more clusters, but smaller and more homogenous ones.
  In the example, at 100% agreement we get 6 clusters: furniture, fruit, vegetables, and three singles — cucumber, tomato, and avocado — since participants disagreed on where they belong.
• Sliding it right gives you fewer clusters, but bigger and less homogenous ones.
  In the example, at 1% agreement, we get 2 groups: fruits/vegetables (a broader group) and furniture.

Linkage methods
Linkage methods define how the distance between clusters is calculated. Different linkage methods can produce different trees from the same data. You can choose between 3 linkage methods:

• Average linkage (most balanced)
  Uses the average distance between all pairs of cards across two clusters
• Single linkage (broader)
  Uses the distance between the closest pair of cards across two cluster
• Complete linkage (tighter) 
  Uses the distance between the furthest pair of cards across two clusters

Placement tab

Important note: In open and hybrid card sorts, you will need to first standardize categories for this tab to be useful.

The Placement tab provides a table where each cell shows what % of participants placed that card in a specific category. 

Similarity tab

Note: This tab is only available in open and hybrid card sort tasks. It isn't affected by standardization.

The similarity tab shows what percentage of participants placed each 2 cards into the same category. 100% means all participants placed these 2 cards together. 

Categories tab

Important note: In open and hybrid card sorts, you will need to first standardize categories for this tab to be useful. Actually, it's on this tab that you'll do the standardization.

The Categories tab provides a table which automatically indicates for each pre-defined category:

• all the cards placed in the category
• how many participants placed each card
• the agreement score that indicates the percentage of participants who agreed that these cards belong in a category.

About the agreement score in the categories tab

The agreement score for categories is expressed as a color-coded percentage:

Make sure your sample size follows best practice and consider whether participants thinking out loud during the task could add clarity.

Cards tab

Note: This tab is affected by standardization. For more information, see note below about open and hybrid card sorts.

The cards tab provides a table showing for each card:

• how many categories it was placed into
• the names of the categories it was placed into
• how many participants placed it in each category

Note about open and hybrid card sorts

For categories created by participants, initially for each card you'll get all the names of the categories into which the card was placed (see screenshot below). 

Once you standardize these categories, you'll only get the standardized category names (see screenshot below).

Responses tab

The responses tab shows for each participant's:

• their clusters as well
• their recording 
• their time-on-task

FAQs

Q: Do I need to add instructions around how a card sort works?
A: No, participants will receive instructions on how to sort the cards before the task  (see screenshot below). Participants can open the instructions back up whenever they need a reminder (“Open instructions” option at the top left).

Q: Will participants see the Card Sort while answering follow-up questions?
A: Participants will see their Card Sort responses in read-only view while they answer any follow-up questions added within the task group ("Popular follow-up questions" or "Add follow-up question" options).

Q: Do participants need to sort all the cards?
A: Yes, participants cannot continue to the next task until all cards have been moved into a category and will get a warning about it if needed.

Q: Can participants create a category and leave it unnamed?
A: While a category may stay unnamed while the participant is still sorting, they will be prompted to name the category so that they can submit their responses.

Q: Can participants complete a card sort task using their keyboard?
A: Yes. In addition to dragging and dropping cards into a category, participants can also use the dropdown on each card to move it into a category. This allows them to complete the task using their keyboard.

 

Q: Can I add an image to a card?
A: This functionality isn't available at the moment, but is being assessed. If you need to use images in your card sort, you can create a classic Card Sort instead.

Q: Can I change the type (open/closed/hybrid) of a card sort task after launching a test?
A: No, when a test gets launched, the card sort type gets locked for consistency across participants.

Q: When I standardize or unstandardize categories, am I affecting how other users in my workspace see the results of this card sort task?
A: Yes, standardization is not at the user level.
 

Q: Is manually merging categories reversible, and does it flow into exports?
A: Yes, manual merging is reversible, and yes, merged categories flow into exports.

Q: Do card sort results appear in the Excel/CSV export, and what do they contain for Open and Hybrid card sorts?
A: Currently, card sort results export as participant-level data showing each participant’s answers for every card sort selection. For open and hybrid card sorts, the similarity grid, card and category tabs, and other task data aren’t included in the export. To get that raw data, use the Copy to clipboard option and paste it into Excel for analysis.

Q: Should I use the New experience card sort, the Classic card sort, or the one on the IA platform?
A: We encourage you to use the New experience and raise any concerns with Classic or the IA platform. The new version can bulk copy cards, which the IA platform does not offer, and it has dendrograms, which the Classic card sort does not have.

Q: Can Open and Hybrid sorts be duplicated from Classic?
A: Yes, duplication from Classic works for Open, Hybrid, and Closed card sorts.