AI Moderation (Closed beta)

On this page:

• Availability
• What you need
• Enable the Moderator
• Prompting the Moderator
• What participants experience
• Time limits
• Reading results
• Language
• Features not yet included at beta release
• Troubleshooting
• Giving feedback
• Related content

AI Moderation adds an AI moderator to a Think Out Loud (TOL) test. The moderator reads the test to the participant and, after any verbal response question, asks up to three follow-up questions based on the participant's reply and a customer prompt. This gives deeper, higher-quality insight on verbal response questions at scale.

This is beta functionality, available to nominated customers only. It is switched on per account during the beta, and the behaviour described here will change as we iterate.

AI Moderation is not available to accounts that have opted out of generative AI features.

• Access to the New Think Out Loud test type.

• Participants taking the test on desktop with the UserTesting browser extension.

  • Mobile and extensionless sessions are not yet supported in this beta.

1. Open your new TOL test in the test builder.

2. Open the test settings modal by clicking ‘Edit’.

3. Toggle on ‘AI Moderation’.

4. (Optional) For any verbal-response question, add context to the AI moderation prompt so the agent probes participants correctly.

5. Use Preview to test the moderator and your prompt.

6. Launch the test as usual.

If the setting is off, AI moderation options are hidden and the test behaves as before.

Note: the AI moderator will ask follow ups for any verbal-response questions if the setting is enabled. If no prompt is entered, our system prompt will verify the participant’s answer without adding further context about what to probe.

When the setting is enabled, each verbal response question includes an optional free-text prompt of up to 1,000 characters. Use it to tell the moderator what you want to learn so follow-ups go where you need them.

You can write the prompt several ways. If you have specific lines of questioning, be as precise as possible for the AI. You can use a moderated research plan as a reference. For example:

[Question] Have you personally used, tested, or directly evaluated an AI coding tool?

[Prompt] If yes:

• Which tool or tools have you used, tested, or evaluated?

• What you have personally done with [tool] so far.

Further Probes: 

• Was it a demo, an internal experiment, a pilot, or a real customer study?

• Approximately how many times have you used it?

• How did you carry out the evaluation?

• Which parts of the workflow did you personally complete?

• Did the findings contribute to a real decision?

• How recently did you use it?

Some things to know:

• Keep the prompt focused on research intent. It steers the questions; it does not script them verbatim.

• If certain probing questions have priority, note that in the prompt.

• UserTesting has moderator instructions on safety, tone and question quality. If your prompt conflicts with those, the moderator will follow our instructions.

• The prompt applies to each verbal response question, not to the entire test.

The moderator reads the test aloud and speaks the questions using synthesised speech. Your questions are read verbatim as you typed them. Speech remains on for the entire test whenever AI moderation is enabled.

When AI is active, the moderator's questions also appear on screen so participants can read them. The moderator speaks only after the participant clicks Next on the verbal response question and does not interrupt mid-answer. Participants click Next to move between follow-ups, as with any other question.

Participants are told before starting that the test is AI-moderated and that they should turn their audio on.

Two limits apply to each agent conversation. They apply to each verbal response independently; whichever is reached first ends the exchange:

• A maximum of three follow-up questions.

• A hard three-minute limit on the moderated exchange.

If the participant is still speaking when the three-minute limit is reached, they may finish their response and then move to the next question, but the moderator will not ask another question.

All follow-ups count toward the three, including prompts to expand ("can you say a bit more about that?") and probes for depth ("why did that feel confusing?").

We apply these limits to ensure quality, though we expect to refine these limits and their configurability throughout the beta.

Moderated sessions appear in your Results alongside other sessions. There is no separate view to learn.

• Recording. The session audio combines the moderator's questions and the participant's answers on a single track.

• Transcript. The full exchange is transcribed.

• Verbal themes. Themes are generated from the participant's responses to all follow-up questions, including themes from the AI moderator's follow-ups.

The first version of our AI Moderation agent supports only English at the launch of the beta. Language selection will not impact the AI moderator. This is an area we expect to develop further throughout the beta.

• Mobile and extensionless participant sessions,

• Follow-ups on question types other than verbal response,

• Turning moderation on or off for individual verbal response questions,

• Configuring the number of follow-ups yourself,

• Predefined prompt templates.

I can't switch AI moderation on. If your account is not in the beta or has opted out of generative AI features, the setting will not appear.

A session has no follow-up questions. If the moderator fails mid-session, the participant moves on to the next question and finishes the test. The session is still valid and still appears in your results, just without the moderated exchange. That noted, please inform us if this happens.

The follow-ups weren't useful. Send us the session. Prompt quality and question quality are key things that we are tuning during the beta, and specific examples are the most useful thing you can give us.

This is a closed beta and your feedback shapes what ships next. Please refer any observations or feedback to the Product team leading the beta.

Related content