How we protect participant quality on UserTesting

At a glance

Learn how we maintain the UserTesting Network as a network of real, engaged participants, and what you can do when a session doesn't meet your standards.

This article is for anyone who runs tests with participants from the UserTesting Network.

How we keep participant quality high

We combine human review, technology safeguards, ongoing monitoring, and customer reporting tools. Together, these help you get feedback from real, engaged participants who match your criteria.

Quality checks happen at every stage of a participant's time with us:

• When they apply: Participants agree to our Terms of Service and Code of Conduct. We validate each participant's location, and each participant needs their own unique PayPal account. Our location checks also look for attempts to hide or fake a location, such as using a VPN.
• Before they take paid tests: Each applicant completes a practice test that a person reviews.
• While they take tests: Ongoing checks and audits look for signs of fraud or low-quality participation.

No participant network can promise that every session is free of quality issues. That's why we also make it easy for you to report a session.

When does a person review participants?

Our Panel Operations team of quality specialists coordinates reviews of participants at these points:

• Practice test: A person reviews every applicant's practice test. They check that the participant can follow tasks, think out loud, and complete a test without technical issues, and that they're in the location they claim.
• First test: A person reviews every new participant's first test.
• Reported problems: We review reports about participant quality and technical issues. Reports about study setup aren't reviewed for participant quality.
• Low-rated sessions (Classic experience): We review every session that a customer rates 1 or 2 stars.
• Live Conversations: We monitor participant attendance and quality concerns to help maintain a reliable Live Conversation experience. To learn how no-shows are handled, read Live Conversation: New experience or Live Conversation participant no-show (Classic experience).

What happens after a review?

Our quality specialists look at the session and the participant's profile, then take action when it's needed. Depending on what they find, this can include coaching the participant, canceling their payment, or removing them from the UserTesting Network.

In the Classic experience, participants with low ratings are also less likely to receive test invitations.

How do you protect quality on surveys and interaction tests?

Surveys and interaction tests don't include spoken feedback, so we protect quality with participant validation, in-test checks, and ongoing monitoring:

• Before they participate: Every participant in the UserTesting Network completes several verification steps, including location validation, a unique PayPal account, and agreeing to our Terms of Service and Code of Conduct.
• Speed checks: We disqualify participants who move through tasks too quickly. A participant is disqualified if they speed through 2 tasks in a row and through 20% of the tasks in the test. The minimum time for a task depends on the task type. Disqualified participants aren't paid. Learn more in Screener questions for the New experience.
• Ongoing monitoring: When trends or concerns come up, our Panel Operations team investigates and takes action.

How do you keep bots and fake accounts out of tests?

We use fraud detection technology when participants sign up, before they start tests, and throughout their time in the network. These checks help confirm that each participant is a real person with one account, located where they say they are.

How can I improve participant quality on my tests?

How you set up your screener, test plan, and analysis also helps keep out participants who misrepresent themselves. A few quick tips:

• Avoid yes or no questions and leading questions in your screener.
• Add a "red herring" to your screener questions: an answer option that no real participant would choose, such as a made-up product or brand. Participants who select it are likely selecting answers just to qualify.
• When it fits your study, add a task that validates that a participant is the right fit for your study. For example, ask them to show their profile on an app they say they use.
• If you ask participants to share or show personal information, let them opt out. This applies to your screener questions as well as your test.
• When you review results, check answers to validation questions first, and look for sessions that are very brief or lack depth.
• Plan your review time by test type. Responses without video, such as surveys, are easier to fake than sessions you can watch, such as Live Conversations, so allow more time to check them.

For the full list of best practices, read Avoid and handle imposter participants.

Report a problem with a session

If a session doesn't meet your standards, report it from the Results tab, the Session Player, or the Monitor page. When you report a problem, you choose how to resolve it:

• Replace the participant: Get credit for the session and recruit a new participant.
• Refund the session: Get credit for the session without a replacement.
• Keep the response: Keep the session and report the problem only.

In the Classic experience, you can also rate sessions. Our quality specialists review every session rated 1 or 2 stars.

Learn more in Report a Problem with a Session or Participant.

For more help, contact our Support team using the support request form.

Frequently asked questions

Q: Can you guarantee every participant is high quality?
A: No, we can't guarantee that every session is free of quality issues. Our checks reduce the risk, and you can report any session that doesn't meet your standards.

Q: What if a participant uses AI to answer?
A: Using AI to generate responses breaks our participant Code of Conduct. When we confirm that a participant has done this, we remove them from the UserTesting Network. If you suspect AI-generated answers, report the session.

Q: Do these reviews apply to participants I recruit myself?
A: No. The reviews in this article apply to the UserTesting Network. Our Panel Operations team doesn't review sessions from participants you recruit yourself through Invite Network or Custom Network. To learn what those participants experience, read Invite Network participant experience.

Related content