Must a usability test be in a laboratory?

No. Remote or field settings can be appropriate, depending on the question and practical constraints.

Is there one correct participant count?

No. It depends on the study's purpose, user diversity, measures, and desired strength of inference.

Can testing start before the product is complete?

Yes. Prototypes can help investigate interaction problems early, with their limitations made clear.

AI Glossary / Governance, ethics and safety / Advanced

Usability testing

Definition

Usability testing evaluates a product by observing relevant users attempting realistic tasks with it. It gathers evidence about successful completion, difficulties, effort, and experience. For an AI system, the test should include how people understand, check, correct, and use its outputs within the intended workflow.

What it means

The object of the test is the system and its interaction with people, not whether participants are clever enough to use it. A difficulty is evidence to investigate, rather than a reason to blame the participant.
Usability testing differs from asking whether people like a demonstration. Participants need an opportunity to act on meaningful tasks. Their behavior may reveal problems they would not mention in a survey or anticipate in an interview.
It also differs from evaluating model accuracy alone. A model can produce useful outputs that people struggle to find or interpret. Conversely, an attractive interface can make an incorrect output easy to accept. Both situations require examination of the complete interaction.

How it works

Define the study questions, intended users, tasks, and criteria for success. Select participants and conditions relevant to those questions. Prepare realistic task scenarios without describing the exact interface steps that would give away the solution.
Observe what happens and record the support provided. Depending on the study, evidence may include completion, errors, time, participant comments, and satisfaction. A test aimed at discovering design problems may use exploratory discussion; a performance comparison requires greater control over conditions and assistance.
Analyze where people struggled and what may explain it. Report the participants, task conditions, product version, measures, and limitations. Improve the design and reassess consequential issues. Small studies can reveal useful problems, but they do not automatically establish population-wide success rates or complete accessibility.

Three practical examples

Staff use an AI assistant to draft a response from supplied reference material. The test examines whether they notice an unsupported statement and can correct it, rather than merely whether they can generate a draft.
Users try to undo an AI-generated change in a document. Observers record whether they understand what will be restored and whether their own edits are preserved as expected.
Participants using relevant assistive technologies attempt the same core tasks as other users. Their observations complement technical accessibility evaluation; one participant's experience does not represent every disability or configuration.

Think of it this way

Think of watching someone prepare a meal with a new kitchen appliance. A polished demonstration tells you what the demonstrator can do. Observing a realistic attempt shows whether another person can find the controls, understand the feedback, handle a mistake, and finish the meal. Usability testing makes that distinction explicit.

Worked example

Consider an invented test with 12 participants. Each must use an AI-assisted document tool to locate a reference and prepare a correctly supported summary. Success criteria are defined before the sessions.
Eight participants complete the task correctly without help. Two complete it only after the facilitator points out where references are displayed. Two do not complete it correctly.
Unaided success is 8/12 = 66.7%. Completion including assisted attempts is 10/12 = 83.3%. Reporting only the second figure would hide the role of help that ordinary users may not receive.
Suppose five participants initially overlook the reference control. That is a useful observed design problem, but it is not proof that exactly 41.7% of all future users will overlook it. The study's recruitment and small sample limit that inference.
After a redesign, a new set of 12 participants produces ten unaided successes. The observed difference is promising, but without a suitable comparative design and analysis it should not be presented as proof that the redesign alone caused the improvement.

Illustrative case study

Fictional company Willow Documents tests an assistant for preparing internal briefing notes. Early sessions ask participants to “click the sources button and check the draft.” Most complete that instruction, so the team initially concludes that verification is easy.
The researchers realize that the task wording has supplied the action they wanted to investigate. They revise the scenario to ask for an accurate briefing note using the provided materials, without naming a control. They then observe whether participants independently find and use the supporting evidence.
The revised sessions reveal confusion about which statements are supported. The team improves the presentation and runs further tests. This fictional case illustrates how task design can influence observations. It does not mean that all instructions are inappropriate; participants still need a clear goal and realistic background information.

Common misconceptions and mistakes

“A product demonstration is a usability test.” Watching an expert differs from observing relevant users attempt tasks.
“Positive comments prove successful use.” Behavior and outcomes may tell a different story.
“Helping participants is always wrong.” Assistance can be useful for exploration, but it must be recorded and interpreted appropriately.
“A handful of users proves the product is accessible.” User observations need broader evaluation and clear scope.
“Fixing a reported issue proves the fix works.” The revised interaction still needs appropriate assessment.

Business use

Use testing to answer specific decisions: whether users can complete a critical task, recover from an error, or understand the status of an AI suggestion. Prioritize issues by their effect on task outcomes and the people affected, not simply by how often someone mentions them.
Keep test evidence separate from product claims. Report assistance, failures, and exclusions as well as successful attempts. For AI products, document the model and configuration used and retain relevant outputs so reviewers can understand what participants encountered. Account for variable outputs when interpreting or comparing sessions.

Check your understanding

Why distinguish assisted from unaided success?

Answer: They describe different levels of support required to achieve the goal.

What was wrong with naming the sources button in the case study?

Answer: It prevented the team from observing whether users would discover that action themselves.

Why report the tested version?

Answer: Later changes may alter the interaction and invalidate assumptions about the results.

Related terms

Usability
User-centered design
Human-computer interaction
task analysis
accessibility evaluation
formative evaluation
summative evaluation
user experience

Learn next

Write one realistic task scenario for an AI tool without naming interface controls. Define successful completion, decide what assistance will be allowed, and specify the observations needed to answer the study question.

Sources

NIST, Usability Testing: https://www.nist.gov/programs-projects/usability-testing NIST, A Standard Reporting Format for Summative Usability Evaluations: https://www.nist.gov/publications/standard-reporting-format-summative-usability-evaluations W3C Web Accessibility Initiative, Involving Users in Evaluating Web Accessibility: https://www.w3.org/WAI/test-evaluate/involving-users/

Previous: Usability Next: Use And Interpretation Bias