Does red teaming require an external company?

No. Internal, external, and mixed teams can contribute, with different strengths and independence considerations.

Can AI generate red-team tests?

Yes. Automated generation can expand exploration, but coverage and judging still require careful evaluation.

Does a zero-failure result prove safety?

No. It means no defined failure was observed within the specified exercise and conditions.

AI Glossary / Governance, ethics and safety / Intermediate

Red teaming

Definition

Red teaming is a structured, authorized effort to find weaknesses, harmful behaviors, or exploitable failure modes in an AI system. Testers deliberately challenge the system, often using adversarial scenarios, rather than only checking ordinary successful use. The purpose is to generate evidence that supports mitigation and better decisions about the system's use, not to declare it safe after a few unsuccessful attacks.

What it means

A conventional demonstration asks whether an assistant can complete a task. A red-team exercise asks how the assistant or its surrounding application might fail under pressure, misuse, misleading inputs, or unexpected circumstances. The target may include model outputs, access controls, tools, retrieval, or human workflows. AI red teaming is broader than testing offensive language alone. It can examine privacy leaks, instruction-boundary failures, unsafe actions, unreliable answers in consequential contexts, and other defined concerns. The exercise needs a clear scope and authorization so testing does not itself expose people or disrupt unrelated systems.

How it works

An exercise defines the system version, threat model, permitted methods, target behaviors, and rules for handling findings. Testers explore cases, record relevant inputs and outputs, and investigate whether apparent failures are reproducible and consequential. Human expertise and automated methods can complement one another. Findings are triaged by severity and context, then assigned to owners for mitigation. Retesting checks whether the fix addresses the failure without undermining legitimate tasks. A report should explain what was tested and what was not. Adaptive testing, where testers learn from responses, answers different questions from a fixed benchmark and should be described accordingly.

Three practical examples

An authorized team checks whether a retrieval assistant follows hostile instructions embedded in a test document.
Domain specialists examine whether a system gives unwarranted certainty when presented with ambiguous information and missing context.
Reviewers test whether an agent can be induced to propose an action outside its permitted workflow, using a sandbox that prevents real external effects. Each example requires a defined success criterion and evidence about actual system behavior.

Think of it this way

Think of a rehearsal in which designated participants deliberately introduce plausible problems so the team can discover weaknesses before real operations. The analogy is limited because AI behavior can vary across prompts, sessions, tools, and model versions. Rehearsing a finite set of scenarios cannot cover every future interaction, and passing the rehearsal does not prove that an unknown failure is impossible.

Worked example

Suppose an authorized exercise contains 40 distinct test scenarios, each attempted three times against the same system version. Testers observe a prohibited action proposal in 12 of the 120 attempts. Those failures occur in 7 distinct scenarios. The per-attempt observed failure rate is 12/120 = 10%, while the fraction of scenarios with at least one observed failure is 7/40 = 17.5%. Both describe this exercise, but they answer different questions. They do not estimate the real-world incident rate because the scenarios were deliberately challenging and not sampled from ordinary use. Repeated attempts may also be dependent. A useful report preserves both counts, the failure definitions, and the testing conditions rather than presenting a single decontextualized safety percentage.

Illustrative case study

Fictional case: Ashford Tools red-teams a document assistant before an internal rollout. Early testing finds that the model refuses an unauthorized request in its final answer but has already prepared a risky tool action. The team changes the application boundary and expands its evaluation to inspect proposed actions as well as visible text. It retests legitimate workflows after the change. This invented case illustrates why the whole system, not just the final conversational response, belongs in a red-team assessment.

Common misconceptions and mistakes

Red teaming is not a guarantee of safety, a substitute for ordinary quality testing, or permission to attack third-party systems. Finding many issues does not necessarily mean a team performed worse; it may indicate a more thorough exercise. Conversely, finding none can reflect limited coverage or weak tests. Automated judges can misclassify outcomes and need validation. A memorable failure demonstration may reveal a real problem without showing its prevalence. The exercise should also avoid treating every refusal as success if users can no longer perform legitimate tasks or receive appropriate help.

Business use

Connect red teaming to the system's intended use and plausible harms. Include people with relevant domain knowledge and perspectives that developers may miss. Protect test data and participants, and use controlled environments for consequential actions. Record enough detail to reproduce findings while limiting unnecessary circulation of exploitable material. Assign remediation owners and track unresolved limitations. Compare results only when scope, system access, attempt budgets, and scoring methods are sufficiently comparable. Repeat relevant tests after material changes, and combine red teaming with model evaluation, user studies, monitoring, and incident review.

Check your understanding

An exercise deliberately selected difficult cases and observed a 10% failure rate. Can that number be reported as the expected failure rate for ordinary users?

Answer: No. The test distribution and attack process differ from normal use. Report the result with its scope and use separate evidence for operational frequency.

Related terms

Prompt injection
adversarial testing
threat model
penetration testing
misuse evaluation
test coverage
regression testing
risk mitigation
safety evaluation

Learn next

Write a one-page exercise brief for a hypothetical assistant. Define authorization, system boundaries, one concrete failure condition, protected test data, and the evidence needed to confirm a finding. Add a legitimate task that must still work after any mitigation.

Sources

NIST CSRC, AI-context red teaming definition: https://csrc.nist.gov/glossary/term/red_teaming Ganguli and colleagues, original red-teaming methods research: https://arxiv.org/abs/2209.07858 Perez and colleagues, Red Teaming Language Models with Language Models: https://arxiv.org/abs/2202.03286 NIST, ARIA evaluation approach: https://ai-challenges.nist.gov/aria

Previous: RED (Regularization by Denoising) Next: Redaction