Is stress testing the same as red teaming?

No. Red teaming often actively searches for exploitable or harmful behavior; its scenarios can overlap with stress tests.

What is a spike test?

In service performance testing, it examines a sudden surge in demand.

It examines sustained operation over an extended period, which can expose problems that a short run misses.

AI Glossary / Governance, ethics and safety / Advanced

Stress test

Definition

A stress test deliberately evaluates an AI model or system under demanding conditions to reveal weaknesses, limits, and failure behavior. The stress may involve difficult inputs, heavy demand, disrupted dependencies, or changing operating conditions. Its meaning depends on what is being tested: a model's reasoning can be stressed without increasing traffic, while a service can be overloaded with otherwise ordinary requests.

What it means

Routine evaluation asks how a system performs under selected normal conditions. Stress testing examines challenging conditions that routine averages may conceal. A result is meaningful only when the stress, expected behavior, and measured outcomes are specified.
For an AI assistant, answering accurately and answering promptly are different requirements. A server returning every response successfully does not establish that those responses are correct. Conversely, a model may produce excellent answers when tested individually but become unusable when the surrounding service cannot handle demand.

How it works

Start with a concrete question: will an extraction model handle degraded scans, will an assistant behave appropriately when retrieval fails, or will a service meet its response target during peak demand? Establish a baseline and choose stress levels relevant to that question.
Define success and failure before running the test. Preserve information about the model version, environment, inputs, workload, and dependencies so comparisons are interpretable. Observe both the stressed period and recovery afterward. A system that resumes answering may still have an unresolved backlog.
Keep model capability tests separate from infrastructure tests when diagnosing a problem, then examine important combinations. Check that the test equipment itself can generate the intended conditions. An overloaded load generator can distort the apparent performance of the system being evaluated.

Three practical examples

A receipt-reading model is evaluated on progressively blurred images. Reviewers measure field accuracy and whether the system flags unreadable content instead of inventing values. The stress concerns input quality.
A recommendation service receives a planned increase in simultaneous demand. Testers record delays, failed requests, and recovery after demand falls. The stress concerns operational capacity.
A language inference model receives deliberately challenging sentence pairs. Researchers examine whether benchmark success carries over to those linguistic conditions. Naik and colleagues used this kind of stress evaluation to investigate weaknesses in sentence-encoder models.

Think of it this way

A rehearsal can include a missed cue, an absent performer, and a crowded entrance. Each challenge reveals something different about the production. Successfully handling one rehearsal scenario does not prove readiness for every disruption, and the rehearsal must still resemble situations worth preparing for.

Worked example

Consider a fictional assistant tested with the same fixed 200 questions under two demand levels. Each question has an independently checked answer, and the team counts success only when a correct answer arrives within three seconds. Unanswered questions remain in the denominator.
At the baseline level, 190 answers are correct and timely, six are correct but late, and four are incorrect or missing. The combined success rate is 190 / 200 = 95%.
At the higher level, 170 are correct and timely, 20 are correct but late, and ten are incorrect or missing. The combined success rate is 170 / 200 = 85%, a decline of ten percentage points.
Counting correct but late answers as successful would produce different figures: 196 / 200 = 98% at baseline and 190 / 200 = 95% under stress. Those percentages answer a different question and do not satisfy the timing requirement.
Suppose the agreed target was at least 90% correct and timely. This run misses that target under the higher demand. Repeated controlled runs and diagnostic evidence are still needed to understand variability and locate the cause. The result does not identify the system's universal maximum capacity.

Illustrative case study

Fictional company CedarManuals tests an assistant before a product launch. Ordinary test questions perform well, so its team initially focuses on adding server capacity for increased traffic.
A separate disruption test makes the document retrieval service unavailable. The assistant still responds quickly, but some answers rely on unsupported guesses. The team discovers that speed monitoring alone would miss this failure.
It changes the workflow so the assistant clearly indicates when supporting material cannot be retrieved and offers an appropriate alternative route. Testers repeat the disruption scenario, inspect answer behavior, and check recovery when retrieval returns. This illustrative case shows why a useful stress plan considers dependencies and content quality as well as request volume; it does not report actual product results.

Common misconceptions and mistakes

“Stress testing means sending more requests.” That is one operational form; difficult inputs and adverse conditions can stress other capabilities.
“A passing test proves robustness.” It supports a bounded claim about the tested scenarios and measurements.
“Any extreme scenario is useful.” A scenario needs a purpose. Implausible conditions can distract from consequential weaknesses.
“Stress tests estimate the everyday failure rate.” Deliberately concentrated difficult cases usually do not represent normal traffic frequencies.
“The system failed, so the model is responsible.” Failures may arise in retrieval, queuing, interfaces, dependencies, or the test setup itself.

Business use

Use stress testing to inform launch scope, capacity planning, fallback design, and improvement priorities. Attach each finding to a specific condition and business consequence. A slow but correct answer and a fast fabricated answer call for different responses.
Preserve results by scenario rather than hiding them in one overall score. Assign corrective work and repeat relevant tests after changes. The aim is a better understanding of operating limits and responses, not merely a reassuring pass label.

Check your understanding

Why include unanswered requests in the worked example's denominator?

Answer: Excluding them would hide failures to deliver the required service.

Can a throughput test establish answer truthfulness?

Answer: No; content quality requires suitable evaluation.

What should a report say after all chosen scenarios pass?

Answer: That the specified criteria were met in those scenarios, with remaining coverage limits identified.

Related terms

Robustness
Resilience
Red teaming
Reliability
load testing
adversarial testing
failure mode
recovery testing

Learn next

Write one challenging scenario for an AI service. Specify the stress, baseline, success criterion, recovery observation, and evidence needed to investigate a failure.

Sources

NIST, AI RMF Playbook, Measure: https://airc.nist.gov/airmf-resources/playbook/measure/ Naik and colleagues, Stress Test Evaluation for Natural Language Inference: https://arxiv.org/abs/1806.00692 Grafana Labs, Automated performance testing: https://grafana.com/docs/k6/latest/testing-guides/automated-performance-testing/ Grafana Labs, Running large tests: https://grafana.com/docs/k6/latest/testing-guides/running-large-tests/

Previous: Streetlight Effect Bias Next: Strict dominance