AI Glossary / Causal inference and experimentation / Intermediate
Test and Evaluation, Verification and Validation
Definition
Test and Evaluation, Verification and Validation, commonly abbreviated TEVV, is a collection of activities that builds evidence about an AI system's behavior, conformity to requirements and suitability for its intended use. The activities span development, integration, deployment and operation rather than forming only a final acceptance test.
What it means
Testing generates observations under specified conditions. Evaluation interprets evidence against criteria or objectives. Verification examines whether stated requirements have been met, while validation examines whether the system meets the needs of its intended use and environment.
These activities overlap: a test can supply evidence for both verification and validation. The distinction lies in the question being answered, not simply which team runs the test or which software tool is used.
For AI, a good benchmark score is only part of the picture. The complete system includes data, models, interfaces, human users and operating processes. Evidence about a model component may not establish the behavior of the workflow in which it is embedded.
How it works
Begin by identifying the intended use, affected users, system boundary and relevant requirements. Translate broad expectations into observable questions, such as whether an extraction system preserves required fields and supports effective review.
Select methods appropriate to those questions: controlled tests, analysis, inspection, demonstrations, user studies or operational monitoring. Define datasets, scenarios, comparison conditions and acceptance criteria before interpreting results. Record limitations and uncertainty along with successes.
Link each finding to the version and requirement it addresses. Resolve problems and repeat affected checks after changes. NIST's AI RMF 1.0 places TEVV throughout the lifecycle and encourages appropriate separation of evaluation roles. The purpose is an accountable body of evidence, not merely a large count of tests.
Three practical examples
- A document-processing team tests extraction accuracy, verifies required output fields, and validates whether reviewers can catch consequential mistakes in the intended workflow.
- A support-assistant project evaluates groundedness and checks that escalation requirements work when the source material is insufficient.
- An operations team monitors a deployed classifier for changed input patterns and reassesses whether earlier performance evidence still applies.
Think of it this way
Think of evaluating a newly designed delivery trolley. You can measure its turning radius, check whether that radius meets the specification, and observe whether staff can use it effectively in the actual aisles.
The analogy separates measurement, conformance and fitness for use. An AI system adds further issues such as uncertain outputs, changing data and human interpretation. Passing a well-defined component test does not remove the need to examine the complete task in context.
Worked example
One system, four different questions
Consider an invented invoice-extraction prototype. A controlled test uses 200 representative documents and finds that 190 have every required field extracted correctly. The measured document-level success rate is 95%.
Evaluation compares this result with the project's stated criteria, examines uncertainty and separates performance by document type. Verification also checks a requirement that missing fields must be explicitly marked: every output is inspected for the required missing-value indicator.
Validation then asks whether accounting staff can complete the intended review task reliably and efficiently using those outputs. A pilot discovers that the interface hides missing-value indicators below the visible area.
The extraction result and output-format check have not become false. They simply did not establish that the workflow met its intended need. The interface finding requires correction and renewed evidence about the revised system. None of these teaching figures is a claim about a real product.
Illustrative case study
Fictional Rowan Services buys an AI document tool after seeing a favorable model benchmark. During integration, its team discovers that scanned attachments follow a different preprocessing route from ordinary files, producing errors the benchmark did not cover.
The team creates a traceable evaluation plan covering both routes, records unresolved cases and tests the review interface with staff. After an upstream model update, it reruns the checks affected by changed behavior rather than assuming the earlier report remains sufficient.
The lesson is that evidence belongs to a defined system, use and configuration. A vendor's useful component results can inform the work, but they do not substitute for assessing the buyer's complete implementation.
Common misconceptions and mistakes
TEVV is not a synonym for model accuracy, and it is not automatically a certification. A report should state what was assessed and what conclusions the evidence supports.
Verification is broader than formal mathematical proof: inspection, analysis, demonstration and testing can provide verification evidence. Validation in this systems sense is also broader than an ML validation split used for tuning.
Avoid treating the four words as a rigid one-pass sequence. Early findings can change requirements or design, and operational evidence can trigger reassessment. No finite test collection proves that every unobserved condition will be handled correctly.
Business use
A business can use TEVV to make procurement and release decisions more concrete. Ask which requirements are supported, which realistic scenarios were examined, which failures remain, and who is responsible for interpreting the evidence.
Maintain a compact evidence record connecting requirements, methods, results and corrective actions. Include the intended user workflow and relevant operating conditions. Decide which changes trigger renewed assessment. The value is better-informed decisions and clearer accountability, not the appearance of assurance created by a long checklist or a single impressive benchmark.
People also ask
Is TEVV only performed before deployment?
No. Development choices, integration changes and operational conditions can all require additional evidence throughout the lifecycle.
Can the same test support verification and validation?
Yes, if it provides evidence relevant to both a stated requirement and an intended-use question. The conclusions should remain explicit.
Does using the NIST AI RMF mean NIST approved the system?
No. The framework is a voluntary risk-management resource. Applying it does not by itself constitute NIST endorsement or a system-specific certification.
Check your understanding
A model meets its documented accuracy threshold, but users cannot recognize when its output is unsuitable for their task. Does the accuracy result establish successful validation of the complete workflow?
Answer: No. It supports one performance claim under the tested conditions. Fitness for intended use also depends on the interface, users, operating context and other relevant requirements.
Related terms
- Evaluation
- Verification
- Validation
- AI risk management
- Testing
- requirements traceability
- acceptance criteria
- system boundary
- operational monitoring
Learn next
Start by separating a user need from a technical requirement and a test result. Build a small traceability table for one AI workflow, linking each requirement to evidence and unresolved limitations. Then examine which changes to data, model, interface or use would invalidate part of that evidence.
More in Causal inference and experimentation
- Squared hinge loss - Squared hinge loss is a classification loss that squares the positive shortfall from a target margin.
- Structural causal model - A structural causal model, or SCM, represents how variables are generated by causal mechanisms.
- Super learner - Super learner is an ensemble-learning approach that uses cross-validation to select or combine predictions from a library of candidate algorithms according...
- Test loss - Test loss is the value of a chosen loss function on a dataset reserved for evaluating a trained and selected model.
- Thurstone model - A Thurstone model represents comparative judgments using latent values disturbed by normally distributed variation.
- Top-k accuracy - Top-k accuracy is the proportion of evaluated examples for which the target label appears among the k highest-ranked predictions.
Sources
Sources: NIST, AI Risk Management Framework 1.0: https://doi.org/10.6028/NIST.AI.100-1 NIST, AI actor tasks and lifecycle TEVV: https://airc.nist.gov/airmf-resources/airmf/appendices/app-a-descriptions-of-ai-actor-tasks/ NASA, Product Verification: https://www.nasa.gov/reference/5-3-product-verification/ NASA, Product Validation: https://www.nasa.gov/reference/5-4-product-validation/ NIST, AI Metrology Center and limits of resource inclusion: https://airc.nist.gov/metrology/
Previous: Ternary search tree Next: Test data management