Is testing the same as quality assurance?

No. Testing evaluates products and work products; quality assurance also concerns the processes used to develop and maintain quality.

Can testing be fully automated?

Many repeatable checks can be automated, but choosing meaningful questions and interpreting unexpected results still requires judgment.

How do you test variable AI outputs?

Use appropriate criteria, repeated trials where relevant and documented evaluation methods. Exact text matching is often insufficient, but some surrounding requirements remain deterministic.

When should testing begin?

Reviewing requirements and design can reveal problems before executable software exists.

AI Glossary / Governance, ethics and safety / Advanced

Software testing

Definition

Software testing is a set of activities used to evaluate software and related work products, discover defects and provide evidence about whether requirements and intended needs are met. It includes examining items without running software, such as reviewing requirements, as well as executing software and checking its behavior. Testing informs decisions; it does not generally prove that a complex system has no remaining defects.

What it means

Testing asks concrete questions. Does it calculate the right amount? Does it handle an empty input? Can a user complete the intended task? Does it remain responsive under the expected workload? Different questions require different evidence.
For AI-enabled software, familiar questions about permissions, interfaces and data handling remain important. Additional evaluation may examine prediction errors, generated content, robustness and differences across relevant groups. A model can perform well on a benchmark while the application around it fails to deliver the intended service.
Testing is distinct from debugging, which investigates and fixes the causes of observed failures. A failed test also needs investigation: the software, test expectation, test data or environment could be wrong.

How it works

A team identifies the test basis, such as requirements, risks and expected user activities. It designs checks with suitable inputs, conditions and ways to judge the results. Tests may focus on individual components, their interactions or the complete service.
For predictable calculations, an exact expected value can be appropriate. For a summarization feature, several phrasings may be acceptable, so evaluation might assess source fidelity, required information and unsupported claims using a defined rubric. A method for judging correctness is called a test oracle.
Results should identify the version, configuration and conditions tested. After a fix, confirmation testing checks the corrected behavior, while regression testing looks for unintended effects elsewhere. Testing effort should reflect risk and uncertainty rather than merely maximizing the number of executed checks.

Three practical examples

Online ordering: Tests check ordinary purchases, invalid quantities and values around a discount boundary. They also examine whether totals remain correct when a user changes the basket.
AI search: Tests check both the relevance of retrieved material and whether access restrictions prevent a user from receiving documents they are not entitled to see.
Personal scheduling: Tests examine overlapping appointments, time-zone changes and failed calendar connections. A convincing suggested schedule is insufficient if saving it creates duplicate events.

Think of it this way

Testing a bicycle includes checking individual brakes, trying them after assembly and riding under relevant conditions. Passing one check does not answer every question about the complete bicycle. Similarly, software testing uses different views of a product, and the relevance of those checks matters more than a large total of easy successes.

Worked example

A fictional application has 200 planned test cases. During a run, 180 execute: 171 pass and 9 fail. The remaining 20 are blocked because the required environment is unavailable.
The pass rate among executed tests is 171 / 180 = 95%. The proportion of planned tests executed is 180 / 200 = 90%. The proportion of all planned tests that passed is 171 / 200 = 85.5%.
Reporting only 95% hides the blocked work. None of the percentages means that the application has a 95% probability of being correct in use, because the cases were not necessarily sampled to estimate that quantity.
Suppose one failure exposes an access-control defect while the other eight concern minor formatting. The overall percentage does not capture that difference in consequence. A release decision needs the findings, their severity, coverage gaps and intended use, not just the pass count.

Illustrative case study

Fictional company Maple Answers updates an AI document assistant. Its existing tests check that the service returns text and that the interface remains available. All pass, but a reviewer notices that answers sometimes use documents outside the selected project.
The team adds tests covering project boundaries and checks the retrieval-to-answer workflow. It reproduces the failure, corrects the filtering issue and verifies both the affected behavior and related permitted-access scenarios. It also evaluates whether the resulting answers are supported by the authorized material.
This fictional case illustrates why tests should examine meaningful requirements, beyond confirming that software runs.

Common misconceptions and mistakes

“All tests pass, so the software is defect-free.” The suite may omit important conditions or use incorrect expectations.
“More tests always mean better testing.” Repeated checks of the same easy behavior may add little evidence.
“Code coverage measures correctness.” It describes which code structures were exercised under a specified measure, not whether every result was properly checked.
“AI evaluation replaces application testing.” Model evaluation does not cover every integration, permission or workflow issue.
“A failed test is automatically a product defect.” Investigate the evidence and test conditions before drawing that conclusion.

Business use

Define acceptance criteria around the service people need and prioritize failures with serious consequences. Give reviewers a clear account of what was tested, what remains untested and which problems are unresolved.
For an AI feature, include representative and difficult cases, explicit evaluation criteria and checks of the surrounding software. Keep model and application versions traceable so results remain interpretable. Revisit tests when requirements or operating conditions change. Production monitoring can reveal new issues, but it does not replace suitable evaluation before a consequential release.

Check your understanding

Why should blocked tests be reported separately?

Answer: They provide no executed evidence about the behavior they were intended to check.

Why is one serious failure more important than many minor successes?

Answer: Test counts do not measure the consequences of defects.

What distinguishes confirmation from regression testing?

Answer: The former checks the fix; the latter checks for unintended effects on other behavior.

Related terms

Verification and validation
Test case
test oracle
regression testing
acceptance testing
code coverage
model evaluation
quality assurance

Learn next

Practice deriving tests from requirements, including boundary and failure conditions. Then compare exact assertions with evaluation rubrics for AI outputs, and learn how coverage measures support rather than replace judgment.

Sources

ISTQB Foundation Level Syllabus v4.0.1, testing concepts, activities and distinctions; copyright ISTQB and syllabus authors. https://istqb.org/?download_id=3345&sdm_process_download=1 ASTQB, Testing Principles, limitations and context of testing. https://astqb.org/1-3-testing-principles/ NIST AI RMF 1.0, Core, evaluation in the intended AI context. https://airc.nist.gov/airmf-resources/airmf/5-sec-core/ The test-count calculations, practical examples and Maple Answers case are original teaching illustrations.

Previous: Software mining Next: Solid compression