Does this replace exploratory testing?

No. Exploration can uncover missing factors, unexpected states, and failure modes outside the design.

Must all tests repeat each combination equally?

That is the orthogonal-array requirement; covering-array testing relaxes it.

Can the approach test AI systems?

Yes, when factors and acceptance rules are meaningful, with separate treatment of stochastic variation and broader evaluation needs.

AI Glossary / Statistics and probability / Intermediate

Orthogonal array testing

Definition

Orthogonal array testing selects software or system test configurations using an orthogonal array: a table in which every combination of levels across any chosen t factors occurs equally often. The strength t determines the interaction coverage. Each row becomes a test configuration, with an expected result or another explicit correctness check.

What it means

A system may have many configurable inputs, making exhaustive testing expensive. Orthogonal array testing organizes a smaller set of configurations so that specified lower-order combinations are represented evenly.
It is a way to choose tests, not a complete quality guarantee. Its usefulness depends on the selected factors, representative levels, interaction strength, and ability to recognize incorrect behavior. It can reveal interactions among settings while still missing faults outside the modeled combinations or requiring more factors than the chosen strength.

How it works

Identify the system boundary and list factors that can influence its behavior. These might include authentication mode, file type, feature settings, and resource conditions. Choose levels that exercise meaningful classes, including relevant boundaries and error conditions.
Construct or select an array whose columns accommodate those factors. State the strength and verify the frequency of every required combination. For constrained systems, ensure that settings are valid and that feasible interaction coverage is actually retained.
Translate each row into a reproducible test: initial state, inputs, configuration, execution steps, and expected behavior. Record environment and software versions so failures can be investigated.
Use an appropriate test oracle. This may be an exact expected output, a trusted reference, an invariant, or a carefully designed relational check. AI outputs can vary, so define measurable acceptance criteria and repeated trials when needed instead of assuming one successful response proves reliability.
Run the suite, investigate failures, and add targeted regression cases. A failure observed under one configuration may involve an interaction or another hidden condition; the array alone does not identify its cause.
Covering arrays often serve a similar testing purpose with a less restrictive balance requirement. They ensure required combinations occur at least once rather than equally often. Choosing between designs should follow coverage needs and constraints, not a belief that equal repetition automatically makes a software suite superior.

Three practical examples

Document processing: testers combine supported input types, optional preprocessing, output modes, and storage settings to reveal configuration-dependent faults.
AI application deployment: a team tests selected combinations of access controls, retrieval settings, and request categories against explicit acceptance rules.
Device software: engineers combine operating modes and sensor settings, then verify both outputs and safe responses to invalid conditions.

Think of it this way

Think of a carefully planned tasting schedule. Each required combination of ingredients appears in the schedule, but somebody still has to judge the result. A well-arranged schedule cannot compensate for missing ingredients in the plan or an unreliable judgment method.

Worked example

Consider an original illustration with four binary settings A, B, C, and D. Generate all eight combinations of A, B, and C, and set D to 0 when A + B + C is even and 1 when it is odd.
The rows are 0000, 0011, 0101, 0110, 1001, 1010, 1100, and 1111, written in A,B,C,D order.
Every selection of three columns contains all eight possible triples exactly once. For example, any given A,B,D values uniquely determine C through the parity rule. The same reasoning works when any other column is omitted. Enumeration verifies all four three-column projections.
This is a strength-three orthogonal array with eight runs. Each pair of columns contains its four possible pairs twice. Exhaustive testing would require 2^4 = 16 configurations, so the planned suite uses 50% fewer configurations.
However, 1110 is absent. A fault triggered only when A, B, and C are on while D is off would escape this suite. The saving is a reduction in planned configurations, not a demonstrated 50% reduction in total testing cost or a proof of complete defect detection.

Illustrative case study

Illustrative fictional case: Alder Forms builds an AI-assisted document workflow with four optional features. Its test lead creates a configuration suite and attaches checks for permissions, output structure, and preservation of required source fields.
One test reveals an inconsistent output, prompting repeated runs and investigation of an unmodeled document property. The team adds that property to the test model and retains a direct regression case. This fictional case shows why a balanced suite needs investigation and revision rather than being treated as a final certificate of correctness.

Common misconceptions and mistakes

Do not confuse coverage of input combinations with coverage of all code paths or real-world behavior. Do not assume that pairwise testing detects every interaction fault. A modeled setting may still contain many distinct inputs, and hidden state can change the outcome.
Avoid deleting invalid rows without recomputing coverage. Do not count an executed test as passed when its outcome was never checked. Equal frequencies do not by themselves establish statistical confidence in a stochastic AI system.

Business use

Businesses can use orthogonal array testing to make configuration coverage explicit and auditable. It helps teams explain why particular tests were selected and where additional testing is needed. Cost decisions should include oracle development, setup, investigation, repeated execution, and maintenance, not only the number of array rows.

Check your understanding

How many configurations are saved in the example?

Answer: Eight of the sixteen full configurations.

Why can a fault at 1110 remain undetected?

Answer: The suite covers every triple, but that fault depends on a specific four-factor combination omitted from the rows.

Related terms

Orthogonal array
covering array
pairwise testing
combinatorial testing
test oracle
regression testing
configuration testing
interaction coverage

Learn next

Next study constrained covering arrays and test-oracle design. Then compare configuration coverage with requirements coverage, sequence testing, and repeated evaluation of systems whose outputs vary between runs.

Sources

Kuhn, Kacker, and Lei, Practical Combinatorial Testing, NIST SP 800-142: https://nvlpubs.nist.gov/nistpubs/legacy/sp/nistspecialpublication800-142.pdf — mathematical array definitions and practical testing context. Kacker, Combinatorial (t-way) testing for software: an adaptation of design of experiments: https://www.nist.gov/publications/combinatorial-t-way-testing-software-adaptation-design-experiments — primary overview of pairwise and higher-strength testing. The parity construction, coverage calculations, and fictional case are original.

Previous: Orthogonal array Next: Orthogonal contrasts