Does pairwise testing mean running two inputs at a time?

No. Each test usually assigns values to all modeled factors, and the suite collectively covers every required pair.

Is the smallest pairwise suite always easy to find?

No. Construction can be challenging, especially with many values and constraints; practical tools may produce useful suites without proving global minimality.

Does 100% pairwise coverage mean 100% defect coverage?

No. The percentages describe different things, and higher-order or unmodeled conditions may still cause failures.

AI Glossary / Statistics and probability / Intermediate

All-pairs testing

Definition

All-pairs testing, also called pairwise testing, is a combinatorial software-testing method that ensures every relevant combination of values for every pair of input factors appears in at least one test. It reduces the number of test configurations compared with exhaustive testing while providing a defined level of interaction coverage.

What it means

The method tests pairs of factor values, not just pairs of factors in the abstract. If browser and language are factors, coverage includes every allowed browser-language value combination. One complete test configuration can cover many such pairs at once.

Pairwise coverage is a structural property of a test suite. It does not guarantee that the software is correct or that every defect will be triggered. Failures requiring three or more conditions together, particular event sequences or omitted input values can remain undetected. It is also different from statistical pairwise hypothesis testing.

How it works

Identify the system factors and choose meaningful values or partitions for each. Include boundary and unusual values where relevant rather than relying only on typical use cases. Specify constraints that make some combinations invalid, and define which feasible pairs the suite must cover.

Use a covering-array method or suitable test generator to construct complete configurations covering those pairs. Verify the coverage against the input model. Execute each configuration with an appropriate test procedure and an oracle that determines whether the observed behavior is acceptable.

Pairwise testing is strength-two combinatorial testing. Higher-strength testing covers combinations involving more factors and may be appropriate where risks or evidence suggest higher-order interactions. Input modeling remains essential at every strength: covering all pairs of a poorly chosen set of values does not cover the real system's behavior. Stateful workflows, timing, nondeterminism and failure detection may require additional techniques beyond configuration coverage.

Three practical examples

1. A web team covers combinations of browser, language and account type to test interface behavior across supported configurations.

2. An integration team checks pairs of authentication modes, message formats and deployment settings while excluding combinations the system explicitly does not support.

3. An AI application team combines pairwise configuration tests with task-based evaluations, recognizing that covering settings does not measure response quality or safety by itself.

Think of it this way

Think of arranging small group meetings so that every pair of people meets at least once. Each meeting can satisfy several pair requirements at the same time. Meeting every pair does not mean that every possible larger group has met; similarly, pairwise configuration coverage does not imply coverage of every three-way interaction.

Worked example

Consider three binary factors A, B and C, each taking values 0 or 1. Exhaustive testing requires 2 times 2 times 2 = 8 configurations. The four configurations 000, 011, 101 and 110 provide pairwise coverage.

For factors A and B, these rows give 00, 01, 10 and 11. For A and C, they give 00, 01, 11 and 10. For B and C, they give 00, 11, 01 and 10. Thus all four value combinations occur for each of the three factor pairs: 12 pair requirements in total.

Each test covers one requirement for each pair, so four tests cover the 12 requirements in this example. Four is also a lower bound because the A-B pair alone has four combinations and one test can cover only one of them. However, a defect triggered only by A = 1, B = 1 and C = 1 will not be exercised, because 111 is absent. Pairwise coverage is complete while three-way coverage is not.

Illustrative case study

Fictional case study: Configlane supports several application settings and cannot afford to test every full combination on each release. Its engineers define factors, supported values and compatibility constraints, then generate a pairwise suite and verify its coverage.

After a defect involving three settings is discovered in production, they add a targeted regression test and increase interaction strength for the relevant configuration area. They retain checks for workflows and boundary values that the configuration model does not represent.

This fictional case illustrates pairwise testing as part of a broader strategy. It does not claim that a particular interaction strength will detect all defects or that reducing configuration counts necessarily reduces total testing effort by the same proportion.

Common misconceptions and mistakes

Testing every value of each factor individually is weaker than testing every pair of values. Conversely, all-pairs coverage is weaker than testing all complete configurations. A test generator's output size is not evidence of correct coverage unless the suite and constraints are checked.

Another mistake is assuming that a covered pair guarantees detection of a defect associated with it. Other settings can mask the behavior, the execution path may not reach the fault, or the test oracle may fail to recognize it. Coverage describes what is exercised under the input model, not the quality of every observation or assertion.

Business use

All-pairs testing helps teams manage large configuration spaces with a clear, auditable coverage target. It is especially useful for identifying interactions among settings that might be missed by a handful of manually selected configurations.

Document the factor model, value-selection rationale, constraints and achieved coverage. Combine the suite with risk-based tests, known-defect regressions and appropriate higher-strength or sequence tests. Track execution outcomes rather than equating generated configurations with completed tests. When product settings change, regenerate or augment coverage and recheck constraints. For AI systems, include meaningful assertions and evaluation criteria, because merely running each configuration does not establish acceptable model behavior.

Check your understanding

A suite covers every pair of values for three binary factors but omits configuration 111. Can it guarantee detection of a fault that occurs only at 111?

Answer: No. That fault requires a specific three-way combination, which pairwise coverage alone does not guarantee.

Related terms

combinatorial testing
covering array
interaction strength
exhaustive testing
test oracle
input partitioning
constraints and regression testing

Learn next

verify pair coverage in a small suite by listing every factor pair and its value combinations. Then introduce a three-factor fault or an invalid combination and examine how the suite and coverage requirements must change.

Sources

Primary sources: NIST, Practical Combinatorial Testing, SP 800-142: https://csrc.nist.gov/pubs/sp/800/142/final

NIST, Combinatorial methods, interactions involved in software failures: https://csrc.nist.gov/projects/automated-combinatorial-testing-for-software/combinatorial-methods-in-testing/interactions-involved-in-software-failures

NIST, Combinatorial testing dos and don'ts: https://csrc.nist.gov/Projects/automated-combinatorial-testing-for-software/software-testing-methodology/dos-and-don-ts-of-testing

Previous: All nearest smaller values Next: All-to-all (parallel pattern)