Is this the same as classifier sensitivity?

No; classifier sensitivity usually means the true-positive rate.

Must a study use random sampling?

No; a planned grid or factorial design can be appropriate.

Yes, when outputs, ranges, distributions or other settings change.

AI Glossary / Statistics and probability / Advanced

Sensitivity study

Definition

A sensitivity study investigates how changes in a process's or model's inputs affect selected outputs. It identifies influential inputs, examines whether their effects depend on other settings, and helps establish how strongly a conclusion depends on the conditions examined.

What it means

A prediction is often conditional on choices about workload, operating conditions, parameter values or uncertain assumptions. A sensitivity study asks what changes when those inputs change. Its answer is specific to the output, input ranges and method used.

A study of a mathematical model describes that model's behavior. It does not, by itself, establish that the model accurately represents the world.

How it works

Define the question and output first. Select inputs, justify their ranges and identify combinations that are feasible. Evaluate the process or model at planned settings, then summarize changes and interactions.

A local study examines behavior near particular settings. A global study explores a broader specified input domain, often incorporating input distributions. Changing one input around a single baseline can miss interactions: one input's effect may depend on another input's value. A factorial design evaluates combinations of factor levels and can expose such dependence.

Separate numerical effects from random variation. Stochastic simulations or noisy physical measurements may require repetitions and uncertainty estimates. Record which inputs were fixed, how dependencies were handled and what the study did not examine.

Three practical examples

1. AI service planning: vary workload and request complexity in a service model to see how their combination changes predicted processing time.

2. Manufacturing: study temperature and feed settings together when assessing product quality, using a suitable experimental design and measurement procedure.

3. Business forecasting: change uncertain demand assumptions and capacity limits to identify when a forecasted service target stops being achievable. Label these conditional scenarios rather than treating them as probabilities without justification.

Think of it this way

Think of testing the controls on a mixing desk. Moving one slider may sound different depending on another slider's position. Testing only one starting configuration can leave you with an incomplete account of how the controls work together.

Worked example

Consider an invented deterministic model of processing time, in milliseconds:

y = 100 + 10a + 5b + 8ab.

Here a represents a coded workload setting and b a coded request-complexity setting. Each takes either -1 for the selected low level or +1 for the selected high level. These codes represent two chosen operational settings; they are not literal negative workloads or measured probabilities.

Evaluate all four combinations. At a = -1 and b = -1, the output is 100 - 10 - 5 + 8 = 93 milliseconds. At a = +1 and b = -1, it is 100 + 10 - 5 - 8 = 97. At a = -1 and b = +1, it is 100 - 10 + 5 - 8 = 87. At a = +1 and b = +1, it is 100 + 10 + 5 + 8 = 123.

At low complexity, raising workload changes the output by 97 - 93 = 4 milliseconds. At high complexity, the same coded workload change produces 123 - 87 = 36 milliseconds. Reporting only the first comparison would conceal a substantially larger effect at the other complexity setting.

Average the workload comparisons with equal weight: (4 + 36)/2 = 20 milliseconds. This is the two-level workload main effect for this balanced design. It is twice the coefficient 10 because the coded input moves two units, from -1 to +1.

Complexity also has different effects at the two workload settings. Its low-to-high change is 87 - 93 = -6 milliseconds at low workload, but 123 - 97 = 26 at high workload. Its equally weighted main effect is (-6 + 26)/2 = 10 milliseconds.

The workload effects differ by 36 - 4 = 32 milliseconds. That difference of differences demonstrates interaction. Under the usual two-level factorial effect convention, the interaction effect is half this difference, 16; the regression coefficient on ab is half that effect, 8.

If a planning threshold is 110 milliseconds, three evaluated combinations are below it and one is above it. That is not evidence of a 25% real-world exceedance probability: no frequencies or probabilities for operating conditions have been supplied.

All outputs follow exactly from the stipulated equation. There is no experimental noise, fitted uncertainty or evidence that a real service behaves this way. Intermediate settings and conditions outside the selected range would require further examination before drawing operational conclusions.

Illustrative case study

Fictional Alder Services initially tests workload only at low request complexity. Its analyst describes workload as having a small effect, based on the four-millisecond difference.

A reviewer requests the other two combinations. The 36-millisecond difference at high complexity changes the interpretation and directs attention to demanding requests under heavier load. Alder plans measurements under those conditions before revising capacity decisions. This invented case illustrates how a sensitivity study can improve a question and reveal an omitted comparison without validating the underlying model.

Common misconceptions and mistakes

"A small effect at one baseline means the input is unimportant." Effects can change across settings.

"The largest coefficient always identifies the most influential input." Units, ranges, coding and interactions matter.

"A sensitivity study proves the forecast is correct." It examines dependence within the chosen process or model and assumptions.

"Four scenarios give four equally likely outcomes." Scenario selection does not establish occurrence probabilities.

Business use

Report the decision-relevant output, tested conditions and effect sizes in meaningful units. In the worked example, a single average obscures the combination that exceeds the planning threshold.

For AI-assisted reports, inspect whether the claimed input ranking comes from comparable changes and whether combined settings were evaluated. Preserve the distinction between model exploration and observed operational evidence. Use the findings to prioritize measurement, further experiments or model revision, then assess the consequences of any proposed action separately.

Check your understanding

Why is the workload main effect 20 rather than 10?

Answer: The coded change spans two units.

What exposes interaction?

Answer: Workload changes the output by 4 at one complexity level and 36 at the other.

Why is one threshold exceedance not a 25% probability?

Answer: The four settings have no assigned occurrence probabilities.

Related terms

Sensitivity analysis
Factorial design
Robustness
interaction effect
response surface
uncertainty analysis
simulation
input parameter

Learn next

Remove the 8ab term and repeat the four evaluations. Compare the conditional effects and explain which interaction evidence disappears.

Sources

NIST/SEMATECH e-Handbook, glossary, sensitivity study: https://www.itl.nist.gov/div898/handbook/glossary.htm

NIST/SEMATECH e-Handbook, Two-level full factorial designs: https://www.itl.nist.gov/div898/handbook/pri/section3/pri3331.htm

European Commission Joint Research Centre, Sensitivity Analysis: Methods: https://joint-research-centre.ec.europa.eu/sensitivity-analysis-samo/methods_en

Previous: Sensitivity analysis Next: Sensor interface layer