Does out of control mean a defective product?

No; it refers to process stability, while defects depend on requirements.

Can an improvement trigger a signal?

Yes, because improvement can shift the process away from its old baseline.

Are three-sigma limits always appropriate?

No. Chart choice and calibration depend on the measure, distribution, dependence, and monitoring objectives.

AI Glossary / Statistics and probability / Intermediate

Out of control

Definition

Out of control describes a process whose observed behavior signals a departure from the stable variation expected under a statistical control model. A signal may be a point beyond a control limit or a specified nonrandom pattern. It prompts investigation of a possible special cause; it is not absolute proof of a particular fault.

What it means

A stable process still varies. Statistical process control distinguishes that routine variation from evidence that the process has changed. Out of control therefore means inconsistent with the monitoring baseline and decision rules, rather than simply undesirable or expensive.
A stable process can consistently produce unacceptable results. Conversely, a newly improved process can trigger an out-of-control signal because it differs from its old baseline. Statistical stability and meeting business requirements are separate questions.

How it works

Define a meaningful process measure and choose a chart suited to its distribution and sampling structure. Individual measurements, subgroup averages, proportions, and counts require different treatment. Establish a credible baseline and document how its parameters were estimated.
Plot observations or statistics in time order against control limits. For a simple three-sigma chart, the limits are three standard deviations of the plotted statistic above and below its baseline mean. The standard deviation of a subgroup average differs from that of individual observations.
Use predetermined signal rules. A single extreme point is one rule; sustained runs on one side of the center line or other patterns can provide additional evidence. Adding rules increases sensitivity but also changes false-alarm behavior.
Investigate the process, measurement system, and data pipeline when a signal occurs. Record equipment changes, staffing, input composition, software releases, and other plausible explanations. An AI monitoring metric may shift because the incoming population changed, even when the model itself did not.
Decide on action using the operational consequences and the evidence. Correct a verified special cause where possible, preserve the investigation record, and establish a new baseline only when justified. Continually widening limits to absorb unfavorable results defeats the monitoring purpose.
Check assumptions such as independence, stable sampling, and appropriate distributional modeling. Serial correlation or estimated limits can materially change nominal false-alarm probabilities. A chart's textbook properties need not apply exactly to a particular deployment.

Three practical examples

Manufacturing: a subgroup mean moves beyond its control limit, prompting inspection of a recently adjusted machine.
AI operations: a monitored error proportion changes after a document-source update, triggering review of inputs, labeling, and model behavior.
Customer support: repeated daily measures lie above the established center line, leading the team to investigate a possible sustained shift in workload or workflow.

Think of it this way

Think of a smoke alarm calibrated to detect unusual smoke levels. An alarm calls for investigation; it does not identify the cause by itself. A silent alarm also does not establish that every aspect of the building is satisfactory.

Worked example

Suppose a fictional stable process has a known mean of 100 units and known individual standard deviation of 4 units. Each plotted point averages 16 independent, normally distributed observations collected under comparable conditions.
The subgroup mean has standard deviation 4/sqrt(16) = 1 unit. A three-sigma mean chart therefore has lower limit 97 and upper limit 103. A new subgroup mean of 103.6 exceeds the upper limit and triggers the selected rule.
Using the individual standard deviation would instead produce limits of 88 and 112, which would be inappropriate for this mean chart under the stated assumptions. The example highlights the need to match limits to the plotted statistic.
Even with no process change, the probability that a normal standardized mean exceeds either three-sigma limit is about 0.002700 per subgroup. For 100 independent subgroup means with fixed known limits, the probability of at least one such signal is 1 - (1 - 0.002699796)^100, approximately 0.2369, or 23.69%.
This calculation does not mean that 23.69% of observed alarms are false. It is a repeated-monitoring probability conditional on the process remaining in control and the idealized assumptions holding.

Illustrative case study

Illustrative fictional case: Willow Services monitors the time required to validate AI-generated summaries. After a workflow release, a chart signals an unusually low average time.
The team checks whether reviewers are working more efficiently or skipping required checks. It reviews sampled outputs and discovers that a routing change altered the mix of easy and difficult cases. Rather than claiming immediate productivity improvement, it adjusts the monitoring analysis for the changed case mix and documents the transition. The case is fictional and illustrates investigation of a favorable-looking signal.

Common misconceptions and mistakes

Do not confuse control limits with specification limits. Do not conclude that an in-control process is automatically capable of meeting requirements. A signal does not identify its cause, and a point inside the limits does not rule out a meaningful pattern.
Avoid applying normal independent-data false-alarm calculations to correlated or highly skewed data without checking suitability. Do not silently change signal rules after seeing the chart or repeatedly reset limits to hide shifts.

Business use

Businesses use out-of-control signals to prioritize investigation and distinguish routine fluctuation from possible change. A useful operating procedure names the metric owner, investigation steps, escalation criteria, and rules for revising the baseline. The purpose is informed process management, including recognition of sustained improvements, rather than automatic blame for every unusual observation.

Check your understanding

Why is the example's standard deviation 1 rather than 4?

Answer: The chart plots averages of 16 independent observations.

Does the 23.69% repeated-monitoring probability prove that a particular signal is false?

Answer: No; it describes at least one alarm under an unchanged idealized process.

Related terms

Learn next

Next study subgroup selection and the difference between monitoring averages and variability. Then examine average run length, false-alarm tradeoffs, and methods suitable for dependent operational data.

Sources

NIST/SEMATECH, What are Control Charts?: https://www.itl.nist.gov/div898/handbook/pmc/section3/pmc31.htm — signals and nonrandom patterns. NIST/SEMATECH, What are Variables Control Charts?: https://www.itl.nist.gov/div898/handbook/pmc/section3/pmc32.htm — limit construction, specifications, and additional signal rules. The original numerical illustration states known-parameter and independence assumptions explicitly.

Previous: Out-of-bag evaluation Next: Out-of-core