Can a criterion be categorical?

Yes. It need not be a continuous number.

Can a study have several criterion variables?

Yes; multi-output tasks require clear definitions and suitable evaluation for each outcome.

Is the criterion always perfectly measured ground truth?

No. It is the chosen observed or operationalized outcome, whose reliability and validity require examination.

AI Glossary / Statistics and probability / Intermediate

Criterion variable

Definition

A criterion variable is the variable a statistical model or prediction study aims to predict. It is also often called the outcome, response, dependent variable, or target. Clearly defining it determines what success means for the model and what the resulting predictions can legitimately be interpreted as describing.

What it means

In a model predicting how long a customer request takes to resolve, resolution time is the criterion variable. Request type, arrival time, and workload may be predictors. The observed resolution time is the target value; the model's forecast is an estimate of that value or its distribution.

The word criterion here refers to the predicted variable, not to a decision threshold or a model-selection rule such as an information criterion. It also differs from an evaluation metric: mean absolute error summarizes prediction performance but is not the resolution-time outcome itself.

Changing the target changes the task. Predicting whether a request is closed within one day is different from predicting its exact resolution time, even when both targets come from the same records.

How it works

Define the unit being predicted, the measurement procedure, units or categories, and the prediction horizon. Specify when the prediction is made and when the criterion becomes observable. Decide how missing, disputed, or censored outcomes will be represented.

Collect paired predictor and criterion records. Fit the model using training data, and assess predictions against criterion values in an appropriate held-out evaluation set. Predictors must contain only information available at the intended prediction time.

Choose the model and evaluation methods to suit the criterion. A continuous quantity, a binary event, an ordered rating, and a censored duration need different treatment. A reliable prediction of a poorly chosen proxy may still fail to answer the underlying business question.

Three practical examples

A demand model predicts units ordered next week. Next week's order count is the criterion, while past orders and known planned promotions are candidate predictors.
An AI document classifier predicts the document category assigned under a defined labeling policy. The category is the criterion; classification accuracy is an evaluation measure.
An educational model predicts a later assessment score. Researchers check whether that score adequately represents the learning outcome they actually care about.

Think of it this way

Think of the criterion variable as the destination written on a navigation request. Predictors provide information for finding a route, and evaluation measures tell you how well the trip went. Reaching the wrong destination efficiently is still a failure of the original purpose.

Worked example

Consider an original example predicting service-request resolution time in hours. Three held-out requests have observed criterion values y = [10,20,30]. A model predicts [12,18,33].

The prediction errors, predicted minus observed, are [2,-2,3] hours. Mean absolute error is (2 + 2 + 3)/3 = 7/3, approximately 2.33 hours. Mean squared error is (4 + 4 + 9)/3 = 17/3, approximately 5.67 square hours.

The criterion remains resolution time; neither error metric replaces it. Predictions remain distinct from observed outcomes.

Suppose the business instead defines success as resolution within 24 hours. The observed binary target becomes [1,1,0]. Applying the same threshold to these three time predictions also yields [1,1,0], giving three correct classifications in this tiny illustration.

That perfect threshold classification does not mean the time predictions are exact. It shows that different target definitions and evaluation questions can produce different performance summaries. The sample is far too small to support a deployment claim, and a model trained directly for the binary event need not behave like a thresholded time predictor.

Illustrative case study

Fictional case: Riverdesk trains an AI tool to predict whether support tickets will be marked closed quickly. Its apparent performance is excellent because one input field records the closing team's final disposition, which is entered only after resolution.

The analysts remove that unavailable predictor and rebuild the evaluation around the actual prediction time. They also discover that closed does not always mean the customer problem is solved. Some tickets are closed administratively and later reopened.

Riverdesk revises its outcome definition and labeling rules. The criterion now matches the operational question and the evaluation uses information available at the right time.

Common misconceptions and mistakes

The criterion is not the same as the score used to evaluate predictions.
A dependent-variable label does not establish that predictors cause the outcome.
Observed target labels can contain measurement error, inconsistent judgments, or systematic bias.
An outcome measured later can be a valid target, but information obtained only later is generally invalid as a baseline predictor.
A convenient proxy can differ from the intended construct, such as clicks versus satisfaction.
Redefining the criterion after seeing results changes the question and should be documented rather than presented as the original analysis.

Business use

Criterion definition is an important early decision in an AI project. Record what one row represents, the horizon, eligibility conditions, measurement source, and treatment of exceptions. Give labeling teams examples of ambiguous cases and a process for resolving them.

Check whether the outcome can actually be observed for the people or items the model will serve. Selective observation can produce a misleading training target: outcomes may be recorded only for cases that passed an earlier selection process.

When the model changes behavior, consider whether the recorded criterion changes in response. A system trained to reproduce past administrative decisions may reproduce the old decision process rather than estimate an independent measure of quality. Evaluate the target's relevance throughout deployment, not only at initial training.

Check your understanding

Answer: 1.

What is the criterion in the worked example?

Answer: Resolution time in hours, before the alternative binary definition is introduced. 2.

Why is mean absolute error different?

Answer: It summarizes discrepancies between predictions and criterion observations. 3.

What was wrong with Riverdesk's final-disposition input?

Answer: It was unavailable when the intended prediction had to be made.

Related terms

Dependent variable
Data leakage
Target variable
predictor
outcome measurement
label quality
proxy variable

Learn next

Learn the differences between predictors, targets, predictions, losses, and evaluation metrics. Then study measurement validity and time-aware data preparation. Practice writing a target definition that another team could apply consistently without guessing the intended outcome.

Sources

David Howell, Statistical Methods for Psychology glossary, Criterion variable: https://www.uvm.edu/~statdhtx/methods8/Glossary/Glossary.html

Scikit-learn developers, Common pitfalls and recommended practices, data leakage: https://scikit-learn.org/stable/common_pitfalls.html

The author definition and official evaluation guidance were consulted. The resolution-time arithmetic and fictional support case are original.

Previous: Criterion validity Next: Critic