AI Glossary / Statistics and probability / Foundation
Standardization
Definition
Standardization, in the statistical preprocessing sense used here, transforms a variable by subtracting a chosen mean and dividing by a chosen positive standard deviation. It expresses values in standard-deviation units and is also commonly spelled standardization. The transformation is often used to put features on comparable numerical scales before modeling. It does not make an arbitrary distribution normal, remove correlations between variables or guarantee improved model performance.
What it means
Features can have very different units and numerical ranges. A quantity measured in thousands can dominate a distance calculation or penalty even when that dominance has no substantive justification. Standardization changes the coordinate scale so the fitted mean and spread have a common convention. This can help some algorithms behave as intended. It also changes how coefficients and distances are interpreted, so the transformation and its reference data should remain part of the documented model.
How it works
Within the training data, calculate a mean and standard deviation for each feature using a stated convention. Store those values and transform observations with z = (x minus mean)/standard deviation. Apply the stored training transformation to validation, test and later deployment data rather than fitting a new transformation to each dataset. In cross-validation, fit preprocessing within each training fold. Handle constant features, missing values and sparse storage explicitly. Different software may use different variance divisors or options, so check the implementation rather than assuming all standardized outputs are identical.
Three practical examples
Example 1: A distance-based classifier uses standardized continuous features to avoid arbitrary domination by measurement units. Example 2: A penalized regression uses a consistent feature scale before tuning its penalty. Example 3: An optimization routine benefits from more comparable coordinate magnitudes. These examples do not imply that every model needs the same preprocessing. Some tree-based methods are largely insensitive to positive linear rescaling of individual features, while other methods depend strongly on it.
Think of it this way
Think of converting several rulers to a common unit before comparing distances. Standardization uses each feature's reference spread as that unit and moves its reference center to zero. The conversion makes the arithmetic more comparable but does not make the measured concepts equally important. It also does not change a skewed pattern into a symmetric one simply by relabeling the ruler.
Worked example
Take a training feature with values two, four and six. Its mean is four. Using the population-style divisor n, its variance is (4 + 0 + 4)/3 = 8/3 and its standard deviation is approximately 1.633. The transformed training values are approximately -1.2247, zero and 1.2247, with mean zero and variance one under that same divisor. A later value of eight is transformed using the stored training quantities, giving (8 minus 4)/1.633, approximately 2.4495. It should not be rescaled using a mean newly calculated from the test set. If the sample divisor n minus one had instead been chosen, the training standard deviation would be two and eight would transform to two. Both conventions can be defined consistently, but they should not be mixed silently.
Illustrative case study
In this fictional case study, Hazel Analytics compares AI models on a held-out dataset. A review finds that its preprocessing mean was calculated from all observations, including the test data. Engineers move the scaler into the training workflow, repeat validation with preprocessing fitted separately inside each fold and reserve the final test set for evaluation. They save the final training scaler with the model. The resulting report explains the transformation and assesses performance after the leakage has been removed.
Common misconceptions and mistakes
Common mistakes include fitting the scaler on the full dataset before evaluation, re-fitting it independently on each new batch, and assuming standardization is equivalent to normalization in every context. The word normalization can refer to several other operations, including unit-vector scaling and min-max scaling. Standardization is sensitive to outliers because means and standard deviations are sensitive. Centering a sparse matrix can destroy its sparsity and create substantial storage costs, so implementation options need attention.
Business use
Businesses should treat preprocessing as part of the deployed model rather than an informal spreadsheet adjustment. Store feature order, units, missing-value handling and fitted scaling values, and ensure that production uses the same procedure. A shift in new data can produce transformed values with a nonzero mean or variance different from one; this is not necessarily a coding error. It may indicate distribution change. Decide whether retraining is appropriate through the monitoring and evaluation process rather than automatically resetting the scale and hiding the shift.
People also ask
Does standardization make data Gaussian?
No.
Does it remove correlations?
No; ordinary feature-wise positive scaling preserves pairwise correlations.
Must test data have mean zero after transformation?
No; training parameters define the reference.
What happens to a constant feature?
The ordinary division is undefined, so software needs an explicit convention.
Is standardization always beneficial?
No; its value depends on the model, features and task.
Check your understanding
What training mean is used in the example?
Answer: Four.
What population-style training variance is used?
Answer: 8/3.
What transformed value does a new observation of eight receive?
Answer: Approximately 2.4495.
Why store the training parameters?
Answer: New observations must be interpreted on the same reference scale.
Does a new batch's nonzero transformed mean automatically require rescaling?
Answer: No.
Related terms
- Normalization
- Related terms include z-score
- feature scaling
- min-max scaling
- robust scaling
- whitening and data leakage
Learn next
Learn next which algorithms depend on feature scale and how scaling changes regularization and distance calculations. Compare sample and population variance conventions, then practice fitting preprocessing within cross-validation. Examine how saved preprocessing interacts with missing values, sparse data and monitoring of new observations.
More in Statistics and probability
- Standard normal distribution - The standard normal distribution is the normal, or Gaussian, probability distribution with mean zero and variance one.
- Standard normal random vector - A standard normal random vector is a vector whose components are independent standard normal random variables.
- Standard scores - Standard scores express observations on a reference scale with a specified mean and standard deviation.
- Standardized coefficient - A standardized coefficient is a regression coefficient expressed using standardized variable scales.
- Standardizing apart - Standardizing apart is the logical operation of consistently renaming variables so that variables belonging to separate clauses, scopes or rule applications...
- State observer - A state observer is a model-based system that estimates the internal state of a dynamic process from its measured outputs and known inputs.
Sources
Sources: scikit-learn developers, StandardScaler, https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html. This primary implementation reference describes training-based centring and scaling, outlier sensitivity, sparse-data constraints and variance conventions. The three-value training example is original and independently checked. The analytics case is fictional and illustrates a preprocessing evaluation error rather than measured results from a real project.
Previous: Standard test image Next: Standardized coefficient