AI Glossary / Causal inference and experimentation / Beginner
z-score normalization
Definition
Z-score normalization is a transformation that subtracts a reference mean from each numerical value and divides by a reference standard deviation. Also called standardization, it expresses values in standard-deviation units. In machine-learning pipelines, the reference statistics are usually learned from the training data.
What it means
Features can have very different numerical scales. A distance measured in meters may contain values in the thousands, while a ratio lies between zero and one. Some learning methods respond to those scales, so the larger numbers can dominate distances or affect optimization and regularization.
Standardization provides a consistent location-and-scale transformation. For a nonconstant feature, the data used to fit the transformation have mean zero and unit variance under the matching calculation convention, apart from numerical precision.
This does not force every future batch to have those statistics. Nor does it turn a skewed distribution into a normal distribution, remove outliers or make all features equally useful. The term normalization has other meanings, including scaling a vector to unit length; identify the actual operation.
How it works
For each selected numerical feature, calculate a training mean and standard deviation. Store these values with the model. Transform a value x using z = (x - mean) / standard deviation, applying the same stored statistics to validation, test and later prediction inputs.
During cross-validation, fit the scaler separately within each training fold. A pipeline helps keep preprocessing attached to the fitting procedure. Calculating statistics on the entire dataset before splitting can allow information from the assessment portion to affect development.
Handle constant columns explicitly because their standard deviation is zero. Libraries may choose a safe convention, such as using a scale factor of one; that convention does not make the feature's variance one. Missing values and weighted observations also require a documented policy.
Mean-and-standard-deviation scaling is sensitive to extreme values. Alternative transformations may be useful when the data contain substantial outliers or skewness. Centering a sparse matrix can destroy its sparse representation, so implementations sometimes omit centering to preserve practical memory use.
Three practical examples
- A nearest-neighbor system standardizes chosen numerical inputs before calculating distances so original measurement units do not silently set the relative scale.
- A regularized linear model uses consistent feature scaling when the penalty would otherwise interact strongly with the inputs' units.
- A neural-network training pipeline stores its fitted scaler so production inputs receive the same transformation used during development.
Think of it this way
Think of converting several measurements to rulers whose unit is each measurement's reference spread and whose zero is its reference mean. The resulting numbers describe comparable kinds of relative deviation.
The rulers do not decide which measurement deserves more importance. Standardization is a numerical preparation step, not a guarantee that a distance formula or model now represents the business problem correctly.
Worked example
Fit once, apply consistently
An invented training feature contains values 2, 4, 6 and 8. Its mean is 5. Using the variance convention that divides by four, its variance is (9 + 1 + 1 + 9) / 4 = 5, and its standard deviation is approximately 2.236.
The transformed training values are approximately -1.342, -0.447, 0.447 and 1.342. A later value of 10 becomes (10 - 5) / 2.236, approximately 2.236.
Do not recompute the mean and spread using that one later value. Its standalone standard deviation would be zero, and a new scale would no longer match the trained model's input meaning. If using a different standard-deviation convention, the numerical results change, so the convention belongs with the saved transformation.
Illustrative case study
A fictional equipment-monitoring project, Birch Sensors, trains a model using standardized vibration and temperature inputs. During deployment, a separate service normalizes each incoming batch using that batch's own mean and spread.
A sustained rise in temperature partly disappears from the transformed values because the new batch recenters itself. Predictions no longer use the coordinate system on which the model was trained. The team restores the saved training transformation and separately monitors whether incoming measurements have shifted.
It also checks sensor units and feature order. Correct standardization cannot repair a temperature supplied in the wrong unit or a swapped input column. The fictional case shows why preprocessing details are part of the deployed model's behavior.
Common misconceptions and mistakes
Standardization does not require a bell-shaped distribution, and applying it does not create one. It changes location and scale while preserving the shape of a single feature's distribution under the linear transformation.
It is not min-max scaling to a fixed interval. Standard scores can be negative or exceed one by a large amount.
Avoid automatically scaling every column without considering its meaning. Encoded categories, sparse indicators and tree-based methods may call for different treatment. Also avoid fitting a new scaler independently on test data merely to make its transformed mean zero; that changes the assessment protocol.
Business use
Treat the scaler as a versioned part of the prediction pipeline. Store feature names, order, units, fitted statistics and policies for missing or constant inputs with the model.
Validate whether scaling helps the selected learning method on relevant data. If the operating population changes, investigate the effect instead of silently updating the reference statistics. Updating the scaler changes what the model sees and may require a coordinated model update and renewed evaluation.
People also ask
Will transformed test data have mean zero?
Not necessarily. They use the training reference, and differences can reflect ordinary sampling or a distribution shift.
Is Z-score normalization the same as making every row length one?
No. Feature standardization usually operates column by column, while unit-length normalization operates on each vector's norm.
Can it remove outliers?
No. Extreme observations remain extreme relative to the transformed scale and can influence the fitted mean and spread.
Check your understanding
A model was trained with mean 100 and standard deviation 20 for an input. How should a new value of 130 be transformed?
Answer: (130 - 100) / 20 = 1.5, using the stored reference. Do not fit new statistics to the new observation.
Related terms
- Z-score
- Mean
- Standard deviation
- Normalization
- Data leakage
- feature scaling
- min-max scaling
- robust scaling
- preprocessing pipeline
Learn next
Standardize a small column by hand and check the result using the same variance convention. Next apply the saved statistics to a separate sample. Compare standardization with min-max and unit-length normalization, focusing on which dimension each operation changes.
More in Causal inference and experimentation
- Validation error - Validation error is the average loss a trained model incurs on a validation dataset that is separate from the examples used to fit it.
- Validation loss - Validation loss is the value of a specified loss function measured on validation data for a model checkpoint or candidate.
- Validation set - A validation set is a collection of examples used to assess candidate models during development, separately from the examples used to fit each candidate.
- Variable importances - Variable importances are scores that describe how input variables matter to a fitted model according to a particular assessment method.
- Verification and validation - Verification and validation, often abbreviated V&V, are complementary activities for assessing a system.
- Wasserstein loss - Wasserstein loss is an objective based on a Wasserstein distance between probability distributions.
Sources
Sources: Scikit-learn, StandardScaler: https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html SciPy, Z-score calculation conventions: https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.zscore.html Scikit-learn, preprocessing leakage and pipelines: https://scikit-learn.org/stable/common_pitfalls.html Google Machine Learning Glossary, z-score normalization: https://developers.google.com/machine-learning/glossary#z-score-normalization