Standard deviation in machine learning

The transformation

Standardisation, also called z-score scaling, replaces each value with the number of standard deviations it sits from its feature's mean:

z = (x − μ) / σ

After the transformation every feature has a mean of 0 and a standard deviation of 1. It is the same calculation as a z-score for a single exam result, applied column by column to a whole table. The shape of each feature does not change; only its location and units do.

Why models care about units

Many algorithms compare rows by distance or add features together with weights. When one feature is measured in tens and another in tens of thousands, the large one takes over. Take three hypothetical customers described by age and annual income in dollars:

On the raw numbers, the Euclidean distance from X to Y is √(35² + 1000²) = 1,000.6 and from X to Z is √(1² + 4000²) = 4,000.0. A nearest-neighbour model would call Y, who is 35 years older, X's closest match. The age gap contributes 35² = 1,225 to a squared distance of about a million; it is invisible.

Standardise both features (using the training statistics worked out in the next section: age SD 12.76 years, income SD $20,823) and the picture flips. The age gap of 35 years becomes 2.74 standard deviations, while the $4,000 income gap becomes 0.19. The scaled distances are 2.74 from X to Y and 0.21 from X to Z, so Z is now the near neighbour, which matches common sense. The choice of dollars over thousands of dollars no longer decides the answer.

The same problem affects k-means clustering, support vector machines with RBF kernels, principal component analysis (where a large-variance feature dominates the first component) and anything trained by gradient descent, where features on very different scales make the loss surface long and narrow and training slow. Tree-based models such as random forests and gradient-boosted trees split on one feature at a time and are unaffected by scaling.

A worked feature-scaling table

Here is a small training set of five rows, with income now in thousands of dollars. The scaler learns a mean and a population standard deviation for each column:

Age: μ = 43, σ = 12.76 Income: μ = 61, σ = 20.82 (thousands of $)

Row one, for example: (25 − 43) ÷ 12.76 = −1.41 and (30 − 61) ÷ 20.82 = −1.49. Both columns now run from about −1.5 to +1.4, and a model can weigh them against each other on equal terms. You can check the column statistics by pasting either column into the population standard deviation calculator; the sample SD would be 14.27 and 23.28 instead, the ddof difference covered in the FAQ below.

Try it: population standard deviation calculator

The calculator below holds the training set's age column in population mode, the ddof = 0 version a scaler uses; swap in the income column (30, 48, 75, 62, 90) or a feature of your own.

Open the full population standard deviation calculator for the step-by-step working and a chart of the values.

Fit on the training data only

The mean and SD are learned parameters, just like a model's weights, and they must come from the training data alone. The common mistake is to scale the whole dataset first and split it into training and test sets afterwards. The test rows then influence the mean and SD that the training rows are scaled with, so the model has seen a summary of the data it is about to be graded on. This is called data leakage, and it makes test scores look better than performance on genuinely new data will be.

In the example, suppose a new customer aged 40 with an income of $55k arrives. With the training statistics, they score (40 − 43) ÷ 12.76 = −0.24 and (55 − 61) ÷ 20.82 = −0.29. If that row had been included when fitting the scaler, the age statistics would have shifted to a mean of 42.5 and an SD of 11.70, and every training row would have been scaled slightly differently because of a row that was supposed to be unseen. On five rows the effect is obvious; on real data it is small and silent, which is why it survives into production.

The safe order:

1. Split the data into training, validation and test sets.
2. Compute each feature's mean and SD on the training set only.
3. Apply those stored numbers to the training, validation and test sets, and later to live data.
4. In cross-validation, refit the scaler inside each fold. A pipeline object that bundles the scaler with the model does this for you.

Save the scaler with the model. A model trained on standardised inputs and then fed raw numbers at prediction time will produce confident nonsense.

When the standard deviation is a poor scale

Because the SD squares each deviation, a single extreme value can inflate it and squash all the ordinary values towards zero. For a feature with heavy outliers, a scaler based on the median and interquartile range keeps the bulk of the data spread out; scikit-learn calls this RobustScaler. Our post on how outliers affect the standard deviation shows the size of the effect with numbers. Min–max scaling to [0, 1] is the other common alternative, and it is even more sensitive to extremes, since the single largest and smallest values set the whole scale.

Batch normalisation in one paragraph

Deep networks apply the same idea to their own internal values. A batch normalisation layer, introduced by Ioffe and Szegedy in 2015, takes the activations of a layer for one mini-batch, subtracts the batch mean and divides by the batch standard deviation (with a small constant added to avoid dividing by zero), then multiplies by a learned scale γ and adds a learned shift β, so the network can undo the standardisation if that helps. During training the statistics come from each batch; at inference time the layer uses running averages collected during training, so a prediction does not depend on which other examples happen to be in the batch. Layer normalisation, used in transformers, standardises across the features of a single example instead of across the batch.