Scaling Features Before Clustering

Why Scale Features Before Clustering?

Distance-based clustering treats each input feature as a contribution to the distance between observations. If one feature is recorded in dollars and another in years, the dollar values may be numerically much larger, so they can dominate Euclidean distance even when the underlying differences are not more important. A change from dollars to thousands of dollars could then change the clustering result without changing the information in the data.

Scaling is a way to make the distance calculation reflect a deliberate comparison across features rather than their arbitrary measurement units. For hierarchical clustering, scaling is normally performed before the pairwise dissimilarities are calculated. The distance metric still matters; scaling changes the coordinates on which that metric operates.

Definition: For training observations $i=1,\ldots,n$ and numeric feature $j$, let $\bar{x}_j$ be the training-set mean and $s_j$ the training-set sample standard deviation. If $s_j>0$, the standardized feature value is $$ z_{ij}=\frac{x_{ij}-\bar{x}_j}{s_j}. $$ The resulting feature has training-set mean zero and sample standard deviation one. Standardized Euclidean distance between observations $a$ and $b$ is $$ d(a,b)=\sqrt{\sum_{j=1}^{p}(z_{aj}-z_{bj})^2} =\sqrt{\sum_{j=1}^{p}\left(\frac{x_{aj}-x_{bj}}{s_j}\right)^2}. $$

Here $p$ is the number of numeric features included in the distance calculation. The training-set mean centers a feature, while its standard deviation sets the unit of comparison. Centering alone does not change pairwise Euclidean distances: subtracting the same mean from both observations leaves their difference unchanged. Dividing by the standard deviation does change the feature's contribution.

What Standardization Does to Distance

The formula shows that a difference of $s_j$ original units in feature $j$ contributes one squared unit to the standardized squared distance. Equivalently, standardization gives a feature with a small standard deviation a larger distance contribution for a fixed original-unit difference, and a feature with a large standard deviation a smaller contribution. This is not a neutral technical adjustment: it is a choice to compare differences relative to each feature's observed training spread.

The justification follows directly by subtracting standardized values:

z_{aj}-z_{bj} =\frac{x_{aj}-\bar{x}_j}{s_j} -\frac{x_{bj}-\bar{x}_j}{s_j} =\frac{x_{aj}-x_{bj}}{s_j}.

The common mean cancels, and the original difference is divided by the feature's standard deviation. Squaring these standardized differences and adding across features gives the standardized Euclidean distance formula. Thus, a change in measurement units alone need not change standardized distances: if every value of a feature is multiplied by a positive conversion factor and its mean and standard deviation are recomputed in those units, its standardized values remain the same.

Worked Example: A Nearest Neighbor Changes After Scaling

A hypothetical insurer compares four claim profiles using two numeric features. The training observations are $A=(0,0)$, $B=(0,6)$, $C=(8,0)$, and $D=(8,0)$. Determine whether $B$ or $C$ is closer to $A$ using raw Euclidean distance and then using features standardized with the four-observation training means and sample standard deviations.

Method. Compute the raw distances from $A$, then find each feature's training mean and sample standard deviation and recompute the distances in standardized units.

Raw distances. From $A$ to $B$, the feature differences are $0$ and $6$: $$ d(A,B)=\sqrt{0^2+6^2}=6. $$ From $A$ to $C$, they are $8$ and $0$: $$ d(A,C)=\sqrt{8^2+0^2}=8. $$ Without scaling, $B$ is closer to $A$.

Training statistics. The first-feature values are $0,0,8,8$, with mean $4$. Their squared deviations sum to $4^2+4^2+4^2+4^2=64$, so the sample variance is $64/(4-1)=64/3$ and $s_1=8/\sqrt{3}$. The second-feature values are $0,6,0,0$, with mean $1.5$. Their squared deviations sum to $1.5^2+4.5^2+1.5^2+1.5^2=27$, so the sample variance is $27/3=9$ and $s_2=3$.

Standardized distances from $A$. The first feature differs by $8$ between $A$ and $C$, while the second differs by $6$ between $A$ and $B$. Therefore, $$ d(A,B)=\sqrt{(0/s_1)^2+(6/3)^2}=2, $$ $$ d(A,C)=\sqrt{(8/(8/\sqrt{3}))^2+(0/s_2)^2}=\sqrt{3}\approx1.7321. $$ Now $C$ is closer to $A$. The nearest of these two candidates has changed because the second feature's original-unit difference of $6$ is two training standard deviations, while the first feature's difference of $8$ is only $\sqrt{3}$ training standard deviations. The duplicate observation $D$ has the same coordinates as $C$; it does not change the two comparisons from $A$.

Check. The computed standardized distances are positive, and $\sqrt{3}<2$, so the standardized comparison does select $C$. Common mistake avoided: comparing raw values alone and assuming the feature with the larger numerical differences should carry more weight.

Use Training Statistics for Every Data Split

In a predictive workflow, calculate $\bar{x}_j$ and $s_j$ using the training data only. Transform each training observation using those values, then use those same values to transform validation, test, or future observations. Recomputing the mean or standard deviation separately for another data split changes the coordinate system and allows information from that split to influence preprocessing.

Choose the numeric features.
Decide which measured variables should contribute to the distance. Standardization does not determine whether a feature belongs in the analysis.

Fit the scaling values on training data.
For each feature, calculate its training mean and sample standard deviation. Record both values.

Transform all observations consistently.
Subtract the recorded training mean and divide by the recorded training standard deviation for that feature.

Calculate distances.
Use the transformed values to construct the distances or dissimilarities required for clustering.

Worked Example: Scaling a New Profile With Training Values

A hypothetical portfolio has four training profiles with feature pairs $(10,2)$, $(10,4)$, $(20,2)$, and $(20,4)$. A new profile has values $(25,5)$. Standardize the new profile using the training statistics, then find its standardized Euclidean distance to training profile $(20,4)$.

Method. Compute the sample mean and standard deviation of each feature from the four training profiles. Apply those fixed values to both the new profile and the selected training profile.

First feature. Its training mean is $(10+10+20+20)/4=15$. The squared deviations sum to $25+25+25+25=100$, so the sample variance is $100/3$ and $s_1=10/\sqrt{3}$.

Second feature. Its training mean is $(2+4+2+4)/4=3$. The squared deviations sum to $1+1+1+1=4$, so the sample variance is $4/3$ and $s_2=2/\sqrt{3}$.

The new profile standardizes to $$ z_{\text{new},1}=\frac{25-15}{10/\sqrt{3}}=\sqrt{3}, \qquad z_{\text{new},2}=\frac{5-3}{2/\sqrt{3}}=\sqrt{3}. $$ The training profile $(20,4)$ standardizes to $$ z_{T,1}=\frac{20-15}{10/\sqrt{3}}=\frac{\sqrt{3}}{2}, \qquad z_{T,2}=\frac{4-3}{2/\sqrt{3}}=\frac{\sqrt{3}}{2}. $$ Each standardized difference is $\sqrt{3}/2$, so $$ d(\text{new},T)=\sqrt{(\sqrt{3}/2)^2+(\sqrt{3}/2)^2} =\sqrt{\frac{3}{2}}\approx1.2247. $$

Check. Each feature contributes $3/4$ to squared distance, for a total of $3/2$; its square root is approximately $1.2247$. Common mistake avoided: recalculating the means and standard deviations after adding the new profile. That would let the new observation change the scaling applied to the training profiles.

Units, Weights, and Features That Cannot Be Scaled

Standardization makes distances invariant to a positive change in measurement units for an individual feature, provided its mean and standard deviation are recomputed in the converted units. It does not mean that every feature must be treated as equally important. After standardization, a one-standard-deviation difference has the same numerical contribution in each feature; whether that is an appropriate comparison depends on the analysis.

For example, if domain knowledge says a feature should count twice as much in squared distance, that choice can be represented with an explicit feature weight $w_j$: $$ d(a,b)=\sqrt{\sum_{j=1}^{p}w_j(z_{aj}-z_{bj})^2}. $$ A weight is a modeling decision, not a consequence of standardization. Standardizing and then assigning weights is different from assuming all standardized features should automatically receive equal importance.

A feature with $s_j=0$ in the training set is constant there, so the standardization formula would divide by zero. It cannot be scaled by this rule. Usually the feature is removed from the distance calculation because it provides no variation among training observations. If a later observation has a different value, that discrepancy requires a deliberate data-handling decision; it does not make division by zero valid.

Sample standard deviation is used here because it describes the spread of the observed training sample. Using a population standard deviation instead would multiply every standardized value for a given feature by a common factor. For consistent sample sizes, this differs from sample-based scaling by the same factor across features; nevertheless, choose a convention and apply it consistently, especially when comparing reported distances.

Worked Example: A Unit Conversion and a Nearly Ignored Feature

A hypothetical analyst has four training records with two features: claim amount in dollars, $(1000,1)$, $(2000,2)$, $(3000,3)$, and $(4000,4)$. Compare the raw and standardized Euclidean distances between the first two records. Then explain what happens if claim amount is recorded in thousands of dollars instead.

Method. Calculate the raw distance in each unit convention, then calculate training means and sample standard deviations and standardize both features.

In dollars, the raw differences are $1000$ and $1$, so $$ d_{\text{raw}}=\sqrt{1000^2+1^2}=\sqrt{1{,}000{,}001}\approx1000.0005. $$ The claim-amount feature dominates this raw distance.

The claim amounts have mean $(1000+2000+3000+4000)/4=2500$. The squared deviations sum to $2{,}250{,}000+250{,}000+250{,}000+2{,}250{,}000=5{,}000{,}000$. Thus their sample standard deviation is $s_1=\sqrt{5{,}000{,}000/3}=1000\sqrt{5/3}$. The second feature has mean $2.5$, squared deviations summing to $5$, and sample standard deviation $s_2=\sqrt{5/3}$. Between the first two observations, each feature differs by one quarter of its respective standard deviation: $1000/s_1=1/\sqrt{5/3}$ and $1/s_2=1/\sqrt{5/3}$. Therefore, $$ d_{\text{standardized}} =\sqrt{\frac{3}{5}+\frac{3}{5}} =\sqrt{\frac{6}{5}} \approx1.0954. $$

If claim amount is recorded in thousands of dollars, the raw difference becomes $1$ instead of $1000$, while the second-feature difference remains $1$. The raw distance is then $\sqrt{1^2+1^2}=\sqrt{2}\approx1.4142$. The standardized distance remains $\sqrt{6/5}$: both the claim-amount difference and its standard deviation have been divided by $1000$, so their ratio is unchanged.

Check. The standardized contribution of each feature is $3/5$ to squared distance, while the raw distance changes substantially under the unit conversion. Common mistake avoided: assuming a clustering result based on raw Euclidean distance is unaffected by whether dollars or thousands of dollars are recorded.

Common Scaling Traps

Scaling before splitting data. Computing scaling statistics from all observations before creating training and validation sets leaks information. Fit the transformation on training data only.
Using different scaling values for different sets. A validation observation and a training observation must be expressed in the same coordinate system. Reuse the training means and standard deviations.
Thinking standardization removes the need for judgment. Equal standard deviations do not prove that features are equally relevant. Consider the meaning of the variables and any justified weights.
Dividing by a zero standard deviation. A constant training feature cannot be standardized with the formula. Remove it or handle it through an explicit rule.
Assuming all distance metrics respond identically. The displayed formula is for Euclidean distance. Scaling also changes the feature differences used by other distance measures, but their distance formulas and effects must be evaluated on their own terms.

For a clustering analysis, the practical importance is direct: the distance matrix summarizes which observations are considered near or far. If an arbitrary unit determines those distances, the resulting cluster structure may reflect the units rather than the intended comparison. This tutorial is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.

Check Your Understanding

Work each problem using the stated training-set statistics or data, as appropriate.

A training feature has values $2,4,6$. Using the sample standard deviation, standardize a new value of $8$ with the training mean and standard deviation. (Answer: The training mean is $4$, the sample standard deviation is $2$, and the standardized value is $2$.)
Two features have training means $10$ and $5$, and training sample standard deviations $2$ and $4$. An observation has values $14$ and $1$. What is its standardized Euclidean distance from the training-mean point $(10,5)$? (Answer: $\sqrt{2^2+(-1)^2}=\sqrt{5}\approx2.2361$.)
A feature is converted from meters to centimeters, multiplying every value by $100$. If its mean and sample standard deviation are recomputed from the converted training data, what happens to each standardized value? (Answer: It is unchanged, because both the centered value and standard deviation are multiplied by $100$.)
A feature is constant at $7$ for every training observation. What prevents the usual standardization formula from being applied, and what is the usual distance-preprocessing response? (Answer: Its training standard deviation is zero, so division by it is undefined; usually remove the feature from the distance calculation.)
Two training features have sample standard deviations $5$ and $2$. For a pair of observations, their original-unit differences are $10$ and $2$. What is the standardized Euclidean distance? (Answer: The standardized differences are $2$ and $1$, so the distance is $\sqrt{2^2+1^2}=\sqrt{5}\approx2.2361$.)

What You'll Learn