A Worked Example Of Standardizing Before Clustering

Why Work Through the Scaling Arithmetic?

K-means assigns observations using Euclidean distances. When features use different units or have very different spreads, a feature with large numerical values can dominate those distances. Standardizing each feature provides a concrete way to put the features on comparable scales. The arithmetic matters: a wrong standard deviation changes every standardized value and can change which centroid is nearest.

In the examples below, standardization uses the sample standard deviation, calculated with denominator \(n-1\). Other software may use the population standard deviation, so always check the convention. Within a clustering analysis, calculate each feature’s scaling values once and apply those same values to every observation being compared.

Definition: For feature \(j\), let \(x_{1j},\ldots,x_{nj}\) be its observed values. Its sample mean and sample standard deviation are

\bar{x}_j=\frac{1}{n}\sum_{i=1}^n x_{ij}, \qquad s_j=\sqrt{\frac{\sum_{i=1}^n(x_{ij}-\bar{x}_j)^2}{n-1}}.

The sample-standardized value for observation \(i\) on feature \(j\) is \(z_{ij}=(x_{ij}-\bar{x}_j)/s_j\), provided \(s_j>0\). Subtracting the mean centers the feature; dividing by its standard deviation expresses deviations in standard-deviation units.

Worked Example: Calculate the Standardized Observations

Worked Example: Standardize Two Features

A hypothetical analyst records two numeric features for four regional insurance portfolios: a rating index and an annual event count. The index values are \(10,20,30,40\); the event counts are \(100,100,300,300\). Use sample standard deviations to standardize both features, then compare the distance between portfolios A and B before and after standardization. No interest-rate or mortality assumption is relevant: this is a deterministic calculation on the stated observations.

Method. Calculate each feature’s mean and sample standard deviation separately, then subtract its mean and divide by that standard deviation. The common mistake to avoid is dividing both features by one standard deviation or using the raw units after centering.

For the rating index, the mean is \((10+20+30+40)/4=25\). The deviations are \(-15,-5,5,15\), and their squared values sum to \(225+25+25+225=500\). Thus the sample variance is \(500/(4-1)=500/3\), and the sample standard deviation is \(\sqrt{500/3}\approx12.9099\).

For the event count, the mean is \((100+100+300+300)/4=200\). The deviations are \(-100,-100,100,100\), and their squares sum to \(40{,}000\). The sample variance is \(40{,}000/3\), so the sample standard deviation is \(\sqrt{40{,}000/3}\approx115.4701\).

The standardized values, rounded to four decimal places, are:

\begin{array}{c|rr} \text{Portfolio} & \text{Rating index} & \text{Annual event count}\\ A & -15/12.9099=-1.1619 & -100/115.4701=-0.8660\\ B & -5/12.9099=-0.3873 & -100/115.4701=-0.8660\\ C & 5/12.9099=0.3873 & 100/115.4701=0.8660\\ D & 15/12.9099=1.1619 & 100/115.4701=0.8660 \end{array}

In the original units, the difference from A to B is \(10\) on the rating index and \(0\) on event count. Their Euclidean distance is \(\sqrt{10^2+0^2}=10\). After standardization, their coordinate differences are \(10/12.9099\approx0.7746\) and \(0\), so the distance is approximately \(0.7746\). The result is checked directly: only the rating index changes between A and B, and its standardized difference is exactly \(10/\sqrt{500/3}=\sqrt{0.6}\approx0.7746\).

Compare Distances and Assignments

In \(p\) features, Euclidean distance between vectors \(a\) and \(b\) is \(\sqrt{\sum_{j=1}^p(a_j-b_j)^2}\). K-means assigns an observation to a nearest centroid by comparing these distances. Standardizing features changes the scale of each coordinate in that comparison; it does not guarantee that every assignment changes.

Worked Example: A Nearest-Centroid Assignment Changes

Use the two sample standard deviations from the preceding calculation: \(s_1=\sqrt{500/3}\approx12.9099\) for the rating index and \(s_2=\sqrt{40{,}000/3}\approx115.4701\) for the event count. Suppose an observation \(P=(9,0)\) is compared with two proposed centroids in the original units, \(C_1=(0,0)\) and \(C_2=(10,50)\). Determine the assignment before and after standardizing. Treat the given centers as fixed; the scale values come from the four portfolios above.

Raw-unit comparison. The distance from \(P\) to \(C_1\) is \(\sqrt{(9-0)^2+(0-0)^2}=9\). The distance to \(C_2\) is \(\sqrt{(9-10)^2+(0-50)^2}=\sqrt{1+2500}=\sqrt{2501}\approx50.0100\). Thus \(P\) is assigned to \(C_1\) using raw units.

Standardized comparison. Both points and centroids must use the same feature means and standard deviations. Subtracting a common feature mean from both coordinates does not change their difference, so the distance can be calculated from raw coordinate differences divided by the corresponding standard deviations. For \(C_1\), the squared distance is

\left(\frac{9}{12.9099}\right)^2+0^2 =\frac{81}{500/3} =\frac{243}{500} =0.486,

so its distance is \(\sqrt{0.486}\approx0.6971\). For \(C_2\), the squared distance is

\left(\frac{1}{12.9099}\right)^2+ \left(\frac{50}{115.4701}\right)^2 =\frac{1}{500/3}+\frac{2500}{40{,}000/3} =0.006+0.1875 =0.1935,

and the distance is \(\sqrt{0.1935}\approx0.4399\). The standardized assignment is therefore \(C_2\). The assignment changes because raw event-count differences dominate the original distance, whereas after scaling the 50-count difference is less influential relative to that feature’s observed spread. The mistake to avoid is standardizing \(P\) but leaving the centroids in raw units; distances between coordinates on different scales would then have no consistent interpretation.

As a check, the standardized squared distances are \(0.486\) and \(0.1935\); the latter is smaller, agreeing with the nearest-centroid decision. The means cancel in each point-to-centroid difference, but the feature-specific standard deviations do not.

Why the Standard Deviation Convention Matters

The sample standard deviation uses denominator \(n-1\). If instead the population standard deviation is calculated for the same \(n\) observed values, its denominator is \(n\). The relationship between the two follows directly from their definitions.

Compare the variances.
The sum of squared deviations is the same under both conventions. Therefore, the population variance is \(\sum_i(x_i-\bar{x})^2/n\), while the sample variance is \(\sum_i(x_i-\bar{x})^2/(n-1)\).

Take the ratio of standard deviations.
Taking square roots gives \(s_{\mathrm{pop}}=s_{\mathrm{sample}}\sqrt{(n-1)/n}\).

Compare standardized values.
The centered numerator is unchanged, so \(z_{\mathrm{pop}}=z_{\mathrm{sample}}\sqrt{n/(n-1)}\). Thus population-standardized coordinates, and distances formed from them when all features use the same \(n\), are larger by \(\sqrt{n/(n-1)}\).

Worked Example: Check the Sample-versus-Population Factor

Four observations on one feature are \(1,2,3,4\). Calculate the standardized value for \(4\) under both conventions, then verify their ratio. This calculation avoids the common trap of reversing the population-to-sample factor.

The mean is \((1+2+3+4)/4=2.5\). The squared deviations are \(2.25,0.25,0.25,2.25\), with sum \(5\). The sample standard deviation is \(\sqrt{5/3}\approx1.2910\); the population standard deviation is \(\sqrt{5/4}\approx1.1180\). The centered value for observation \(4\) is \(4-2.5=1.5\).

Using the sample standard deviation gives \(z_{\mathrm{sample}}=1.5/\sqrt{5/3}\approx1.1619\). Using the population standard deviation gives \(z_{\mathrm{pop}}=1.5/\sqrt{5/4}\approx1.3416\). Their ratio is \(1.3416/1.1619\approx1.1547\), which agrees with \(\sqrt{4/3}\approx1.1547\). Therefore the population-standardized value is larger, not smaller. If every feature uses this population convention on the same \(n=4\) observations, each standardized Euclidean distance is multiplied by the same factor \(\sqrt{4/3}\); nearest-centroid rankings are unchanged by that common factor.

Practical Checks Before Clustering

Standardization is a calculation choice, not an automatic guarantee that the resulting clusters answer the intended question. It gives each nonconstant feature a standard-deviation-based scale. Whether that is appropriate depends on the analysis; here the central task is to calculate consistently and understand the resulting distances.

Use one convention throughout. Record whether standard deviations use \(n-1\) or \(n\). Do not mix conventions across features.
Use the same scale values for all compared points. When assigning a new observation using existing centers, apply the means and standard deviations calculated for the reference data. Recalculating them separately for that observation would make the coordinates incomparable.
Check for zero spread. If a feature is constant, then \(s_j=0\) and its standardized value is undefined. It cannot be standardized by this formula; do not divide by zero.
Do not confuse centering with scaling. Subtracting the same feature mean from every point preserves point-to-point coordinate differences. Dividing by a feature-specific standard deviation changes the relative contributions of features to distance.

Key takeaway: For each feature, calculate its own mean and standard deviation, transform every compared coordinate using those same values, and then calculate Euclidean distances in the transformed coordinates. State the standard-deviation convention and check the assignment by comparing the resulting distances.

Check Your Understanding

Work each problem using the stated convention. Round displayed distances to four decimal places.

Three values for a feature are \(2,4,6\). Using the sample standard deviation, calculate the mean, standard deviation, and standardized value of \(6\). (Answer: mean \(=4\), sample SD \(=2\), standardized value \(=1.0000\))
Two observations have raw feature differences \(3\) and \(40\), and the corresponding sample standard deviations are \(3\) and \(80\). Calculate their raw and standardized Euclidean distances. (Answer: raw distance \(=\sqrt{1609}\approx40.1123\); standardized distance \(=\sqrt{1.25}\approx1.1180\))
For two features, the sample-standardized coordinate differences between a point and a centroid are \(0.6\) and \(0.8\). Find their Euclidean distance. If the same data are standardized using population standard deviations with \(n=5\), what is the new distance? (Answer: sample-standardized distance \(=1.0000\); population-standardized distance \(=\sqrt{5/4}=1.1180\))
A point is \(P=(4,0)\), and two centroids are \(C_1=(0,0)\) and \(C_2=(5,20)\). The two feature standard deviations are \(s_1=2\) and \(s_2=10\). Which centroid is nearest before scaling, and which is nearest after scaling? (Answer: raw distances are \(4\) and \(\sqrt{401}\approx20.0250\), so \(C_1\); standardized distances are \(2\) and \(\sqrt{1/4+4}=\sqrt{4.25}\approx2.0616\), so \(C_1\) remains nearest)
A feature has values \(5,5,5,5\). Can its observations be standardized using the sample z-score formula? Explain. (Answer: No. Its sample standard deviation is \(0\), so the formula divides by zero; the feature has no variation.)

What You'll Learn