How Feature Scaling Changes Euclidean Distance

Why Feature Scales Matter for Distance

Euclidean distance is a way to summarize how different two observations are across several features. In multi-omics data, those features may have very different units and numerical ranges. A difference of 20 might be small for one feature and large for another; the distance calculation itself does not know which. It squares each difference and adds the results.

As a result, a feature with large numerical differences can dominate a raw Euclidean distance, even if that feature is not more scientifically important. Scaling changes the geometry: it changes the amount each feature contributes to the distance. The purpose here is to make that change explicit, rather than to treat scaling as a purely mechanical preprocessing step.

Definition: For observations \(a\) and \(b\) measured on \(p\) features, let \(x_{aj}\) and \(x_{bj}\) denote their values for feature \(j\). Their raw Euclidean distance is \(d(a,b)=\sqrt{\sum_{j=1}^{p}(x_{aj}-x_{bj})^2}\). The contribution of feature \(j\) to the squared distance is \((x_{aj}-x_{bj})^2\); the squared contributions add to \(d(a,b)^2\).

The raw contribution is in squared units. If feature \(j\) is measured in milligrams per litre, its contribution is in squared milligrams per squared litre. It is therefore useful to compare contributions within one calculation, while remembering that their numerical sizes depend on the units used.

Distance After Standardization

For standardization, each feature is centered and divided by its sample standard deviation. In a typical workflow, the feature means and standard deviations are estimated from a reference or training data set. Write these estimates as \(\bar{x}_j\) and \(s_j\). The standardized value for observation \(i\) and feature \(j\) is \(z_{ij}=(x_{ij}-\bar{x}_j)/s_j\), provided \(s_j>0\).

d_z(a,b) = \sqrt{\sum_{j=1}^{p}(z_{aj}-z_{bj})^2} = \sqrt{\sum_{j=1}^{p}\frac{(x_{aj}-x_{bj})^2}{s_j^2}}.

Centering does not change a pairwise difference: \(z_{aj}-z_{bj}=[(x_{aj}-\bar{x}_j)-(x_{bj}-\bar{x}_j)]/s_j=(x_{aj}-x_{bj})/s_j\). Thus, after standardization, feature \(j\)'s contribution to the squared distance is its raw squared difference divided by \(s_j^2\). The scale estimate, not the feature mean, determines the change in pairwise distance.

Key result: Standardized Euclidean distance measures differences in units of each feature’s sample standard deviation. A raw difference of \(s_j\) on feature \(j\) contributes \(1\) to the squared standardized distance, regardless of that feature’s original measurement units.

This interpretation relies on applying the same feature-wise transformation to both observations. It also requires a finite, positive scale estimate for every included feature. If a feature is constant in the reference data, then \(s_j=0\) and standardization is undefined for that feature; it must be handled separately rather than divided by zero. The formulas describe distances between numerical feature vectors. They do not establish that the features are biologically comparable or that a larger distance represents a meaningful biological difference.

Worked Example: Comparing Raw and Standardized Contributions

Suppose a hypothetical reference set of five samples has ages of 40, 40, 50, 60, and 60 years, and transformed assay values of 100, 100, 200, 300, and 300 assay units. In this reference set, the sample mean and sample standard deviation are 50 years and 10 years for age, and 200 assay units and 100 assay units for the assay feature. Consider sample A, with age 40 and assay value 300, and sample B, with age 60 and assay value 100.

The age difference is \(40-60=-20\) years, and the assay difference is \(300-100=200\) assay units. The raw squared contributions are \( (-20)^2=400\) years squared and \(200^2=40{,}000\) assay units squared. Therefore,

d_{\mathrm{raw}}(A,B) = \sqrt{400+40{,}000} = \sqrt{40{,}400} \approx 201.0.

The numerical assay contribution is 100 times the age contribution, so it accounts for \(40{,}000/40{,}400\approx 0.9901\), or about 99.0%, of the raw squared distance. This comparison is a numerical description, not evidence that the assay feature is more important.

After standardization, the age difference is \(-20/10=-2\) standard deviations and the assay difference is \(200/100=2\) standard deviations. Each contributes \(2^2=4\) to the squared standardized distance, giving \(d_z(A,B)=\sqrt{4+4}=\sqrt{8}\approx 2.828\). Standardization has made these two particular differences equal in their contributions to squared distance.

Scaling Can Change Which Pair Is Farther Apart

Changing contributions can change more than the numerical value of a distance. It can change the ordering of pairs from nearer to farther apart. The following calculation uses the same reference standard deviations as the previous example: 10 years for age and 100 assay units for the assay feature.

Worked Example: A Pairwise Ranking Reverses

Compare two hypothetical pairs of samples. For pair P–Q, the absolute age difference is 2 years and the absolute assay difference is 40 units. For pair R–S, those differences are 10 years and 10 units. The raw squared contributions for P–Q are \(2^2=4\) and \(40^2=1{,}600\), so its raw distance is \(\sqrt{4+1{,}600}=\sqrt{1{,}604}\approx 40.05\).

For R–S, the raw squared contributions are \(10^2=100\) and \(10^2=100\). Its raw distance is \(\sqrt{100+100}=\sqrt{200}\approx 14.14\). On the original numerical scales, P–Q is farther apart than R–S.

After standardization, the P–Q differences are \(2/10=0.2\) and \(40/100=0.4\). Their squared contributions are \(0.04\) and \(0.16\), giving a distance of \(\sqrt{0.20}\approx 0.4472\). The standardized R–S differences are \(10/10=1\) and \(10/100=0.1\), with squared contributions \(1\) and \(0.01\); its distance is \(\sqrt{1.01}\approx 1.005\).

The ordering reverses: P–Q is farther on the raw scale, while R–S is farther after standardization. Neither calculation is arithmetically wrong. They answer different geometric questions: one uses original numerical differences, and the other measures differences relative to each feature’s standard deviation in the reference set.

Measurement Units and a Short Invariance Derivation

A useful check is to ask what happens if a feature is expressed in different units. Raw Euclidean distances can change because the numerical differences change. Standardized distances do not change under a positive unit conversion if the conversion is applied consistently to the observations and to the reference data used to estimate the mean and standard deviation.

Justification: If a feature is converted by \(x'_{ij}=c x_{ij}\), where \(c>0\), then its reference mean becomes \(\bar{x}'_j=c\bar{x}_j\) and its sample standard deviation becomes \(s'_j=cs_j\). Consequently, \((x'_{aj}-x'_{bj})/s'_j=c(x_{aj}-x_{bj})/(cs_j)=(x_{aj}-x_{bj})/s_j\). Its standardized difference, and therefore its contribution to standardized distance, is unchanged.

Worked Example: Converting an Assay Feature’s Units

Consider two hypothetical samples with an age difference of 2 years and a concentration difference of 0.03 mg/L. Suppose the reference standard deviations are 10 years and 0.10 mg/L, respectively. In the original units, the raw distance is

\sqrt{2^2+(0.03)^2} = \sqrt{4.0009} \approx 2.000.

If concentration is converted to \(\mu\)g/L, the difference becomes 30 \(\mu\)g/L and its reference standard deviation becomes 100 \(\mu\)g/L. The raw distance in the converted units is \(\sqrt{2^2+30^2}=\sqrt{904}\approx 30.07\). The large change in the reported raw distance comes from changing the concentration’s numerical scale, not from changing the samples.

For standardized distance, the age difference is \(2/10=0.2\), and the concentration difference is \(0.03/0.10=0.3\). The standardized distance is \(\sqrt{0.2^2+0.3^2}=\sqrt{0.13}\approx 0.3606\). After conversion, the concentration difference is \(30/100=0.3\), so the standardized distance remains 0.3606. The squared contributions are 0.04 for age and 0.09 for concentration in either set of units.

Choosing and Checking a Distance Scale

Standardization is one choice of geometry, not an automatic guarantee that a distance is scientifically appropriate. It treats a one-standard-deviation difference on each feature as equally large in the distance calculation. That may be useful when features have different units, but equal standardized contributions do not imply equal measurement reliability, biological relevance, or importance to the research question.

Distance approach	Feature contribution to squared distance	Useful interpretation and caution
Raw Euclidean distance	\((x_{aj}-x_{bj})^2\)	Uses original units; results can be dominated by numerical scale and change under unit conversion.
Standardized Euclidean distance	\((x_{aj}-x_{bj})^2/s_j^2\)	Measures differences in sample-standard-deviation units; the result depends on the reference standard deviations.
Robustly scaled Euclidean distance	Squared difference divided by a robust scale squared	Can reduce sensitivity of the scale estimate to extreme observations; the chosen robust scale changes the distance interpretation.
Pareto-scaled Euclidean distance	Squared difference divided by a feature scale, rather than its square	Applies less scale adjustment than standardization; it is a different weighting choice, not an equivalent way to obtain standardized distances.

The alternatives in the table correspond to different definitions of how a feature difference should count. They should not be treated as interchangeable settings. For example, dividing by \(s_j\) before squaring gives a standardized contribution of \((x_{aj}-x_{bj})^2/s_j^2\); dividing the squared difference by \(s_j\) instead gives a different contribution. The earlier material on standard, Pareto, and robust scaling distinguishes those preprocessing choices.

Several checks help reveal when a distance analysis may be going wrong:

Verify the scale estimates. Check that each \(s_j\) is positive, finite, and estimated from the intended reference samples. A very small scale can make modest differences contribute greatly.
Check for extreme observations. A sample standard deviation can be strongly affected by extreme values. If it changes substantially when a small number of observations are removed, standardized distances may also be sensitive to those observations.
Check the actual contributions. Calculate \((x_{aj}-x_{bj})^2/s_j^2\) for features of interest. A total distance alone hides which features are driving it.
Keep the transformation consistent. In prediction or other held-out evaluation, estimate means and scales using training data, then apply those same estimates to held-out samples. Re-estimating with held-out data changes the distance definition and can leak information from the evaluation set.
Consider dependence between features. Standardization adjusts each feature’s scale separately; it does not remove correlation. Highly correlated features can represent overlapping variation, and separate contributions in Euclidean distance do not account for that dependence.

A common mistake is to say that standardization makes all features equally influential. The correction is narrower: it makes a difference of one estimated standard deviation contribute one unit to the squared standardized distance. For two particular samples, their differences in standard-deviation units may be unequal, so their contributions will be unequal too. Nor does this operation adjust for reliability, remove correlation, or establish a causal interpretation.

Key takeaway: To understand a Euclidean distance, inspect its feature-wise squared contributions. Raw distance weights numerical differences; standardized distance weights differences relative to feature-specific standard deviations. Choose between them based on the intended interpretation, and state the scale used.

Check Your Understanding

Work through each question, then compare your reasoning with the solutions.

Two observations differ by 3 units on feature A and 4 units on feature B. What are the raw squared contributions and raw Euclidean distance?
The sample standard deviations for those features are 3 and 2. What is the standardized Euclidean distance, and which feature contributes more to its square?
If every value of feature A is converted from one unit to another by multiplying by 10, what happens to its raw squared contribution and its standardized contribution, assuming the reference mean and standard deviation are converted consistently?
Why can standardizing each feature separately fail to account for strong correlation between two features?

Solutions. (1) The squared contributions are \(3^2=9\) and \(4^2=16\), so the distance is \(\sqrt{9+16}=5\). (2) The standardized differences are \(3/3=1\) and \(4/2=2\). Their squared contributions are 1 and 4, so the distance is \(\sqrt{5}\approx 2.236\); feature B contributes more. (3) The raw squared contribution for A is multiplied by \(10^2=100\), because its difference is multiplied by 10 before squaring. Its standardized contribution is unchanged, because both the difference and its standard deviation are multiplied by 10. (4) Separate standardization rescales each feature but does not change their correlation. The distance still adds their squared differences separately, without representing the overlap in variation between them.