Choosing a Distance Measure for the Data

Why the Distance Measure Matters

Hierarchical clustering starts by comparing observations. The distance measure determines what counts as similar before any linkage rule is applied: if one feature is measured in dollars and another in days, or if a large difference in one feature should count more than several small differences, the choice of measure can change which observations are nearest. A useful distance is therefore not simply a formula selected by convention; it expresses a choice about the meaning of similarity for the task.

This tutorial compares common distances for numeric observations and explains how units, correlation, and unusual values affect their interpretation. The discussion is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.

Common Distances for Numeric Observations

Let \(x=(x_1,\ldots,x_p)\) and \(y=(y_1,\ldots,y_p)\) be two observations on \(p\) numeric features. The difference in feature \(j\) is \(x_j-y_j\). The most familiar measures aggregate these coordinate differences in different ways.

Euclidean distance gives greater weight to a coordinate difference when that difference is large, because it is squared before the sum is taken. Manhattan distance adds absolute differences, so a given difference contributes in direct proportion to its size. Chebyshev distance keeps only the largest coordinate difference. These are distinct definitions of separation, and can rank pairs differently.

Squared Euclidean dissimilarity, \(\sum_j(x_j-y_j)^2\), is sometimes used in clustering calculations. It is not a metric: for points \(0,1,2\) on a line, the squared distance from \(0\) to \(2\) is \(4\), while the sum of the squared distances from \(0\) to \(1\) and \(1\) to \(2\) is \(1+1=2\). Thus the triangle inequality fails. Do not report a squared distance as if it were an ordinary Euclidean distance.

Units, Scaling, and Standardized Euclidean Distance

Ordinary Euclidean and Manhattan distances use the recorded units directly. If one feature is recorded in large numerical units, its differences can dominate the total even when the feature is not more important to the clustering task. Changing a feature from dollars to cents multiplies its differences by 100, which can substantially change the distances.

One response is to scale each feature by a positive reference scale \(s_j\), often its standard deviation in a specified reference data set. The resulting standardized Euclidean distance is

This puts differences in units of their feature-specific scales. It does not mean that all features are equally important in the substantive problem; it means that a one-scale-unit difference in each feature contributes equally to this particular distance. The scales should be selected and justified for the task. If every feature already has a meaningful common unit and intended weight, standardization may remove information rather than improve comparability.

The formula follows directly from transforming the coordinates. Define \(z_j(x)=(x_j-c_j)/s_j\), where \(c_j\) is any common center for feature \(j\). Then the difference between transformed observations is \(z_j(x)-z_j(y)=(x_j-y_j)/s_j\). Applying ordinary Euclidean distance to the transformed observations gives the standardized formula. The center cancels; the scale does not.

Mahalanobis Distance and Correlated Features

Standardized Euclidean distance adjusts for feature scales but treats the features as uncorrelated. When numeric features are correlated, a difference in a direction that commonly varies together may be less unusual than a difference of the same Euclidean length in a direction rarely observed. Mahalanobis distance accounts for both scales and linear correlation using a covariance matrix.

The inverse covariance matrix weights differences according to their covariance structure. To see why, suppose \(\Sigma=LL^\mathsf{T}\) is a Cholesky factorization, and set \(w=L^{-1}(x-y)\). Then \[ w^\mathsf{T}w=(x-y)^\mathsf{T}(L^{-1})^\mathsf{T}L^{-1}(x-y) =(x-y)^\mathsf{T}\Sigma^{-1}(x-y). \] Thus Mahalanobis distance is Euclidean distance after transforming differences to remove covariance and scale. The covariance estimate matters: if it is poorly estimated or distorted by extreme observations, the resulting distances can be misleading.

Outliers and Choosing a Measure for the Task

A large coordinate difference can strongly affect Euclidean distance because it is squared. Manhattan distance still responds to a large difference, but its contribution grows linearly rather than quadratically. Neither choice makes the data immune to outliers: an extreme observation can still be far from many other observations and can affect the cluster structure. In Mahalanobis distance, an extreme value can also affect the estimated covariance matrix, which may then change distances for the whole data set.

Choosing robust scales or a robust covariance estimate can reduce sensitivity to unusual observations, but those choices require justification and do not repair a mismatch between the features and the task. Consider whether the unusual values are errors, rare but valid cases, or the very cases the analysis needs to distinguish. Removing or downweighting them without that assessment can erase relevant structure.

A practical choice starts by stating what “similar” means in context. Use the feature units directly when they are comparable and meaningful for the objective. Consider standardized Euclidean distance when features have different scales and equal importance is intended per reference-scale unit. Consider Mahalanobis distance when covariance-adjusted differences are the desired notion of unusualness and a suitable covariance estimate is available. Manhattan distance may be useful when a sum of absolute coordinate differences is meaningful. These are guides, not automatic rules; the distance and its inputs should be reported together.