Distances and Membership Strength as Predictors

Why Look Beyond the Cluster Label?

A hard cluster assignment gives each record one label: cluster 1, cluster 2, and so on. This is a compact summary, but it treats all records in the same cluster as equivalent. A record close to its assigned centroid and one near a boundary can receive the same label even though their positions in the feature space differ substantially. When clusters are used as predictors, those differences may carry useful information about the response.

Distance-based features retain some of that within-cluster detail. They can describe how far a record is from each centroid, how strongly it is associated with its nearest centroid relative to alternatives, or how close it is to the cluster’s center. These are candidate predictors, not guaranteed improvements: their predictive value must be assessed using an appropriate model evaluation design.

Definition: Suppose a clustering method has \(K\) centroids \(c_1,\ldots,c_K\), and record \(x\) is represented in the same feature space used to fit the clustering. Its hard assignment is \(g(x)=\arg\min_k d(x,c_k)\), where \(d(x,c_k)\) is the chosen distance from the record to centroid \(k\). Its distance profile is the vector \((d(x,c_1),\ldots,d(x,c_K))\). A hard assignment records only the index of the nearest centroid; the distance profile records the distances to all centroids.

For Euclidean distance in a \(p\)-dimensional feature space, with record coordinates \(x=(x_1,\ldots,x_p)\) and centroid coordinates \(c_k=(c_{k1},\ldots,c_{kp})\), the distance is

d(x,c_k)=\sqrt{\sum_{j=1}^{p}(x_j-c_{kj})^2}.

This expression assumes that Euclidean distance is appropriate for the variables and their scales. If clustering was performed on standardized variables, distances must be calculated in that standardized space, not by mixing standardized centroids with raw record values. If another distance was used to form the clusters, distance-based features should be consistent with that choice. A distance has no universal interpretation independent of the feature scaling and metric.

What the Hard Assignment Leaves Out

The assignment \(g(x)\) is a many-to-one transformation: many different records can have the same nearest centroid. Consequently, it does not preserve how far a record is from that centroid, whether it is nearly tied with another centroid, or how its distances to the remaining centroids compare. A model using only a cluster label cannot distinguish two records with the same label on the basis of those omitted quantities.

A distance profile provides a richer representation. Common choices include the distance to the assigned centroid, the distances to all centroids, and the difference between the second-smallest and smallest distances. A small difference between the two nearest distances indicates that the record is relatively close to a boundary between those centroids; a large difference indicates a clearer nearest-centroid preference. The difference is a descriptive feature, not a probability that the assignment is correct.

Worked Example: Two Records With One Hard Label

A hypothetical clustering uses three centroids in a two-dimensional feature space: \(c_1=(2,0)\), \(c_2=(0,0)\), and \(c_3=(2,1)\). Compare records \(A=(-1,0)\) and \(B=(0.6,0.8)\). Find each record’s distance profile and hard assignment, then state what the labels omit.

Method. Calculate the Euclidean distance from each record to each centroid, then choose the centroid with the smallest distance. This avoids inferring the assignment from a visual impression or comparing only one pair of centroids.

For \(A=(-1,0)\), the distances are

d(A,c_1)=\sqrt{(-1-2)^2+(0-0)^2}=3,\qquad d(A,c_2)=\sqrt{(-1-0)^2+(0-0)^2}=1,

and

d(A,c_3)=\sqrt{(-1-2)^2+(0-1)^2}=\sqrt{10}\approx 3.162.

Thus \(A\)’s distance profile is approximately \((3,1,3.162)\), and its hard assignment is cluster 2.

For \(B=(0.6,0.8)\), the distances are

d(B,c_1)=\sqrt{(0.6-2)^2+(0.8-0)^2} =\sqrt{1.96+0.64}=\sqrt{2.6}\approx 1.612,

d(B,c_2)=\sqrt{(0.6-0)^2+(0.8-0)^2} =\sqrt{0.36+0.64}=1,

and

d(B,c_3)=\sqrt{(0.6-2)^2+(0.8-1)^2} =\sqrt{1.96+0.04}=\sqrt{2}\approx 1.414.

So \(B\)’s distance profile is approximately \((1.612,1,1.414)\), and its hard assignment is also cluster 2. The comparison is precise: \(B\) is much closer to cluster 1, has the same distance to cluster 2, and is closer to cluster 3 than \(A\). Their shared hard label does not preserve these different distance patterns.

Check. For each record, the distance to cluster 2 is 1 and is smaller than its distances to clusters 1 and 3. Both assignments are therefore consistent with the computed profiles.

Membership-Strength Summaries

Instead of keeping raw distances, one can transform them into nonnegative weights that sum to one. A common construction is normalized inverse distance. For a positive power \(q\), define

w_k(x)=\frac{d(x,c_k)^{-q}}{\sum_{r=1}^{K}d(x,c_r)^{-q}}.

Here \(w_k(x)\) is larger when record \(x\) is closer to centroid \(k\), and the weights sum to one. The choice of \(q\) controls how sharply weight concentrates on the closest centroid: larger \(q\) generally gives more weight to the nearest centroids. These weights are a constructed membership-strength summary; unless a particular probabilistic clustering model justifies that interpretation, they are not probabilities that the record belongs to the clusters.

If a distance is zero, the inverse-distance expression is undefined. In that case, use a stated convention, such as assigning weight 1 to the coincident centroid and 0 to the others. If multiple centroids coincide with the record, a convention for dividing the weight among them must also be specified.

Worked Example: Converting Distances to Relative Weights

Use the distances from the previous example and set \(q=2\). Calculate the normalized inverse-distance weights for records \(A\) and \(B\), and compare the resulting summaries with their hard labels.

Method. Square each distance, take its reciprocal, and divide each reciprocal by the sum of the reciprocals. The squared distances are used because \(q=2\); the weights must be normalized after this transformation.

For \(A\), the squared distances are \(9,1,10\), so the unnormalized values are \(1/9,1,1/10\). Their sum is

\frac{1}{9}+1+\frac{1}{10} =\frac{10+90+9}{90} =\frac{109}{90}.

Therefore,

(w_1(A),w_2(A),w_3(A)) =\left(\frac{10}{109},\frac{90}{109},\frac{9}{109}\right) \approx (0.0917,0.8257,0.0826).

For \(B\), the squared distances are \(2.6,1,2\), so the unnormalized values are \(5/13,1,1/2\). Their sum is

\frac{5}{13}+1+\frac{1}{2} =\frac{10+26+13}{26} =\frac{49}{26}.

Thus,

(w_1(B),w_2(B),w_3(B)) =\left(\frac{10}{49},\frac{26}{49},\frac{13}{49}\right) \approx (0.2041,0.5306,0.2653).

The weights for each record sum to 1. Both records still have cluster 2 as their nearest-centroid assignment, but \(B\)’s relative weights for clusters 1 and 3 are larger than \(A\)’s. This representation retains some of the difference that the hard label alone omits.

Common mistake avoided. Do not call these weights posterior probabilities without a model that supports that interpretation. They are normalized functions of distances, and their values depend on the distance metric and the selected power \(q\).

Why Normalized Weights and Raw Distances Differ

A raw distance profile includes information about absolute separation from the centroids. Normalized inverse-distance weights primarily express relative separation. For example, if every distance is multiplied by the same positive constant \(a\), then

\frac{(a\,d(x,c_k))^{-q}}{\sum_{r=1}^{K}(a\,d(x,c_r))^{-q}} = \frac{a^{-q}d(x,c_k)^{-q}}{a^{-q}\sum_{r=1}^{K}d(x,c_r)^{-q}} =w_k(x).

The common factor cancels. Thus the weights cannot distinguish two profiles that differ only by a common multiplier, even though one record may be much farther from all centroids. If absolute distance could matter, include a separate feature such as the distance to the nearest centroid alongside the normalized weights.

This cancellation also explains why “strongest membership” should not automatically be read as “close to a cluster.” A record can have one weight that dominates the others while still being far from every centroid. The nearest-centroid distance supplies information that relative weights alone discard.

Justification: The hard assignment keeps only which distance is smallest, so it cannot distinguish records with the same nearest-centroid index. A distance profile retains each distance. Normalized inverse-distance weights retain relative distance information because their denominator rescales the transformed distances to sum to one, but this normalization removes common absolute scale. Keeping a nearest-centroid distance as an additional feature can restore that particular scale information.

Assessing Whether the Extra Detail Helps

A more detailed feature representation is not necessarily a better predictor. It can preserve useful variation, but it can also add noise, redundancy, or sensitivity to the chosen scaling and distance metric. A fair comparison holds the outcome, data split, model class, and other modelling choices constant while comparing, for example, a cluster-label feature with a distance-based alternative. The evaluation should use records not used to fit the predictive model, and any clustering or feature transformations should be fit without using those evaluation outcomes.

The following small example illustrates the information that a distance feature can add. Its deliberately simple relationship is not evidence that distance features will improve performance in other data.

Worked Example: Comparing Label-Only and Distance-Based Predictions

In a hypothetical training sample, four records all have hard assignment “cluster 1.” Their distances to that assigned centroid are \(1,2,3,4\), and their numeric outcomes are \(12,14,16,18\). Two held-out records also have hard assignment “cluster 1”; their distances are \(1.5\) and \(3.5\), and their observed outcomes are \(13\) and \(17\). Compare a model using only the cluster label with a linear model using the distance to the assigned centroid. Use root mean squared error (RMSE) on the two held-out records.

Method. Fit each model on the four training records, generate predictions for the held-out records, and calculate RMSE as the square root of the mean squared prediction error. This avoids declaring a feature useful merely because it distinguishes the training records.

With only the cluster label, the training model can predict only one value for this cluster. The least-squares prediction is the training mean:

\bar{y}=\frac{12+14+16+18}{4}=15.

It therefore predicts 15 for both held-out records. The errors are \(13-15=-2\) and \(17-15=2\); the squared errors are 4 and 4. Hence

\operatorname{RMSE}_{\text{label}} =\sqrt{\frac{4+4}{2}} =\sqrt{4}=2.

For the distance model, let \(d\) denote distance to the assigned centroid and fit \(\hat{y}=b_0+b_1d\). The training means are \(\bar{d}=2.5\) and \(\bar{y}=15\). The least-squares slope is

b_1= \frac{\sum(d_i-\bar{d})(y_i-\bar{y})} {\sum(d_i-\bar{d})^2} = \frac{4.5+0.5+0.5+4.5}{2.25+0.25+0.25+2.25} =\frac{10}{5}=2.

The intercept is \(b_0=\bar{y}-b_1\bar{d}=15-2(2.5)=10\), so the fitted equation is \(\hat{y}=10+2d\). At distances 1.5 and 3.5, the predictions are 13 and 17. The held-out errors are both zero, giving \(\operatorname{RMSE}_{\text{distance}}=0\).

In this constructed example, distance distinguishes two records that the label-only representation treats identically, and the distance model predicts the held-out outcomes exactly. The check is consistent with the stated relationship \(y=10+2d\). In an actual analysis, a perfect result on a small holdout would not establish reliable performance; a larger evaluation design and attention to sampling variability would be needed.

Common mistake avoided. Do not compare training error for one feature set with test error for another. Here both models are assessed on the same two held-out records, and both are trained on the same four records.

Practical Interpretation and Common Traps

Distance-based features are most interpretable when the analyst can explain the feature space, scaling, and metric. For example, “distance to the assigned centroid” is a measure of separation in the transformed feature space; it is not automatically a real-world distance or a risk score. Similarly, a small gap between the two closest centroids indicates relative proximity to both, not a calibrated uncertainty measure.

Mixing scales. If clustering used scaled variables, calculate distances using the same scaling. Changing scales changes distances and can change which centroid is nearest.
Keeping only normalized weights. Weights can describe relative proximity but discard common absolute distance. Consider retaining a nearest-centroid distance when that distinction may matter.
Treating weights as probabilities. A sum of one does not make a vector probabilistic. Use “normalized membership-strength weights” unless a probabilistic model justifies a probability interpretation.
Assuming more features must help. Extra distances may be redundant or noisy. Compare alternatives using a consistent evaluation design rather than relying on a more detailed representation’s apparent plausibility.
Using the response to define unsupervised features. The clustering and distance transformation should be constructed without using the outcome to determine the clusters. Otherwise, apparent predictive value can be misleading.

Key takeaway: A hard cluster label preserves the identity of the nearest centroid but discards within-cluster distance information. Distance profiles and normalized membership-strength summaries can retain useful distinctions, but they preserve different information and should be compared as candidate predictors rather than assumed improvements. This topic is aligned with the SOA Exam PA (Predictive Analytics) syllabus.

Check Your Understanding

For each problem, state the method and check that assignments, weights, and prediction errors agree with the given values.

A record has distances \((2,5,7)\) to three centroids. Give its hard assignment and its nearest-versus-second-nearest distance gap. (Answer: Cluster 1; gap \(5-2=3\).)
Two records have distance profiles \((1,4,6)\) and \((3,4,5)\). Identify their hard assignments and compare their distances to clusters 1, 2, and 3. (Answer: Both are assigned to cluster 1; the second is farther from cluster 1, equally far from cluster 2, and closer to cluster 3.)
For distances \((1,2,4)\), calculate normalized inverse-distance weights using \(q=2\), rounding to four decimals. (Answer: \((16/21,4/21,1/21)\approx(0.7619,0.1905,0.0476)\).)
A record has distances \((2,4)\), and another has distances \((6,12)\). Using normalized inverse-distance weights with \(q=1\), compare their weights and state what feature information differs. (Answer: Both have weights \((2/3,1/3)\); the relative weights match, but the second record is three times as far from both centroids.)
A label-only model makes held-out predictions \(20,20,20\) for outcomes \(18,20,23\). A distance model predicts \(19,21,22\) for the same outcomes. Calculate both RMSEs to four decimals and identify which is lower. (Answer: Label-only RMSE \(=\sqrt{(4+0+9)/3}\approx2.0817\); distance-model RMSE \(=\sqrt{(1+1+1)/3}=1.0000\), so the distance model is lower on these records.)

What You'll Learn