Redundant Predictors And Double-Counted Similarity

Why Redundant Predictors Matter

A distance calculation treats each included predictor as a source of information about how two observations differ. That is appropriate when the predictors represent distinct aspects of the problem. But if several predictors measure nearly the same underlying characteristic, their differences can be counted repeatedly. The resulting distance may give that characteristic more influence than the analyst intended.

This issue arises even after predictors have been put on comparable scales. Standardization addresses differences in units and spread; it does not remove the influence of having several similar columns. Consequently, a clustering result can reflect the number of measurements available for one characteristic as much as the intended balance among characteristics.

For \(p\) numeric predictors, let \(z_{ij}\) be the value of predictor \(j\) for observation \(i\). The Euclidean distance between observations \(i\) and \(r\) is

A common preparation is to standardize each numeric predictor, for example by subtracting its sample mean and dividing by its sample standard deviation. This makes the units more comparable, but each standardized column still contributes its own squared difference to the sum.

How Similar Predictors Count Repeatedly

The effect is easiest to see for exact duplicates. If predictors 1 and 2 have the same standardized value for every observation, write their difference between observations \(i\) and \(r\) as \(\Delta\). Their combined contribution to squared Euclidean distance is

With just one copy, the contribution would be \(\Delta^2\). Thus two exact copies double that characteristic’s contribution to squared distance, or multiply its contribution to the distance itself by \(\sqrt{2}\) when all other feature differences are zero.

This is not restricted to identical columns. Strongly correlated predictors often move together, so their pairwise differences may repeatedly reflect much the same underlying contrast. They need not contribute equal values, and their combined influence is not always exactly equivalent to a simple multiplier. The practical question is whether the extra columns represent meaningful separate information for the clustering objective.

A Short Derivation: Why an Exact Duplicate Doubles a Contribution

Consider two predictors \(z_1\) and \(z_2\) that are exact copies, so \(z_{i1}=z_{i2}\) for every observation \(i\). For a pair of observations, let their difference in the first predictor be \(\Delta=z_{i1}-z_{r1}\). Since the second predictor is identical, \(z_{i2}-z_{r2}=\Delta\) as well. The two terms in squared Euclidean distance are therefore

The square root is taken only after the squared contributions from all predictors have been added. That is why the pair doubles the squared contribution, rather than doubling the full distance in general. If the duplicated predictors are the only ones that differ, the distance is \(\sqrt{2}|\Delta|\); if other predictors also differ, the effect depends on their contributions too.

Diagnosing the Influence of a Predictor Group

A useful diagnosis is to inspect predictors as groups of related measurements rather than as isolated columns. For each group, compare the sum of squared differences with the total squared distance for representative pairs of observations. A group that accounts for a large share of the total may be important, but it may also contain several near-duplicates. Both possibilities should be considered before changing the distance.

A correlation matrix can help identify pairs of numeric predictors that move together. It is a diagnostic, not an automatic deletion rule. Correlation depends on the observations in the data, and two highly correlated variables can still have distinct interpretations or differ meaningfully for some cases. Conversely, variables with moderate pairwise correlations may jointly represent overlapping information.

The analyst should also consider the meaning and construction of each variable. For example, a hypothetical customer file might contain both a total number of monthly sessions and a sessions-per-week measure calculated from that total. Including both may partly duplicate activity level. But a count of sessions and a measure of session duration could capture different aspects of engagement, even if they are correlated.

Two Remedies: Selection and Weighting

There are two direct ways to address unintended repeated influence. Variable selection removes one or more predictors judged redundant for the analysis. Weighting keeps the predictors but reduces the combined influence of a group. Neither remedy is automatically correct; the choice should reflect the intended meaning of similarity.

A weighted Euclidean distance can be written as

where \(w_j\) is a nonnegative weight for predictor \(j\). A larger weight gives that predictor more influence. If every weight is positive, the formula gives a distance for distinct predictor vectors; if some weights are zero, differences only in zero-weight predictors are ignored, so distinct rows can have distance zero.

One transparent group-weighting choice is to assign the same total weight to each substantive group. For a group \(G\) with \(m\) predictors, dividing each member’s squared-difference contribution by \(m\) makes the group contribution the average of its \(m\) squared differences. This does not make the group equivalent to one specific predictor in every case; it makes the group’s overall scale less dependent on how many measurements it contains.

Choosing a Remedy Without Losing Useful Information

Variable selection is often easiest to explain and audit. If two columns are exact duplicates, retaining both usually has no informational benefit for a distance based on those columns, and retaining one prevents repeated counting. For highly correlated but non-identical variables, the decision is less mechanical. One may be more reliable, more interpretable, or more relevant to the intended notion of similarity.

Weighting can be useful when an analyst wants to retain multiple related measurements but avoid letting a large group dominate merely because it has more columns. The weights should be stated and justified before interpreting the resulting clusters. A weighted distance is not a neutral correction: it encodes a choice about which differences matter and by how much.

In either approach, compare the resulting distances or cluster descriptions under plausible alternatives. If modest changes in a group’s representation substantially change which observations are similar, that sensitivity is important to report. It indicates that conclusions depend on the feature design, not only on the clustering algorithm.

Common Mistakes and Practical Checks

• Assuming standardization solves redundancy. Standardization changes a column’s scale; it does not remove a second column that measures the same characteristic.
• Treating correlation as proof of duplication. High correlation is a prompt for investigation, not by itself a reason to delete a variable.
• Ignoring the square root. Predictors contribute additively to squared distance. The effect on the final distance is not generally a simple addition or doubling.
• Weighting after calculating the distance. In weighted Euclidean distance, weights multiply squared differences before the terms are summed and the square root is taken.
• Choosing weights without a substantive rationale. Equal group weights are one possible design, not a universal standard. The intended definition of similarity should guide the choice.