Why Predictor Redundancy Matters
A predictive data set can contain many columns without containing the same amount of distinct information. Two columns might record exactly the same quantity, measure nearly the same thing, or be linked by a rule that makes one recoverable from others. These are different sources of predictor redundancy. Recognizing which source is present helps explain whether a model can be fitted uniquely, whether its coefficients have a clear interpretation, and whether predictions are likely to be sensitive to small changes in the data.
Here, \(X_1,\ldots,X_p\) denote predictors and \(Y\) denotes the response. A model uses information in the predictors to represent or predict \(Y\). Redundancy concerns relationships among the predictors; it does not, by itself, establish that any predictor is irrelevant to the response. In particular, two highly related measurements can each carry information that matters for a particular prediction task.
Four Sources of Redundancy
It is useful to distinguish the mechanism creating overlap, not just to label a pair of predictors “related.” The table summarizes four common cases.
| Source | Relationship | Potential modelling consequence |
|---|---|---|
| Exact duplicate | Two columns have identical recorded values | Separate coefficients cannot be identified in a model that includes both |
| Deterministic relationship | One predictor is an exact function or combination of others | Some predictor effects are aliased; a unique coefficient representation may not exist |
| Near-duplicate | Two columns are almost, but not exactly, equal or proportional | Coefficients may be sensitive to small changes in data or model specification |
| Correlated measurements | Predictors vary together but are not exact functions of one another | They may carry overlapping information and make separate effects difficult to interpret |
An exact duplicate can result from accidentally copying a column or retaining two differently named versions of the same measure. A deterministic relationship can arise when a feature is calculated from other fields: for example, total distance may equal urban distance plus rural distance. A near-duplicate can arise when two systems record almost the same quantity with small rounding or measurement differences. Correlated measurements can arise when different variables reflect a shared underlying factor, such as two distinct measures of customer activity.
The distinction is important. Exact redundancy creates a mathematical identification problem for certain models. Near-duplicates and correlated measurements generally do not make the model mathematically impossible to fit, but they can make coefficient estimates less stable or less informative about the predictors’ separate contributions.
What Exact Redundancy Does to a Model
In a regression model with a linear predictor \(\eta=\beta_0+\beta_1X_1+\beta_2X_2\), suppose the recorded columns satisfy \(X_2=X_1\) for every observation. Substitution gives
The data can inform the combined coefficient \(\beta_1+\beta_2\), but not how that sum should be divided between \(\beta_1\) and \(\beta_2\). For instance, if the sum is 2, the pairs \((2,0)\), \((1,1)\), and \((0,2)\) all give the same linear predictor for every observation. This is not merely high uncertainty about a unique answer: there is no unique separate answer in this model.
The same structural issue applies to generalized linear models whose linear predictor includes these columns. Since their fitted values are determined from \(\eta\), identical values of \(\eta\) from different coefficient pairs give identical fitted values and likelihood. Software may drop a column, report a coefficient as unavailable, or warn that the model is singular. The precise behavior depends on the software and fitting method; it does not create information that the data lack.
A deterministic relationship can create the same kind of problem even when no two columns are identical. For example, if \(X_3=X_1+X_2\), then
The fitted linear predictor depends on the two combined effects, not on a unique allocation among all three coefficients. A related issue appears with categorical predictors: including indicators for every level of a factor alongside an intercept creates an exact dependency, because the indicators sum to 1 for each observation. Appropriate reference-level coding avoids that particular dependency.
Worked Example: Two Identical Experience Columns
A hypothetical insurer fits a model for an account-level response using years of recorded driving experience, \(X_1\), and a second column, \(X_2\), created during a data merge. The three observations are \((X_1,X_2,Y)=(0,0,1),(1,1,3),(2,2,5)\). The proposed model is \(Y=\beta_0+\beta_1X_1+\beta_2X_2+\varepsilon\). Determine what can be learned about the coefficients.
Step 1: Check the columns. For each row, \(X_2=X_1\): the pairs are \((0,0)\), \((1,1)\), and \((2,2)\). The columns are exact duplicates.
Step 2: Substitute the relationship. The fitted value is \(\hat{Y}=\beta_0+(\beta_1+\beta_2)X_1\). Only the intercept and the sum of the two slopes can affect fitted values.
Step 3: Use the observed response pattern. The response increases from 1 to 3 to 5 as \(X_1\) increases from 0 to 1 to 2. The line through these points has intercept 1 and slope 2: \(1+2(0)=1\), \(1+2(1)=3\), and \(1+2(2)=5\). Thus \(\beta_0=1\) and \(\beta_1+\beta_2=2\).
Step 4: State the conclusion. Coefficient pairs \((\beta_1,\beta_2)=(2,0),(1,1),(0,2)\) all reproduce these fitted values. The separate effects of the two columns are not identifiable; the combined slope is 2. The check is that each pair sums to 2 and therefore produces predictions 1, 3, and 5.
Method and trap avoided. Compare the predictor columns, then simplify the linear predictor before interpreting coefficients. Reporting both slopes as uniquely estimated effects would mistake an arbitrary software choice for information supplied by the data.
Near-Duplicates and Correlated Measurements
With near-duplicates, the columns differ, so the model may have a unique coefficient estimate. However, the difference between the columns can be small relative to their overall variation. The model then has limited information for separating their effects. Small changes in observations, the selected predictors, or the modelling assumptions can produce substantial changes in individual coefficients.
Correlated measurements need not be almost identical. For example, two different measures of activity might often rise together, while still differing for particular observations. Each may contribute useful information, but their separate coefficients describe conditional relationships: the contribution associated with changing one predictor while holding the other fixed. If that “holding fixed” comparison is rare or implausible in the observed data, the separate coefficient interpretations may be difficult to support.
This leads to an important distinction: coefficient stability and prediction stability are not the same thing. When predictors move together in the data, a model may predict well over the observed range even though it cannot estimate their separate effects precisely. Predictions for unusual combinations of those predictors may be less reliable, because the data provide little evidence about such combinations. Redundancy therefore calls for attention to the purpose of the analysis: estimating distinct effects, explaining a model, or predicting within a familiar population are not identical goals.
Worked Example: A Derived Total and Its Components
A hypothetical travel-risk model uses urban miles \(U\), rural miles \(R\), and total miles \(T\), where the recorded definition is \(T=U+R\). Its linear predictor is \(\eta=\beta_0+\beta_U U+\beta_R R+\beta_T T\). Consider four accounts: \((U,R,T)=(1,2,3),(2,1,3),(3,2,5),(1,1,2)\). A response pattern is given by \(Y=4+3U+2R\). Show why including all three distance columns does not give three separately identifiable coefficients.
Step 1: Verify the definition. The totals are \(1+2=3\), \(2+1=3\), \(3+2=5\), and \(1+1=2\). Thus \(T=U+R\) in every row.
Step 2: Calculate the response values. The specified response rule gives \(4+3(1)+2(2)=11\), \(4+3(2)+2(1)=12\), \(4+3(3)+2(2)=17\), and \(4+3(1)+2(1)=9\). The four response values are therefore 11, 12, 17, and 9.
Step 3: Substitute the deterministic relationship. The model’s predictor terms become \(\beta_U U+\beta_R R+\beta_T(U+R)=(\beta_U+\beta_T)U+(\beta_R+\beta_T)R\). Only these two sums determine the distance contribution.
Step 4: Compare equivalent coefficients. The response rule corresponds to \(\beta_U+\beta_T=3\) and \(\beta_R+\beta_T=2\), with \(\beta_0=4\). One choice is \((\beta_U,\beta_R,\beta_T)=(3,2,0)\); another is \((2,1,1)\). Both give effective coefficients \(2+1=3\) for \(U\) and \(1+1=2\) for \(R\), so both produce 11, 12, 17, and 9. The check confirms identical fitted values despite different individual coefficients.
Method and trap avoided. Substitute the feature definition into the model and identify which coefficient combinations actually determine predictions. Interpreting \(\beta_T\) as a distinct “effect of total miles, holding both components fixed” is not meaningful here: holding \(U\) and \(R\) fixed also fixes \(T\).
A near-duplicate illustrates a different issue. Let \(X\) take values \(0,1,2,3\), let the small measurement difference \(E\) take values \(0.1,-0.1,-0.1,0.1\), and define a second measurement \(Z=X+E\). Suppose the response follows \(Y=5+2X+0.5E\). Since \(E=Z-X\), the same rule can be written \(Y=5+1.5X+0.5Z\). The row-by-row values are:
| Row | \(X\) | \(E\) | \(Z=X+E\) | \(Y=5+2X+0.5E\) |
|---|---|---|---|---|
| 1 | 0 | 0.1 | 0.1 | 5.05 |
| 2 | 1 | -0.1 | 0.9 | 6.95 |
| 3 | 2 | -0.1 | 1.9 | 8.95 |
| 4 | 3 | 0.1 | 3.1 | 11.05 |
Here \(Z\) is not an exact duplicate: the differences \(Z-X\) are \(0.1,-0.1,-0.1,0.1\). The two measurements share a strong common pattern, while the smaller difference distinguishes them. In an actual model, noise in \(Y\) could make the estimated effects associated with that small difference unstable. The exact coefficients above follow from the constructed response rule; they do not guarantee that noisy data would estimate the separate effects precisely.
Practical Interpretation and Exam Traps
When investigating suspected redundancy, first ask what relationship actually holds across the observed rows. Identical columns, an exact formula, and a strong but imperfect association have different implications. Then ask what the modelling goal requires. If distinct coefficient interpretations matter, duplicated or deterministically derived predictors need particular care. If prediction is the primary goal, overlapping predictors may still be useful, but coefficient instability and predictions for unfamiliar predictor combinations should be considered.
- Do not treat a software output as proof of identifiability. A dropped column, arbitrary constraint, or reported coefficient does not make an exact relationship disappear.
- Do not assume related predictors are interchangeable. Two measurements can be correlated yet differ in ways that matter to the response.
- Do not equate redundancy with irrelevance. A predictor can be useful for prediction even when much of its information overlaps with another predictor.
- Do not interpret a conditional effect without checking its meaning. A coefficient “holding the other predictors fixed” may describe an impossible or unsupported comparison when predictors are linked by definition or by the data.
A disciplined conclusion names the relationship and its consequence. For instance: “These columns are identical, so their separate slopes cannot be identified,” or “These measurements are similar but not identical, so both may contribute information while their separate coefficient estimates remain sensitive.” This is more precise than saying only that the predictors are “too correlated.”
Check Your Understanding
For each problem, identify the source of redundancy and state its main modelling implication.
- A model includes \(A=(2,4,6)\) and \(B=(2,4,6)\) as predictors. In a linear predictor with an intercept, the fitted coefficient sum for these columns is 1.5. Give two different slope pairs with that sum and state what is identifiable. (Answer: For example, \((1.5,0)\) and \((0.5,1.0)\); only the combined slope 1.5 is identifiable.)
- A model includes \(C\), \(D\), and \(S\), where \(S=C+D\) in every record. The effective coefficients on \(C\) and \(D\) are 4 and 1. Give one coefficient triple \((\beta_C,\beta_D,\beta_S)\) that produces those effective coefficients. (Answer: \((4,1,0)\); any triple satisfying \(\beta_C+\beta_S=4\) and \(\beta_D+\beta_S=1\) has the same linear predictor.)
- Two distinct sensors report \(M=(10,20,30)\) and \(N=(10.2,19.9,30.1)\). Are they exact duplicates? Name a plausible concern if both are used to estimate separate effects. (Answer: No; the values differ in each row. Their near-duplicate measurements may make separate coefficient estimates sensitive to small data changes.)
- A model uses two measures that often rise together, but neither is calculated from the other. Does that fact alone prove that one should be removed? Explain the main distinction to check. (Answer: No. They may contain distinct useful information; distinguish overlapping association from an exact or deterministic relationship and consider whether separate effects are meaningfully supported.)
- A factor has three levels represented by three indicator columns, and the model also includes an intercept. For every record, exactly one indicator equals 1. State the deterministic relationship and its implication. (Answer: The three indicators sum to 1 for every record, so their columns are linearly dependent with the intercept; all three level effects cannot be separately identified in that parameterization.)