Choosing Which Redundant Predictors to Retain

Choosing Predictors Is a Scientific Decision

When two predictors contain much of the same information, a regression model may have difficulty separating their individual contributions. That difficulty can make coefficient estimates imprecise or sensitive to modest data changes. It does not, by itself, tell us which predictor to keep. The choice should depend on what the research question asks, how well each variable measures the intended concept, and which variables matter scientifically.

A predictor that is less strongly associated with the outcome is not automatically expendable, and the predictor with the smaller standard error is not automatically the better choice. A useful decision begins by identifying the purpose of the analysis: estimating an association of particular interest, adjusting for relevant variables, or predicting outcomes for new cases. Different purposes can support different predictor sets.

Definition: Predictors are redundant for an analysis when they contain substantially overlapping information about the cases being modeled. Redundancy is a matter of degree: variables can be highly related without being identical. The decision about which to retain depends on the research question and the quality and meaning of the measurements, not on redundancy alone.

State the Model and the Target First

For a response $Y$ and predictors $X_1,\ldots,X_k$, the multiple linear regression model describes the conditional mean of the response as a linear function of the predictors:

E(Y\mid X_1,\ldots,X_k)=\beta_0+\beta_1X_1+\cdots+\beta_kX_k.

Here, $\beta_j$ is the change in the model’s conditional mean for a one-unit increase in $X_j$, holding the other included predictors fixed. This is a conditional interpretation. If a correlated predictor is removed, the set of variables held fixed changes, so the coefficient may answer a different question.

For ordinary least-squares estimates and their usual standard errors and tests, the familiar assumptions include a correctly specified linear conditional mean, independent observations (or an appropriate model for dependence), and constant error variance; normal errors are additionally used for exact small-sample inference. These assumptions do not decide which scientifically relevant predictors to include. They are conditions for interpreting the regression calculations and inferential results. In the examples below, regression summaries are hypothetical; they are intended to illustrate predictor choice, not to assert that assumptions can be checked from a coefficient table alone.

Before comparing candidate predictors, state the target. If the aim is to describe the association between a particular exposure and an outcome while adjusting for a relevant factor, that factor may need to remain even when it overlaps with the exposure. If the aim is prediction, the central concern is performance on appropriate new cases, alongside measurement availability and quality. A model chosen for one aim need not be the best model for another.

Evidence to Use in a Retention Decision

Three kinds of evidence are especially useful. First, consider measurement quality: how directly and consistently does each variable measure the intended concept? Relevant information might include calibration, repeatability, missingness, timing, and whether the recorded values have a clear meaning. Second, consider subject-matter relevance: does the variable represent a scientifically important exposure, adjustment factor, or feature of the target population? Third, consider the research question: what comparison or prediction is the model meant to support?

These considerations can point in different directions. A highly reliable measure may be less relevant to the stated question than a less precise measure that captures the intended concept directly. A variable that is essential for adjustment should not be dropped simply because its individual coefficient is imprecise. Conversely, including every available version of a concept can make the coefficients difficult to interpret without making the model more useful.

Question	Evidence to examine	How it informs the choice
What is the variable meant to measure?	Definition, units, timing, and connection to the scientific concept	Prefer a predictor whose meaning matches the target of the analysis
How well is it measured?	Calibration, repeatability, missing values, and recording procedures	A better-quality measure may be more informative, provided it is relevant
Why is it in the model?	Exposure of interest, adjustment factor, or prediction feature	Retain variables that serve the stated purpose, not just those with small p-values
What changes if it is omitted?	Coefficient meaning, uncertainty, predictions, and population coverage	Describe the new model’s target and limitations rather than treating it as an equivalent model

A variance inflation factor or a sensitivity analysis can alert us to overlap and coefficient instability, but neither supplies a universal keep-or-drop rule. A large VIF does not mean a predictor is unimportant, and a small VIF does not make a predictor scientifically useful. The numerical diagnostics describe the fitted design; the substantive decision still requires context.

Worked Example: Choosing a Measure of Physical Activity

Suppose a hypothetical study relates physical activity to systolic blood pressure. Activity is recorded both by a calibrated wearable device and by a questionnaire asking participants to recall their weekly activity. The research question concerns the association with measured movement. Among 100 participants, usable device records are available for 82, while 96 complete the questionnaire. In a calibration subset, the device differs from a reference measure by an average of 6 minutes per day; questionnaire reports differ by an average of 24 minutes per day.

Both predictors describe activity, but the device is closer to the target concept and has better agreement with the reference in this hypothetical comparison. Its lower availability is a real disadvantage: only $82/100=0.82$, or 82%, have usable device records, compared with $96/100=0.96$, or 96%, for the questionnaire. The choice therefore needs to consider whether device availability changes which participants the analysis represents. Measurement quality is not a single number that overrides every other concern.

Assume, for illustration, that separate, otherwise comparable fitted models report a device coefficient of $-0.30$ mmHg per 10 additional minutes of daily activity and a questionnaire coefficient of $-0.12$ mmHg per 10 additional minutes. The device-model interpretation is: among participants with the same values of the other included predictors, a 10-minute higher device-recorded daily activity level is associated with a model-predicted systolic blood pressure that is 0.30 mmHg lower. The questionnaire-model interpretation uses questionnaire-reported activity instead. Neither coefficient establishes that increasing activity causes blood pressure to fall.

For this research question, the device is the stronger primary choice because it measures the intended concept more accurately. The analysis should also report the lower device availability and consider whether the people with usable records differ in important ways. The questionnaire can serve as a clearly labeled sensitivity comparison; it should not be treated as an interchangeable measurement just because it is available for more participants.

Why Coefficients Can Change When a Predictor Is Removed

Dropping one of two related predictors does not merely simplify the same interpretation. The remaining coefficient can change because it no longer describes a comparison among cases held constant on the omitted predictor. The following simple result makes this point precise.

Result: Suppose the population conditional mean is $E(Y\mid X,Z)=\beta_0+\beta_X X+\beta_Z Z$, with finite variances and $\operatorname{Var}(X)>0$. The slope in the linear regression of $Y$ on $X$ alone is $$ \beta_X+\beta_Z\frac{\operatorname{Cov}(X,Z)}{\operatorname{Var}(X)}. $$ Thus, unless $\beta_Z=0$ or $X$ and $Z$ have zero covariance, the one-predictor slope generally differs from the coefficient of $X$ in the model that also includes $Z$.

To justify the result, center $X$, $Z$, and $Y$, which removes the intercept without changing slopes. Under the stated conditional-mean model, write $Y=\beta_X X+\beta_Z Z+\varepsilon$, where the conditional mean of $\varepsilon$ given $X,Z$ is zero. The one-predictor population least-squares slope is $\operatorname{Cov}(X,Y)/\operatorname{Var}(X)$. Taking covariance with $X$ gives $\operatorname{Cov}(X,Y)=\beta_X\operatorname{Var}(X)+\beta_Z\operatorname{Cov}(X,Z)$, because $\operatorname{Cov}(X,\varepsilon)=0$. Dividing by $\operatorname{Var}(X)$ yields the stated expression.

This result explains a change in the statistical comparison, not necessarily a causal bias. Whether either coefficient has a causal interpretation requires additional scientific and design-based justification. For predictor selection, the practical lesson is to name the model being interpreted and state what is held fixed.

Worked Example: Retaining an Adjustment Variable Despite Overlap

In a hypothetical analysis of weekly earnings, the predictor of interest is years in a job, and age is also considered relevant to the research question. The two predictors are positively related. Suppose the model including both reports a years-in-job coefficient of \$20 per additional year, holding age fixed, and an age coefficient of \$5 per year, holding years in the job fixed. For this illustration, suppose $\operatorname{Cov}(X,Z)/\operatorname{Var}(X)=0.70$, where $X$ is years in the job and $Z$ is age.

If age is omitted, the result above gives the population one-predictor slope:

$$ 20+5(0.70)=20+3.50=23.50. $$

The one-predictor slope is therefore \$23.50 per additional year in the job, rather than \$20.00 per year holding age fixed. These coefficients answer different questions. In the two-predictor model, the comparison changes years in the job while holding age constant; in the reduced model, the comparison includes the way years in the job and age tend to vary together. The values are hypothetical and do not imply that either model’s coefficient is a causal effect.

If the research question requires an association with years in the job adjusted for age, the overlap alone is not a reason to remove age. The analysis should retain the relevant adjustment variable, acknowledge that overlap can make individual estimates uncertain, and report the conditional interpretation. If the research aim instead calls for an unadjusted descriptive association, the one-predictor model may be appropriate, but its changed target should be stated.

Make the Decision Traceable

A defensible choice can be documented without presenting it as a mechanical rule. For each overlapping predictor, record its intended meaning, its measurement strengths and limitations, its scientific role, and the consequence of including or excluding it. Then explain why the selected model fits the stated research aim. If more than one choice is reasonable, report how the coefficient interpretations, uncertainty, or predictions differ across the alternatives.

Name the target.
State whether the analysis is focused on an association, adjustment, or prediction, and identify the quantity the model is intended to describe.

Compare the measurements.
Check definitions, units, timing, availability, calibration, and repeatability; explain which limitations matter for the target.

Assess scientific roles.
Decide whether each candidate is an exposure of interest, a relevant adjustment variable, or a useful prediction feature.

State the consequences.
Explain what holding other variables fixed means in the chosen model and how the interpretation changes if a candidate is omitted.

Report the rationale and limitations.
Give the reason for retaining or excluding each candidate, and disclose meaningful uncertainty, missingness, or sensitivity to the choice.

Worked Example: Choosing Between Two Measures of Sleep

A hypothetical workplace analysis considers nightly time in bed and nightly sleep duration as predictors of next-day concentration score. The research question specifically concerns the association with time actually asleep. In a sample of 120 employees, both measures are available for everyone. A diary records time in bed to the nearest 5 minutes, while a validated wrist device estimates sleep duration. The device has occasional recording errors, but the diary cannot distinguish time asleep from time awake in bed.

For the stated question, sleep duration is the more direct measure, despite its recording limitations. Suppose a fitted model reports a coefficient of 0.40 concentration-score points per additional 30 minutes of device-estimated sleep, with the other included predictors held fixed. The interpretation is a 0.40-point higher model-predicted score per additional 30 minutes of recorded sleep, conditional on those predictors. It is not the coefficient for 30 more minutes in bed.

Retaining time in bed as well could be justified if the research question specifically asks about the association with sleep duration at a fixed time in bed. That conditional comparison is different and may be hard to estimate precisely when the measures overlap. If the goal is simply to estimate an association with actual sleep duration, including both without a clear reason can make the coefficients harder to explain. The analysis should therefore choose according to the target, and report the device’s measurement limitation rather than presenting its values as error-free.

Common Mistakes

Keeping the variable with the smaller p-value. A p-value is not a measure of measurement quality or scientific importance. Decide what the predictor represents and why it belongs in the model.
Dropping every predictor involved in collinearity. Overlap can reduce precision, but a variable may still be essential to the research question or adjustment strategy.
Calling coefficients from different models the same effect. A coefficient after removing a correlated predictor generally describes a different conditional comparison. State which predictors are held fixed.
Choosing solely by data availability. More complete measurement is useful, but it may be less direct or less accurate for the concept of interest. Report the trade-off and consider who is represented in the available data.
Treating a predictor-selection decision as proof of causation. Retaining a variable does not establish that its coefficient is a causal effect; causal interpretation needs separate justification.

The central principle is to make predictor choice answerable to the research question. A clear rationale connects the variable’s meaning and measurement quality to its scientific role, then explains how the chosen model determines the coefficient interpretation. That rationale is more informative than applying one collinearity threshold or selecting variables by statistical significance alone.

Check Your Understanding

For each question, explain the reasoning rather than naming a mechanical rule.

Two overlapping measures describe the same exposure, but one is more accurate and less available. What evidence should be considered before choosing one?
In a model, a coefficient for $X$ is interpreted holding $Z$ fixed. What changes in the interpretation if $Z$ is removed?
Why is a small p-value not, by itself, a sufficient reason to retain one redundant predictor over another?
A variable is scientifically important for adjustment but strongly related to the exposure of interest. Is overlap alone a reason to remove it? Explain.
Give two reasons why a predictor suitable for prediction might not be the preferred predictor for an analysis focused on a specific scientific association.