Conditional Value Beyond a Correlated Predictor

Why a Predictor’s Marginal Association Can Mislead

A screening measure usually looks at one predictor at a time. That is useful for an initial scan, but it does not answer a different question: does this predictor add information once another, correlated predictor is already available? A predictor can have a modest marginal association with the outcome and still add useful information after adjustment. Conversely, a sizeable marginal association can largely reflect information already carried by another predictor.

This tutorial develops that comparison for an additive linear model with two predictors. The calculations are aligned with the SOA Exam PA (Predictive Analytics) syllabus. They help distinguish a univariate association from an incremental contribution; they do not establish causality or guarantee improved predictions on new data.

Two-Predictor Linear Model and Notation

Let \(Y\), \(X_1\), and \(X_2\) be quantitative variables measured on the same observations. Consider an ordinary least-squares regression with an intercept and two additive predictors. For the formulas below, suppose the variables have been centered and scaled to have variance 1. Write \(r_{Y1}=\operatorname{Corr}(Y,X_1)\), \(r_{Y2}=\operatorname{Corr}(Y,X_2)\), and \(r_{12}=\operatorname{Corr}(X_1,X_2)\).

The standardization makes the coefficients and variance calculations especially direct. The results concern a linear projection: they describe the best-fitting additive linear relationship under squared-error loss. The algebra does not require normally distributed variables. However, inference such as conventional standard errors and tests requires its own assumptions, and the formulas are not a substitute for checking model fit or predictive performance.

In the model containing \(X_1\) alone, the standardized slope is \(r_{Y1}\), and \(R^2=r_{Y1}^2\). In the two-predictor model, the slope for \(X_2\), conditional on \(X_1\), and the added \(R^2\) are:

The full model’s \(R^2\) is \(r_{Y1}^2+\Delta R^2\). The quantity \(r_{Y2}-r_{12}r_{Y1}\) is an adjusted association numerator: it discounts the part of \(X_2\)’s relationship with \(Y\) aligned with \(X_1\). It is not, by itself, a measure of the increase in \(R^2\). The denominator matters too.

Why the Increment Formula Holds

The result follows by removing from \(X_2\) and \(Y\) the parts linearly associated with \(X_1\), then measuring how well the remaining part of \(X_2\) explains the remaining part of \(Y\). With standardized variables, define the residuals from their separate regressions on \(X_1\):

Because \(\operatorname{Var}(X_1)=1\), these residuals have covariance and variance:

The least-squares slope when regressing \(Y^*\) on \(Z_2\) is their covariance divided by the predictor’s variance. This is the conditional slope for \(X_2\) in the full model. The reduction in residual variance from adding \(Z_2\) is the squared covariance divided by its variance. Since \(Y\) has variance 1, that reduction is exactly the added \(R^2\). Thus:

This residualization argument also explains the interpretation: \(X_2\) adds information to the linear model to the extent that its part not linearly predictable from \(X_1\) is associated with the part of \(Y\) not linearly predictable from \(X_1\).

Weak Marginal Association, Meaningful Increment

The marginal correlation \(r_{Y2}\) combines the relationship \(X_2\) has with \(Y\) directly and the association induced by its correlation with \(X_1\). The two components can reinforce or offset each other. When they offset, \(X_2\)’s marginal correlation may look small even though the part of \(X_2\) left after accounting for \(X_1\) is informative about the outcome.

Collinearity Makes the Denominator Essential

The formula for \(\Delta R^2\) has denominator \(1-r_{12}^2\), the variance remaining in \(X_2\) after its linear relationship with \(X_1\) is removed. If \(X_1\) and \(X_2\) are very highly correlated, this residual variance is small. Consequently, a small adjusted numerator does not automatically imply a small increase in \(R^2\); the numerator must be assessed relative to the denominator.

Using Incremental Value Responsibly

For a screening decision, a marginal screen and a conditional comparison answer different questions. A marginal screen asks whether a predictor is associated with the outcome by itself. The two-predictor calculation asks what happens when that predictor is added to a particular baseline model. A predictor can be valuable relative to one baseline and redundant relative to another.

A positive \(\Delta R^2\) is an in-sample fact for the data and model used to calculate it. It does not alone establish that the predictor will improve predictions for new observations. Correlation estimates can be noisy, especially when the sample is small or predictors are highly correlated. Screening many predictors also raises selection concerns; results chosen from the same data can appear stronger than their performance on new data. Assessing generalization requires appropriate validation, and the interpretation should respect the study design and the possibility of data-quality or measurement issues.

The formulas also have boundaries. They apply to the stated standardized, additive, two-predictor least-squares comparison with an intercept and common observations. A small or zero conditional contribution in this model does not rule out value under a different model structure. For example, nonlinear relationships or interactions require methods that represent those structures; they are not measured by this two-predictor additive formula.