Why a Weak Marginal Signal Can Matter
A univariate screening rule evaluates each predictor separately, often retaining variables whose marginal correlation with the response is large enough. It is appealing because it is simple and can reduce the number of candidate predictors before subset selection. But a predictor’s separate association with the response is not the same as its contribution after another predictor is included. When predictors are correlated, one can help remove variation in the other that obscures the response relationship. This is the risk of univariate prescreening: a valuable predictor can look unimportant precisely because it is being judged alone.
The example below uses linear regression. Let \(y\) be a response and \(x_1,\ldots,x_p\) predictors. The usual model is \(y_i=\beta_0+\sum_{j=1}^{p}\beta_jx_{ij}+\varepsilon_i\). For ordinary least squares, the fitted coefficients minimize the residual sum of squares. For the familiar standard errors and exact \(t\)- and \(F\)-tests, assume errors have mean zero, common variance \(\sigma^2\), are independent, and are normally distributed; the design matrix must also have full column rank for unique coefficient estimates. The screening issue itself is about comparing separate and conditional associations and does not require normal errors.
“Suppressor” describes a pattern of association, not a guarantee that the predictor is causal or that it will improve performance on new data. It is also not limited to a predictor whose marginal correlation is exactly zero. The key warning is that a small marginal measure alone does not establish that a predictor is useless in a subset.
Two-Predictor Regression: The Conditional Calculation
Consider centered predictors \(x_1,x_2\) and a centered response \(y\), so the intercept has been accounted for. Define \(S_{jk}=x_j'x_k\) and \(S_{jy}=x_j'y\), where a prime denotes a transpose. In the regression of \(y\) on both predictors, the normal equations are
If \(S_{11}S_{22}-S_{12}^2>0\), the predictors are not perfectly collinear and the equations have a unique solution. Solving for the coefficient of \(x_2\) gives
The numerator depends on both \(x_2\)'s marginal cross-product with \(y\) and the part attributable to its relationship with \(x_1\). Thus, \(S_{2y}\) can be close to zero while \(\hat\beta_2\) is not. If the denominator is zero, the predictors are perfectly collinear, and separate coefficients cannot be identified from these data.
Why Adding a Predictor Can Reduce Error Sharply
There is a useful geometric calculation behind the conditional contribution. First fit \(y\) using the intercept and \(x_1\). Let \(r\) be the resulting residual vector, and let \(z\) be the residual vector from regressing \(x_2\) on the intercept and \(x_1\). Both vectors are orthogonal to the intercept and \(x_1\). Adding \(x_2\) is therefore equivalent to fitting the residuals \(r\) using \(z\).
For a proposed coefficient \(b\), the residual sum of squares after adding \(x_2\) is \((r-bz)'(r-bz)\).
Expanding gives \(r'r-2b z'r+b^2z'z\). When \(z'z>0\), the minimum occurs at \(b=z'r/(z'z)\).
Substituting the minimizing coefficient gives an RSS reduction of \((z'r)^2/(z'z)\), which is nonnegative and is positive whenever \(z'r\ne0\).
This derivation explains the screening risk: a marginal screen looks at \(x_2'y\), whereas the conditional improvement depends on the association between the residualized predictor \(z\) and residualized response \(r\). The two quantities need not be similar. This formula assumes \(x_2\) is not already an exact linear combination of the intercept and \(x_1\); if \(z'z=0\), adding it supplies no separately identifiable direction.
Worked Example: A Predictor With Zero Marginal Association
A hypothetical pricing analyst centers four observations on a response \(y\) and two candidate predictors. The columns are shown below. Calculate the separate and joint least-squares fits, and determine whether \(x_2\) is useful alongside \(x_1\). Use ordinary least squares with an intercept; the displayed columns already have mean zero. The method is to compare marginal fits with the two-predictor fit. A common mistake to avoid is discarding \(x_2\) solely because its one-predictor slope is zero.
| Observation | \(x_1\) | \(x_2\) | \(y\) |
|---|---|---|---|
| 1 | -1 | -0.2 | -1.18 |
| 2 | -1 | -1.4 | -0.26 |
| 3 | 1 | 0.2 | 2.18 |
| 4 | 1 | 1.4 | -0.74 |
Step 1: Calculate the required cross-products. From the four rows, \(x_1'x_1=1+1+1+1=4\), \(x_2'x_2=0.04+1.96+0.04+1.96=4\), and \(x_1'x_2=0.2+1.4+0.2+1.4=3.2\). Also, \(x_1'y=1.18+0.26+2.18-0.74=2.88\), while \(x_2'y=0.236+0.364+0.436-1.036=0\). Finally, \(y'y=1.3924+0.0676+4.7524+0.5476=6.76\).
Step 2: Fit each predictor separately. The slope using \(x_1\) alone is \(2.88/4=0.72\). Its fitted values are \((-0.72,-0.72,0.72,0.72)\), so the residuals are \((-0.46,0.46,1.46,-1.46)\). The RSS is \(0.2116+0.2116+2.1316+2.1316=4.6864\). The slope using \(x_2\) alone is \(0/4=0\), so this model’s RSS remains \(y'y=6.76\).
Step 3: Fit both predictors. The determinant of the cross-product matrix is \(4(4)-3.2^2=16-10.24=5.76>0\), so the coefficients are uniquely determined. The formula gives \(\hat\beta_2=(4(0)-3.2(2.88))/5.76=-9.216/5.76=-1.6\). The other coefficient is \(\hat\beta_1=(4(2.88)-3.2(0))/5.76=11.52/5.76=2\).
Step 4: Check the joint fit. The model states \(y=2x_1-1.6x_2+0.5w\), where \(w=(1,-1,1,-1)'\). This vector is orthogonal to both predictors: \(x_1'w=0\) and \(x_2'w=0\). Therefore, the joint-model residuals are \(0.5w=(0.5,-0.5,0.5,-0.5)\), and the RSS is \(4(0.5^2)=1\). The reduction from adding \(x_2\) to the \(x_1\)-only model is \(4.6864-1=3.6864\), a substantial improvement in this constructed data set.
The arithmetic confirms the pattern: \(x_2\) has zero marginal slope, but its joint coefficient is \(-1.6\). The predictors are correlated, since their cross-product is \(3.2\) rather than zero, and their joint specification reveals a relationship that the separate fit misses.
Reading the Same Risk From Correlations
For standardized variables, write \(r_{1y}\) and \(r_{2y}\) for the marginal correlations with the response and \(r_{12}\) for the predictor correlation. In a two-predictor regression, solving the standardized normal equations gives
This expression makes the contrast explicit. The numerator is not just the marginal correlation \(r_{2y}\); it subtracts the part associated with \(x_1\). The denominator is positive when \(|r_{12}|<1\). Near-perfect correlation makes the separate coefficients unstable, so a large conditional coefficient is not by itself evidence of a reliable or useful model.
Worked Example: A Small Correlation, a Large Conditional Contribution
Suppose a hypothetical claims study has standardized variables with \(r_{1y}=0.25\), \(r_{2y}=0.01\), and \(r_{12}=0.90\). Calculate the standardized coefficient of \(x_2\), the increase in \(R^2\) when it is added after \(x_1\), and the partial correlation of \(x_2\) with \(y\) controlling for \(x_1\). Use the two-predictor formulas. The common mistake is treating \(0.01\), the marginal correlation, as the conditional coefficient.
Step 1: Calculate the conditional coefficient. The numerator is \(0.01-(0.90)(0.25)=0.01-0.225=-0.215\). The denominator is \(1-0.90^2=1-0.81=0.19\). Thus \(\hat\beta_2=-0.215/0.19=-1.1316\), to four decimal places. The conditional coefficient is much larger in magnitude than the marginal correlation.
Step 2: Calculate the added \(R^2\). The response and predictor residuals after removing \(x_1\) have covariance \(-0.215\); the residual variance of \(x_2\) is \(0.19\). The one-predictor fit to these residuals therefore adds \(0.215^2/0.19=0.046225/0.19=0.2433\) to \(R^2\), to four decimal places. The \(x_1\)-only \(R^2\) is \(0.25^2=0.0625\), so the two-predictor \(R^2\) is \(0.0625+0.2432895=0.3058\), to four decimal places.
Step 3: Calculate the partial correlation. After removing \(x_1\), the residual variance of \(y\) is \(1-0.25^2=0.9375\). The partial correlation is \(-0.215/\sqrt{(0.19)(0.9375)}=-0.215/\sqrt{0.178125}\), which is approximately \(-0.5094\), to four decimal places. The small marginal correlation of \(0.01\) therefore coexists with a substantial conditional association in these hypothetical summaries.
Check. The added \(R^2\) is positive, and \(0.0625+0.2433=0.3058\). This is consistent with adding a predictor to an ordinary least-squares model on the same observations, which cannot increase the training RSS.
What a Screening Rule Can Exclude
Prescreening rules differ. Some rank predictors by absolute correlation; others use separate regression tests or an informal minimum-effect threshold. Each rule has the same structural limitation when it evaluates predictors one at a time: once a variable is excluded, a later subset search among the retained predictors cannot include it. Searching thoroughly within that reduced set does not undo the screening decision.
Worked Example: Screening Before Subset Search
Use the standardized correlation summary from the preceding example, and add a third predictor \(x_3\) with \(r_{3y}=0.30\), \(r_{13}=0\), and \(r_{23}=0\). A screening rule retains predictors with absolute marginal correlation at least \(0.05\). Compare the best training \(R^2\) available after screening with the \(R^2\) from the full three-predictor model. This is an illustrative correlation summary, not an empirical data claim. The method is to apply the rule first and then compare the attainable subsets; the trap is assuming the subsequent subset search can recover a screened-out variable.
Step 1: Apply the rule. The marginal correlations are \(0.25\) for \(x_1\), \(0.01\) for \(x_2\), and \(0.30\) for \(x_3\). Therefore, \(x_1\) and \(x_3\) are retained, while \(x_2\) is discarded.
Step 2: Find the best \(R^2\) among retained predictors. Since \(x_1\) and \(x_3\) are uncorrelated, their two-predictor \(R^2\) is \(0.25^2+0.30^2=0.0625+0.09=0.1525\). This exceeds the \(R^2\) for \(x_1\) alone (\(0.0625\)) and \(x_3\) alone (\(0.09\)); hence it is the best fit available after this screen.
Step 3: Compare with the full candidate set. Adding \(x_3\), which is uncorrelated with \(x_1\) and \(x_2\), adds \(0.30^2=0.09\) to the earlier two-predictor \(R^2\) of \(0.3058\). The full model has \(R^2=0.3057895+0.09=0.3958\), to four decimal places. The screened search cannot reach that model because it has removed \(x_2\). This comparison concerns training fit; it does not establish that the larger model predicts better on new observations.
When to Be Cautious
Univariate screening can be computationally convenient, but its results depend on the screening statistic, cutoff, and observed sample. A stringent cutoff can discard predictors with weak marginal associations; a permissive cutoff may retain many noise variables. Even a sound screen cannot guarantee recovery of the best subset unless its assumptions ensure that useful variables will also have adequate marginal signals.
- Do not equate marginal and conditional effects. The coefficient in a one-predictor model is not generally the coefficient in a multiple-predictor model.
- Check predictor relationships. Correlated predictors can share or mask response-related variation. Near-collinearity also makes conditional coefficient estimates unstable.
- Keep comparisons on the same observations. Different missing-data samples can make changes in fit look like predictor effects when they are partly due to changed data.
- Separate fit from evidence. A lower training RSS is an algebraic consequence of adding predictors; it does not alone show that the added variable improves prediction or explains a causal mechanism.
- Account for the selection process when interpreting results. The selected subset was chosen using the observed data, so ordinary coefficient summaries after selection may not reflect the uncertainty of the full search.
For subset selection aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus, the central conclusion is precise: a univariate screen restricts the candidate set using marginal evidence, while useful subset membership is a conditional question. When predictors may be correlated, screening solely on marginal association can remove a predictor that would have contributed meaningfully alongside another.
Check Your Understanding
For each problem, distinguish the marginal screen from the conditional calculation. Answers are given so you can check the arithmetic.
- A centered response and centered predictors have \(S_{11}=9\), \(S_{22}=4\), \(S_{12}=3\), \(S_{1y}=6\), and \(S_{2y}=2\). Calculate the two-predictor coefficient of \(x_2\). (Answer: \(0\); the numerator is \(9(2)-3(6)=0\).)
- Standardized variables have \(r_{1y}=0.40\), \(r_{2y}=0.05\), and \(r_{12}=0.50\). Calculate the standardized coefficient of \(x_2\) in the two-predictor model. (Answer: \((0.05-0.50(0.40))/(1-0.50^2)=-0.15/0.75=-0.20\).)
- In the preceding correlation setting, calculate the increase in \(R^2\) from adding \(x_2\) after \(x_1\). (Answer: \((0.05-0.50(0.40))^2/(1-0.50^2)=0.0225/0.75=0.03\).)
- A marginal-correlation screen retains variables with \(|r_{jy}|\ge0.10\). Three predictors have response correlations \(0.22\), \(-0.06\), and \(0.35\). Which predictor is at risk of exclusion even if it could contribute conditionally, and why? (Answer: the predictor with correlation \(-0.06\); the screen excludes it based on its marginal magnitude, so later subset search among retained predictors cannot include it.)
- For centered vectors, the residualized predictor \(z\) has \(z'z=5\) and \(z'r=-3\), where \(r\) is the response residual after fitting the current model. Calculate the added coefficient and the RSS reduction from adding this predictor. (Answer: the coefficient is \(-3/5=-0.6\), and the RSS reduction is \((-3)^2/5=1.8\).)