Two Different Screening Questions
A large predictor set raises two related but distinct questions. First, do some predictors contain much the same information as other predictors? Second, which predictors appear useful for predicting the outcome of interest? The first question can be investigated without using the outcome; the second cannot. Confusing these aims can lead to removing a useful predictor or retaining a redundant one simply because a screening method was asked to answer the wrong question.
For example, two measurements may be strongly related to each other but have different practical advantages: one could be cheaper to collect, more reliable, or easier to explain. Conversely, a predictor could vary substantially and not be redundant with anything else, yet show little apparent association with the chosen outcome. Predictor-only screening and outcome-informed screening describe these situations differently.
“Unsupervised” does not mean that no analysis is performed. It means that the outcome is not an input to the screening decision. Correlations or other association measures among predictors can identify possible redundancy. “Supervised” means the outcome informs the screen; an outcome-informed association measure is therefore specific to the response being studied.
What Each Screen Can and Cannot Tell You
| Question | Unsupervised screening | Supervised screening |
|---|---|---|
| Information used | Predictor values only | Predictor values and the outcome |
| Typical aim | Find groups of redundant or closely related predictors | Find predictors with apparent relevance to a specified outcome |
| Can identify | Predictor pairs or groups with similar information | Predictors with an observed outcome association under the chosen measure |
| Cannot establish on its own | Whether a predictor predicts the outcome | Whether a predictor is redundant with another predictor or adds value conditionally |
A strong predictor-to-predictor relationship is not evidence, by itself, that either predictor is useful for predicting \(Y\). Likewise, a strong marginal relationship between one predictor and \(Y\) does not establish that the predictor adds information after other predictors are considered. Screening works with a simplified view of the data; the final modeling question is more demanding.
For numeric predictors, a common measure of linear redundancy is the sample Pearson correlation. For paired observations \(x_{ij}\) and \(x_{ik}\), where \(i=1,\ldots,n\) indexes observations, define the sample means \(\bar{x}_j\) and \(\bar{x}_k\), and sample standard deviations \(s_j\) and \(s_k\). Provided both predictors have nonzero sample standard deviation, their sample correlation is
This statistic describes linear association between the two predictors. It does not use \(Y\), and a small or moderate correlation does not rule out every form of redundancy. The earlier material on rank correlation and mixed-type association measures covers alternatives when the predictor types or relationships call for them.
To illustrate outcome-informed screening without choosing a particular inferential test, consider a numeric predictor \(X_j\) and a binary response \(Y\), coded 0 or 1. If both response groups are present, a simple descriptive association score is the difference in sample group means:
This score uses the outcome to compare predictor values across outcome groups. It is not a universal screening statistic: its interpretation depends on the predictor's units, it does not quantify uncertainty, and it does not measure a conditional contribution in a multivariable model. It is useful here to make the distinction between predictor-to-predictor structure and predictor-to-outcome association explicit.
Why Correlation Detects Linear Redundancy
The correlation formula compares paired deviations from the two predictor means and divides by their total scales. If one predictor is an exact positive affine transformation of another, its centered deviations are just a positive multiple of the first predictor's deviations. The two predictors therefore have correlation 1, provided the first predictor varies. This is a precise example of why a predictor-only screen can detect redundant measurements without knowing anything about the target.
To verify the result, suppose \(X_2=a+bX_1\), with \(b>0\). Then \(\bar{X}_2=a+b\bar{X}_1\), so for each observation \(i\),
Substituting into the correlation formula makes its numerator \(b\sum_i(x_{i1}-\bar{x}_1)^2\), while its denominator is also \(b\sum_i(x_{i1}-\bar{x}_1)^2\). Their ratio is 1. If \(b<0\), the ratio is \(-1\). This derivation describes exact linear relationships; in applied data, a correlation close to 1 or \(-1\) indicates a strong linear association, not necessarily that the two variables are interchangeable for every purpose.
Worked Example: Redundant Measurements and a Binary Outcome
A hypothetical insurer records four policy-level predictors: a workload index \(X_1=(1,2,3,4)\) and a second version \(X_2=(12,14,16,18)\). The binary outcome is \(Y=(0,0,1,1)\). Use the predictor-only relationship to assess redundancy, then calculate the outcome-group mean difference for each predictor. State what each result does and does not show.
Method. First check whether the predictors are exact affine transformations. Then calculate the group means for the descriptive supervised score \(\Delta_j\). This separates redundancy from outcome association rather than treating them as the same question.
For each observation, \(X_2=10+2X_1\). Since the slope is positive and \(X_1\) varies, the derivation above gives \(r_{12}=1\). The two measurements contain exactly the same ordering and linear information in this dataset, although their numerical scales differ.
For \(X_1\), the mean among \(Y=0\) observations is \((1+2)/2=1.5\), and the mean among \(Y=1\) observations is \((3+4)/2=3.5\). Thus \(\Delta_1=3.5-1.5=2\).
For \(X_2\), the corresponding means are \((12+14)/2=13\) and \((16+18)/2=17\). Thus \(\Delta_2=17-13=4\). The difference is twice as large because \(X_2\) is \(10+2X_1\); it does not show that \(X_2\) carries twice as much information.
Conclusion and trap avoided. The unsupervised screen identifies exact redundancy. The supervised group-mean contrasts show that both measurements separate the two outcome groups in this small dataset. Choosing \(X_2\) just because its raw contrast is larger would confuse measurement scale with predictive importance. A practical choice between the redundant measurements could also consider reliability, collection cost, and interpretability.
Different Outcomes Can Change the Supervised Screen
A supervised screen is tied to its outcome. A predictor that is associated with one target may be weakly associated with another. Unsupervised relationships among the predictors do not change when the response changes, because the response was never used to calculate them. This makes predictor-only summaries useful for describing structure in a dataset, but insufficient for deciding which features matter for a particular prediction task.
Worked Example: A Predictor Can Be Redundant Without Being Outcome-Relevant
A hypothetical portfolio has eight observations. Let the numeric predictor \(A\) be \(1,2,3,4,5,6,7,8\), and let \(B=5+3A\). A binary outcome \(Y\) equals 1 for observations 2, 3, 5, and 8, and equals 0 for observations 1, 4, 6, and 7. A third predictor \(C\) equals 1 for the four observations with \(Y=1\), and 0 otherwise. Apply a predictor-only redundancy check and the descriptive group-mean screen.
Predictor-only result. Because \(B=5+3A\), the two predictors have correlation \(r_{AB}=1\). This flags them as exactly redundant in their linear information. The calculation does not use \(Y\), so it cannot tell whether \(A\) or \(B\) is useful for predicting the outcome.
Outcome-informed result for \(A\). The \(Y=1\) values of \(A\) are \(2,3,5,8\), with mean \((2+3+5+8)/4=18/4=4.5\). The \(Y=0\) values are \(1,4,6,7\), with mean \((1+4+6+7)/4=18/4=4.5\). Hence \(\Delta_A=4.5-4.5=0\).
For \(B=5+3A\), the \(Y=1\) mean is \(5+3(4.5)=18.5\), and the \(Y=0\) mean is also \(18.5\). Hence \(\Delta_B=0\). For \(C\), the mean is 1 in the \(Y=1\) group and 0 in the \(Y=0\) group, so \(\Delta_C=1-0=1\).
Conclusion and trap avoided. \(A\) and \(B\) are perfectly redundant with each other but have no difference in group means here. \(C\) has a clear group separation under this descriptive measure. A predictor-only screen cannot identify that outcome relationship; a supervised screen can. This result describes the given data and measure, not a guarantee of future predictive performance.
Worked Example: The Target Determines the Screening Question
Consider six hypothetical policies with predictors \(P=(1,2,3,4,5,6)\), \(Q=(1,2,2,4,5,7)\), and \(C=(0,0,1,0,1,0)\). Two possible binary targets are a claim indicator \(Y_1=(0,0,0,1,1,1)\) and a service-contact indicator \(Y_2=(1,0,1,0,1,0)\). Use group-mean differences to compare what an outcome-informed screen sees for each target. Also calculate the predictor correlation between \(P\) and \(Q\).
Predictor-only calculation. Both \(P\) and \(Q\) have mean 3.5. Their centered values are \((-2.5,-1.5,-0.5,0.5,1.5,2.5)\) and \((-2.5,-1.5,-1.5,0.5,1.5,3.5)\). The sum of cross-products is \(20.5\); the sums of squared deviations are \(17.5\) for \(P\) and \(25.5\) for \(Q\). Therefore,
This is a strong positive linear relationship between \(P\) and \(Q\). It is the same regardless of which target is being studied.
Screen for \(Y_1\). The \(Y_1=1\) group is observations 4, 5, and 6. Its means are \(5\) for \(P\), \((4+5+7)/3=16/3\) for \(Q\), and \((0+1+0)/3=1/3\) for \(C\). The \(Y_1=0\) group has means \(2\) for \(P\), \((1+2+2)/3=5/3\) for \(Q\), and \((0+0+1)/3=1/3\) for \(C\). Thus the differences are \(\Delta_P=3\), \(\Delta_Q=11/3\), and \(\Delta_C=0\).
Screen for \(Y_2\). The \(Y_2=1\) group is observations 1, 3, and 5; the \(Y_2=0\) group is observations 2, 4, and 6. The means for \(P\) are 3 and 4, giving \(\Delta_P=-1\). The means for \(Q\) are \(8/3\) and \(13/3\), giving \(\Delta_Q=-5/3\). The means for \(C\) are 1 and 0, giving \(\Delta_C=1\).
Conclusion and trap avoided. The same predictor-only redundancy signal appears for both tasks, while the outcome-group contrasts differ. In particular, \(C\) has no group-mean difference for \(Y_1\) but does for \(Y_2\). Do not treat a supervised screen for one response as a general ranking of predictors for every response. Also, the raw differences above are not directly comparable across predictors with different units; use a screening measure suited to the data and question when making broader comparisons.
Using the Two Screens Responsibly
The screens can be used together, but they answer separate questions. A practical sequence is to describe predictor redundancy first, then examine outcome relevance for the response of interest, and finally evaluate candidate predictors in an appropriate model. A high correlation can suggest that a group needs attention, but it does not dictate which member to retain. A strong outcome association can justify further investigation, but it does not establish a predictor's unique contribution after adjustment for other features.
Decide whether the immediate concern is overlapping predictor information, apparent relevance to a specified outcome, or both.
For redundancy, inspect relationships among predictors. For outcome relevance, use a measure that includes the target and suits the predictor and outcome types.
Consider units, measurement quality, interpretability, and the modeling objective before removing or retaining a feature.
A common exam trap is to call a predictor “irrelevant” because it is highly correlated with another predictor. That conclusion does not follow: the correlation only says the two predictors are linearly related to each other. Another trap is to call an uncorrelated predictor “useful” because it is not redundant. Lack of redundancy does not demonstrate an association with the outcome. Both claims require information the predictor-to-predictor correlation does not provide.
Supervised screening also needs to be interpreted in relation to its target and the data used to calculate it. A relationship observed in a sample is not proof that the same relationship will hold in new data. In predictive work, screening choices must be made in a way consistent with the model evaluation design; otherwise, reported performance may not reflect a genuinely new-data prediction task. A screen also cannot replace consideration of model structure, interactions, or subject-matter rationale.
Check Your Understanding
For each problem, identify what the screen can establish and calculate any requested quantity.
- A numeric predictor \(X=(1,2,3)\) and predictor \(Z=(4,7,10)\) are measured on the same three records. What is their sample correlation, and is its calculation supervised or unsupervised? (Answer: \(r_{XZ}=1\); unsupervised.)
- A numeric predictor has values \(2,4,6,8\); a binary outcome is 0 for the first two records and 1 for the last two. Calculate the difference in predictor means, \(\bar{x}_{Y=1}-\bar{x}_{Y=0}\). (Answer: \(6-3=3\).)
- Two predictors have correlation close to zero. Does this establish that either predicts the outcome? State why. (Answer: No; predictor-to-predictor correlation does not use the outcome.)
- For a binary target \(Y_A\), a predictor has group means 7 for \(Y_A=1\) and 5 for \(Y_A=0\). For a different target \(Y_B\), its group means are 4 and 6, respectively. Calculate both descriptive differences and state why the supervised results differ. (Answer: \(\Delta_A=2\), \(\Delta_B=-2\); the outcome groups differ by target.)
- Two predictors have correlation 0.99, and one has a stronger raw group-mean difference with a binary outcome. Can that raw difference alone establish that it is the better predictor to retain? (Answer: No; scale, measurement quality, redundancy, and performance in an appropriate model also matter.)