Why Outcome-Informed Selection Needs Safeguards
High-dimensional claims and electronic health record data may contain far more candidate features than can be included in a conventional propensity model without careful selection. Ranking features partly by their association with the outcome can help prioritize prognostic information. But a strong observed outcome association is not, by itself, evidence that a feature is a confounder, nor does it establish that adjusting for the feature is causally appropriate.
The risk is not simply that an algorithm selects an unhelpful variable. When selection uses the same outcome data later used to estimate or evaluate an effect, random fluctuations can determine which features enter the analysis. The selected set may look unusually predictive in that sample, and ordinary uncertainty calculations may not account for the search that produced it. Worse, if outcome association draws attention to a collider, a post-treatment variable, or another inappropriate feature, selection can encourage a causally damaging adjustment.
Let \(A\) denote a binary treatment, \(Y\) the observed outcome, and \(X_1,\ldots,X_p\) candidate features. A propensity score model uses features to estimate treatment probabilities, but selecting \(X_j\) because it predicts \(Y\) does not show that \(X_j\) is a pre-treatment common cause of treatment and outcome. Valid causal estimation still requires appropriate design and assumptions, including consistency, conditional exchangeability given an adequate set of pre-treatment covariates, and positivity. Feature ranking cannot make those assumptions true.
How Screening Can Overfit
Suppose many candidate features have no real association with the outcome, but each is tested in a finite sample. Their observed associations will not all be zero. If the analyst retains the most outcome-associated features, the selected associations tend to be more extreme than their underlying associations. This is the winner's curse: selection favors features whose sample results include favorable random error.
The same issue affects apparent predictive performance. If features are chosen to predict an outcome and performance is assessed on those same observations, the assessment reuses information that helped define the model. It will generally be optimistic compared with performance on genuinely independent observations. In a causal workflow, reuse can also make the estimated treatment effect depend on outcome noise through the feature-selection step.
Worked Example: The Chance of a False Outcome Signal
Imagine a hypothetical analysis screening 40 features that are all truly unrelated to the outcome. For illustration, assume each feature produces an independent, correctly calibrated test under its null hypothesis, and the analyst flags a feature when its p-value is below 0.05. What is the chance that at least one feature is flagged?
For one null feature, the chance of not being flagged is \(1-0.05=0.95\). Under the stated independence assumption, the chance that none of the 40 features is flagged is \(0.95^{40}\). Therefore,
Thus, in this simplified setting, the chance of at least one false flag is about 87.1%. This is not the probability that any particular flagged feature is null, and the independence assumption may not hold for correlated health-record features. The calculation illustrates why a small p-value from a large screen should not be treated as if the feature had been specified in advance.
A related problem is inferential uncertainty. A confidence interval calculated after selecting features may be too narrow if its standard error treats the selected set as fixed and ignores the search. The exact effect depends on the selection rule and analysis, so there is no universal correction that makes every outcome-driven workflow valid. At minimum, analysts should report how features were selected and avoid presenting ordinary post-selection intervals as though the feature set had been prespecified.
Outcome Association Is Not Causal Relevance
Outcome ranking can favor a strong prognostic feature, but prognostic strength and confounding are different properties. A feature related to \(Y\) may improve precision or help control confounding under suitable causal conditions. Another feature can predict \(Y\) while being unrelated to treatment, and a feature associated with both treatment and outcome can still be inappropriate to adjust for if its causal role is unsuitable. The earlier material on prognostic covariates and instrumental features distinguishes these roles; outcome ranking alone cannot make that distinction.
In particular, a collider can be associated with the outcome and therefore rise in an outcome-based ranking. Conditioning on it may open a noncausal path between treatment and outcome. A feature measured after treatment can also reflect treatment effects rather than baseline confounding. These concerns make temporal ordering and causal structure essential screens before any ranking rule is applied.
Worked Example: An Outcome-Associated Collider
Consider a hypothetical population with independent binary variables \(A\) and \(U\), each equally likely to be 0 or 1. Let \(A\) be treatment, let \(U\) be an unmeasured cause of the outcome, and suppose the outcome is \(Y=U\), so treatment has no causal effect. A post-treatment feature \(C\) records \(A+U\), taking values 0, 1, or 2. Because \(C\) reflects both treatment and \(U\), it is a collider.
Across the whole population, \(A\) and \(U\) are independent, so the outcome mean is 0.5 in each treatment group: \(E(Y\mid A=0)=E(Y\mid A=1)=0.5\). There is no observed treatment-outcome difference before conditioning on \(C\).
Now consider the stratum \(C=1\). The only possible pairs are \((A,U)=(0,1)\) and \((1,0)\), each with probability one half within that stratum. Thus \(E(Y\mid A=0,C=1)=1\) and \(E(Y\mid A=1,C=1)=0\), producing a treatment difference of \(0-1=-1\) in this stratum even though treatment has no causal effect. Since \(C\) is related to \(Y=U\), an outcome-informed ranking might prioritize it; causal screening should reject it as a post-treatment collider rather than treating its predictive value as a reason to adjust.
A Safer Workflow
Safeguards should address both statistical overfitting and causal appropriateness. They are most effective when built into the analysis plan before inspecting outcome-driven rankings.
Specify the treatment time, outcome period, and lookback window. Exclude features measured after treatment begins or otherwise downstream of treatment. Use subject-matter knowledge and causal structure to identify features that could be adjusted for.
Record which outcome association measure, screening threshold, ranking procedure, and number of retained features will be used. A fixed rule does not eliminate overfitting, but it limits undisclosed analyst discretion and makes the analysis reproducible.
Use external data or a designated discovery sample for outcome-informed ranking, then assess prediction or estimate effects in independent data not used to choose the features. This reduces reuse of outcome noise; it does not establish that the selected variables are causally appropriate.
Check whether small changes in the sample or selection rule produce very different selected sets. Review leading features for timing, possible collider status, and substantive interpretation. Report instability as a limitation rather than hiding it behind one final list.
Check whether the resulting design has adequate covariate balance and treatment overlap, and examine whether conclusions are sensitive to reasonable alternative feature sets. Strong outcome prediction or a high predictive score is not evidence of causal identification.
Independent selection data have a cost: splitting a finite sample leaves fewer observations for estimation and may reduce precision. When no independent data are available, transparently prespecifying a defensible rule and acknowledging selection uncertainty is preferable to choosing features after repeated, unreported experimentation. Repeating the selection step in a resampling procedure can reveal instability, although it does not automatically solve bias or guarantee valid post-selection inference.
Comparison With Other Selection Strategies
| Strategy | What guides selection | Main advantage | Main caution |
|---|---|---|---|
| Outcome-informed ranking | Observed outcome association or prediction | Can prioritize prognostic features | Can overfit outcome noise and favor causally inappropriate features |
| Treatment-informed ranking | Observed treatment association | Can prioritize treatment predictors | May retain features that are not confounders, including problematic instruments |
| Prespecified causal adjustment | Design knowledge and causal structure | Targets variables relevant to confounding control | Depends on adequate subject-matter knowledge and measurement |
| Independent-data selection | A prespecified ranking rule in data separate from estimation | Reduces reuse of outcome noise in evaluation | Costs data and does not make selected variables causally valid |
These approaches are not interchangeable. Outcome-informed ranking can be one component of a high-dimensional propensity workflow, but causal eligibility should constrain the candidates first. A model that predicts treatment or outcome well can still yield a poor causal comparison if important confounding remains, overlap is inadequate, or inappropriate variables are adjusted for.
Common Pitfalls and Diagnostics
- “The feature is strongly associated with the outcome, so it must be a confounder.” Outcome association alone does not show that the feature causes treatment, precedes treatment, or belongs in the adjustment set. Check timing and causal role.
- “The selected feature has a small p-value, so the result is confirmed.” A p-value calculated after searching many candidates does not have the interpretation of a prespecified single test. Report the size of the search and how selection uncertainty is handled.
- “The same data can select and validate the features.” Reusing outcomes for both steps makes evaluation optimistic. Use independent data when feasible, or describe the analysis as exploratory and avoid claiming independent validation.
- “A stable prediction model guarantees a stable causal estimate.” Predictive stability does not ensure covariate balance, overlap, or identification. Compare causal estimates across defensible specifications and examine design diagnostics.
- “More outcome predictors always improve adjustment.” Extra features can consume information, increase instability, or introduce bias if they are post-treatment or colliders. Include a feature because its causal role is defensible, not solely because its score is high.
Useful warning signs include large changes in the selected set under modest changes to the sample or threshold, implausibly strong in-sample prediction relative to independent evaluation, features recorded after treatment initiation, and causal estimates that vary substantially across reasonable specifications. No single diagnostic proves that selection is safe; these signs indicate where additional scrutiny is needed.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- Under independent, calibrated null tests, what is the probability of at least one \(p<0.05\) among 20 screened features?
- Why does selecting features by outcome association and evaluating prediction on the same observations risk an optimistic assessment?
- A feature is measured after treatment starts and strongly predicts the outcome. Is that sufficient reason to include it in the propensity model? Explain.
- What does an independent discovery sample accomplish, and what problem does it not solve?
Solutions. (1) The probability is \(1-0.95^{20}\approx 0.6415\), or about 64.2%, under the stated assumptions. (2) The outcomes helped determine which features were selected, so random outcome patterns that made features look predictive are reused in the evaluation; performance is therefore likely to be optimistic. (3) No. Its post-treatment timing makes it potentially downstream of treatment, so its predictive strength does not establish that adjustment is appropriate. (4) It reduces reuse of the same outcome noise for selection and evaluation, but does not establish that selected features are causally valid, remove confounding, or ensure adequate overlap.