Why Automated Selection Needs Causal Checks
Automated feature selection can process thousands of claims or electronic health record variables, but it cannot determine by itself whether a feature is appropriate for causal adjustment. A feature may predict treatment very well and still be harmful to include. Another may be rare or weakly predictive yet represent an important confounder. The central task is therefore not to maximize prediction alone: it is to select information that is valid for the treatment decision and relevant to the causal question.
Let \(A\) indicate treatment, \(Y\) the outcome, and \(X\) the candidate covariates measured before treatment assignment. A propensity score is \(e(X)=P(A=1\mid X)\). In a high-dimensional analysis, an automated procedure may rank, screen, transform, or penalize many candidate features to estimate this probability. For a causal interpretation, the selected features must support the assumptions required by the chosen estimand and design; selection does not establish those assumptions.
For a baseline treatment comparison, a common identification framework requires consistency, conditional exchangeability \(Y(a)\perp A\mid X\), and positivity \(0<P(A=1\mid X)<1\) in the target population, for treatment levels \(a\). These assumptions are not verified by a good prediction score. They clarify why covariate timing, causal role, overlap, and the target population must be checked alongside predictive performance.
Post-Treatment Features and Outcome Leakage
A baseline propensity model should use covariates available before the treatment decision. A feature recorded after treatment begins can be a mediator, a consequence of treatment, or a collider; adjusting for it can change the estimand or introduce bias. The date a code appears in a database is not always the date the underlying condition began, so timestamps, encounter definitions, and lookback rules matter. A post-treatment feature is not made safe merely because it improves treatment prediction or balance.
Worked Example: A Lab Value Recorded After Treatment Starts
Suppose a hypothetical study compares starting Drug A with starting Drug B at day 0 and measures hospitalization during the next 90 days. A selection algorithm retains a laboratory value measured on day 14 because it predicts which drug a patient is taking and predicts hospitalization. The value is available for 70% of patients, and among those patients its mean is 8.1 units for Drug A and 6.4 units for Drug B.
The difference is \(8.1-6.4=1.7\) units, but that arithmetic does not make the value a baseline confounder. It was recorded after treatment initiation, so treatment may have changed the value, and early illness may have influenced both the measurement and later hospitalization. Before including it, the analyst should inspect the clinical timeline and data-generation process. With the stated timing, it is ineligible for an ordinary baseline propensity model; its strong association with treatment does not override that fact.
A defensible remedy is to restrict baseline features to measurements available before day 0 and repeat extraction with a clearly defined pre-treatment lookback window. If the scientific question concerns changing treatment and covariates over follow-up, that requires a longitudinal design rather than inserting a later measurement into a baseline score.
Outcome leakage occurs when a feature, label, or selection step contains information from the outcome or from a time after outcome ascertainment. For example, a code assigned during a hospitalization that defines \(Y\) must not be used as a baseline treatment predictor. This differs from selecting a genuine pre-treatment prognostic covariate: its outcome association may be useful, but the feature itself must be measured before treatment and selected without contaminating the analysis with future information. Feature construction, imputation, screening, and tuning should respect the time ordering; when cross-fitting is used, selection and fitting must also occur within the appropriate training folds.
Duplicated Codes and Excessive Sparsity
Claims and clinical vocabularies often contain overlapping, nested, or near-duplicate codes. Redundancy can make rankings unstable, let one clinical construct dominate a distance or score, and make coefficient interpretation misleading. Exact duplicate predictors create a particularly clear identification problem in a regression model.
Worked Example: Two Codes for the Same Recorded Feature
Imagine two hypothetical binary codes \(D_1\) and \(D_2\) that are identical for every person in an analysis: both equal 1 for 30 people and 0 for the remaining 170. In a logistic propensity model, their contribution to the linear predictor is \(\beta_1D_1+\beta_2D_2\). Since \(D_1=D_2=D\), this equals \((\beta_1+\beta_2)D\).
For instance, coefficients \(\beta_1=0.8,\ \beta_2=0.2\) and \(\beta_1=0.5,\ \beta_2=0.5\) both contribute \(1.0D\). The data cannot distinguish these coefficient allocations: only their sum affects fitted probabilities. A penalized procedure may select one code, divide the contribution, or change the allocation when small data changes occur. None of those outcomes proves that one code is the uniquely important clinical predictor.
Check code definitions, cross-tabulations, and whether one feature is deterministically or nearly duplicated by another. A remedy may be to use a clinically justified composite or grouped feature, then assess balance for the underlying construct. Do not collapse codes automatically: two similar codes can differ in severity, timing, or meaning, and deleting one without checking may discard useful confounding information.
Excessive sparsity creates a different problem. A rare indicator can perfectly or nearly perfectly predict treatment in a sample. In an unpenalized logistic model, perfect prediction can send a coefficient toward infinity, while near-separation can produce very large, unstable estimates. Yet simply excluding every rare feature is also risky: a sparse feature may mark a clinically important condition associated with both treatment and outcome.
Worked Example: A Rare Code That Separates Treatment Groups
In a hypothetical dataset of 1,000 patients, a code appears in 2 of 200 Drug A initiators and in 0 of 800 Drug B initiators. The observed prevalence is \(2/200=0.010\), or 1.0%, in the Drug A group and \(0/800=0\) in the Drug B group. This zero cell means the code perfectly distinguishes its two observed cases from the comparison group.
A logistic propensity model with this indicator can have a coefficient that diverges as the fitted probability for the two coded patients approaches 1. The observed pattern is a warning about sparse-data instability, not evidence that the code is a strong causal confounder. Inspect the treatment-by-feature table, coefficient stability, convergence, and predicted propensity scores. Then investigate whether the code is reliable and clinically coherent. Possible responses include justified aggregation, carefully chosen penalization, or a prespecified exclusion with sensitivity analysis. Penalization can stabilize estimation, but it cannot create treatment overlap where none exists or justify discarding a genuine confounder.
Distinguish a sparse feature from a positivity problem. Sparse-data separation concerns estimation from limited observed cells; positivity concerns whether treatment options occur with positive probability within relevant covariate patterns in the target population. Examine propensity-score overlap, extreme weights, effective sample size, and balance after the planned design. A computationally stable fit does not resolve poor overlap.
Predictive Instruments: Strong Treatment Prediction Is Not Enough
A pre-treatment variable that strongly predicts treatment but has no direct effect on the outcome except through treatment may be instrument-like. Such a variable is not automatically invalid as a covariate, but its predictive strength alone is not a reason to include it. If important unmeasured confounding remains, conditioning on a strong instrument can amplify bias in some settings; it can also worsen overlap and increase weight or matching instability. Conversely, if exchangeability holds given the adjusted covariates and positivity is adequate, including a valid pre-treatment predictor does not by itself create asymptotic confounding bias. The concern is conditional on the data-generating structure and design.
A short linear calculation illustrates the possible bias amplification. Suppose the true outcome model is \(Y=\tau A+\gamma U+\varepsilon\), where \(U\) is an unmeasured confounder. Treatment follows \(A=\pi Z+\delta U+\nu\), where \(Z\) is a pre-treatment instrument-like predictor independent of \(U,\nu,\varepsilon\). Assume all variables are centered, the errors are mutually independent, and \(\operatorname{Var}(U)=\operatorname{Var}(Z)=\operatorname{Var}(\nu)=1\). The model has no direct \(Z\) term in the outcome equation.
The denominator follows from \(\operatorname{Var}(A)=\pi^2+\delta^2+1\). If we instead regress \(Y\) on both \(A\) and \(Z\), the Frisch–Waugh–Lovell argument first removes the linear contribution of \(Z\). The residual treatment is \(\delta U+\nu\); its covariance with the residual outcome, apart from the causal term \(\tau(\delta U+\nu)\), is \(\gamma\delta\). Its variance is \(\delta^2+1\). Thus the adjusted coefficient has population bias \(\gamma\delta/(\delta^2+1)\).
Worked Example: How an Instrument-Like Predictor Can Amplify Bias
Use the preceding hypothetical model with \(\pi=2,\ \delta=1,\ \gamma=1\). Without adjusting for \(Z\), the bias is \(1/(2^2+1^2+1)=1/6\), approximately 0.167 outcome units per treatment unit. Adjusting for \(Z\) gives bias \(1/(1^2+1)=0.500\) outcome units per treatment unit. The adjustment has increased the absolute bias in this example because it removes treatment variation associated with \(Z\) while the unmeasured confounding through \(U\) remains.
This calculation is not a universal rule against treatment predictors. It demonstrates why a feature selected only for strong prediction can be a poor choice when unmeasured confounding is plausible. Investigate its clinical role, its relation to the outcome, overlap and weight diagnostics, and sensitivity to its inclusion. Do not label a feature an instrument merely because it strongly predicts treatment; the causal claim that it has no outcome path except through treatment requires substantive justification.
A Practical Diagnostic Sequence
No single statistic diagnoses every selection failure. A useful review links each warning to a distinct check and response, rather than treating all problems as a reason to use a more flexible prediction algorithm.
Verify that every candidate feature is defined using information available before treatment, and trace any feature closely related to the outcome to its source and measurement date.
Review code crosswalks, correlations or duplicate patterns, prevalence tables by treatment, zero cells, convergence, and coefficient stability. Group features only when their clinical meaning supports doing so.
Ask whether each selected variable is a confounder, prognostic feature, mediator, collider, or instrument-like predictor. Reassess overlap, extreme weights, effective sample size, and balance after the planned matching or weighting.
Compare a clinically constrained feature set with the automated set, and conduct targeted sensitivity analyses for questionable features, coding rules, and sparsity thresholds. Keep the target estimand and analysis population explicit.
Compared with manual prespecification, automated selection can scale to many candidate features but is sensitive to coding, thresholds, and predictive criteria. Compared with a flexible learner, a sparse penalized model may be easier to inspect but can still select causally inappropriate variables. Neither approach substitutes for subject-matter review. Diagnostics such as balance across covariates and blocks, overlap, and effective sample size evaluate different aspects of the design; a favorable result on one does not repair a failure on another.
Common Mistakes and Limits
- “It predicts treatment, so it belongs in the score.” Prediction is not a causal inclusion rule. Check timing and causal role, especially for post-treatment and instrument-like features.
- “The algorithm selected it, so it must be a confounder.” Selection reflects the algorithm, candidate set, and tuning choices; it does not establish that a feature causes treatment and outcome.
- “Rare codes should all be removed.” Sparsity can destabilize estimation, but automatic deletion may omit an important condition. Investigate meaning and use justified grouping or sensitivity analysis.
- “A converged model has adequate overlap.” Convergence is computational; positivity and practical overlap require separate assessment.
- “Two similar codes are harmless duplicates.” Redundancy can destabilize rankings or overweight a construct, but codes may differ in clinically meaningful ways.
Automated selection also has limits that no diagnostic fully removes. Unmeasured confounding cannot be ruled out by balance on observed features; data errors can make a correctly timed code misleading; and changing the selected set can change the population represented after matching or weighting. Remedies should therefore be tied to the suspected failure and reported with sensitivity analyses, not presented as guarantees of unbiased effects.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A feature recorded two weeks after treatment starts sharply improves propensity prediction. What is the first concern, and what should be checked?
- Two binary codes are identical for every patient. Why might their fitted coefficients or selection rankings be unstable?
- A rare code occurs in treated patients but in no controls. Does a penalized fit establish adequate positivity? Explain.
- Why might adjusting for a strong instrument-like treatment predictor increase bias when unmeasured confounding remains?
Solutions. (1) The feature is post-treatment for a baseline design and may be affected by treatment or later illness. Check measurement and clinical timing; ordinarily restrict the baseline score to pre-treatment information. (2) Only the sum of their coefficients determines the linear predictor, so multiple coefficient allocations fit identically; small changes or penalties may alter rankings. (3) No. Penalization may stabilize estimation, but it does not create treatment observations in the missing covariate pattern; assess overlap and the target population. (4) Conditioning on the predictor can remove treatment variation associated with that predictor while leaving the unmeasured confounding pathway, increasing the confounding bias in some data-generating settings.