Why Compare Models by Their Causal Diagnostics?
In a high-dimensional study, several methods can produce markedly different propensity scores from the same baseline data. A model may predict treatment assignment very well yet leave important covariates imbalanced after weighting, or it may produce extreme scores that make a weighted estimate depend heavily on a few observations. The central comparison is therefore not simply which model best predicts treatment. It is which candidate produces credible balance for the target analysis while retaining adequate overlap and reasonably stable weights.
This comparison does not establish that an observational estimate is causal. The causal interpretation still requires consistency, conditional exchangeability given the measured pre-treatment covariates \(X\), and positivity: each covariate pattern in the target population must have a nonzero probability of receiving either treatment. Covariates must be measured before treatment and selected in a way that avoids conditioning on consequences of treatment. These assumptions are substantive, not consequences of fitting a particular algorithm.
Three Ways to Construct Candidate Scores
The high-dimensional propensity score (HDPS) approach starts from coded features in dimensions of an administrative database, such as diagnoses, procedures, and medications. It generates candidate indicators and ranks them using their prevalence and associations with treatment and, in conventional implementations, outcome-related information. A selected set of indicators is then used in a propensity model. The ranking is a way to prioritize features, not a guarantee that every selected feature is a confounder or that all important confounders were found. For a fair comparison, the feature-generation and selection rules should be specified without searching for a favorable treatment-effect estimate.
Lasso logistic regression estimates treatment probabilities with a logistic model while penalizing the absolute values of covariate coefficients. In a common formulation, \(\lambda\) controls the penalty, and the intercept is not penalized:
The penalty can set some coefficients to zero, making a model with many candidate features estimable and comparatively sparse. The resulting score still depends on the specified feature representation: nonlinearities or interactions not represented in the design matrix will not be learned automatically. Lasso tuning for causal balance can therefore differ from tuning solely for predictive loss.
Flexible treatment-prediction learners include approaches such as random forests, boosted trees, and ensembles. Depending on the learner and its settings, these can represent nonlinear relationships and interactions without specifying each term in advance. Flexibility is not an automatic advantage: a learner can overfit, yield poorly calibrated probabilities, or generate values close to zero or one. Cross-fitted or otherwise out-of-sample predictions help prevent evaluating a score on the same observations used to fit it; all candidates should be evaluated comparably.
What to Compare
Use the same eligible population, treatment definition, baseline covariates, target estimand, and evaluation folds for each candidate. For weighted mean balance, compare the treated and control weighted means of each covariate. A commonly used standardized mean difference (SMD) divides their difference by a fixed, pre-weighting standard deviation. Fixing the denominator before weighting makes changes in the SMD interpretable across candidate models.
For a binary covariate, one can use the original pooled standard deviation of the binary variable as \(s_{j,\mathrm{pre}}\). Inspect the absolute SMDs across the full feature set, including important feature blocks, rather than reporting only an average. A threshold such as 0.1 is a practical flag, not a proof that confounding is absent or a universal pass-fail rule. Balance checks should also reflect the covariate forms that matter: balancing a linear term does not necessarily balance a nonlinear pattern or interaction.
Overlap concerns whether treated and control units have comparable estimated treatment probabilities in the population relevant to the estimand. Plot or summarize the score distributions by treatment group, and inspect their tails. Under the ATE weights above, a treated unit with a very small \(\hat e_i\), or a control unit with \(\hat e_i\) near one, receives a large weight. Large weights increase sensitivity to a few observations and can make estimates imprecise.
The effective sample size (ESS) is the size of an equally weighted sample that would have the same weight concentration by this measure. If every weight is equal, ESS equals \(n\); concentration in a few large weights lowers it. Report it alongside weight quantiles and maxima, not as a replacement for examining which units carry the weights. ESS is a diagnostic of weight concentration, not a measure of bias or a guarantee of valid inference.
Treatment discrimination, often summarized by the area under the receiver operating characteristic curve (AUC), measures how well a score ranks treated above control units. AUC does not directly assess covariate balance after weighting, probability calibration, overlap, or weight stability. A high AUC may reflect strong treatment sorting that makes overlap worse. It should be reported as a predictive diagnostic, not used alone to choose a causal propensity model.
Worked Example: A Higher AUC Does Not Guarantee a Better Weighted Analysis
Suppose three candidate models are evaluated on the same hypothetical cohort of 1,000 people, with cross-fitted scores and ATE weights. The table gives illustrative diagnostic summaries; the maximum absolute SMD is calculated over all measured baseline features.
| Candidate | AUC | Maximum absolute SMD | Features with absolute SMD > 0.1 | Largest weight | ESS |
|---|---|---|---|---|---|
| HDPS-ranked logistic model | 0.71 | 0.08 | 0 | 8 | 690 |
| Lasso logistic model | 0.78 | 0.14 | 7 | 31 | 410 |
| Flexible learner | 0.91 | 0.23 | 26 | 164 | 86 |
The flexible learner has the highest AUC, but its maximum imbalance is larger, more features exceed the illustrative 0.1 flag, its largest weight is higher, and its ESS is lower. The HDPS-ranked candidate has the best values in these listed balance and stability summaries, despite having the lowest AUC. For the flexible learner, the ESS is \(86/1{,}000=0.086\), or 8.6% of the original cohort size by this crude comparison. This does not mean that exactly 86 people determine the estimate, but it signals substantial weight concentration.
These summaries do not, by themselves, establish that HDPS is the correct choice. The analyst should inspect which features remain imbalanced, whether those features are substantively important, and the score distributions in each treatment group. The comparison also depends on the chosen estimand and the credibility of the measured confounders. The example supports rejecting “choose the highest AUC” as a selection rule; it does not prove that any listed model removes unmeasured confounding.
Balance and Weight Calculations
The next example calculates an SMD directly. This makes clear that a model's evaluation requires comparing weighted covariate distributions, rather than inferring balance from a treatment-prediction summary.
Worked Example: Calculating a Weighted Standardized Mean Difference
In a hypothetical study, a binary baseline risk indicator has weighted prevalence 0.42 among treated people and 0.07 among controls under one candidate score. Its pre-weighting pooled standard deviation is 0.70. The weighted difference in prevalence is \(0.42-0.07=0.35\). Therefore,
The absolute SMD is 0.50, which indicates a substantial remaining difference on this standardized scale and is well above the commonly used 0.1 diagnostic flag. The model should not be described as having balanced this feature merely because it has a high AUC. The analyst should investigate why the imbalance remains, check the feature's coding and timing, and compare alternative prespecified models or feature representations. If the imbalance is concentrated in a sparse feature, its frequency and weighted contribution also need inspection.
Worked Example: One Extreme Score Can Collapse Effective Sample Size
Consider four hypothetical people with treatment indicators \(A=(1,1,0,0)\). Candidate model 1 predicts propensities \((0.80,0.70,0.30,0.20)\). The ATE weights are \(1/\hat e_i\) for treated people and \(1/(1-\hat e_i)\) for controls, giving
Thus \(\sum_i w_i=5.358\) and \(\sum_i w_i^2 \approx 7.207\), so
Now suppose candidate model 2 predicts \((0.80,0.02,0.30,0.20)\), with the same treatment indicators. Its weights are approximately \((1.25,50,1.429,1.25)\). The sum is 53.929 and the sum of squared weights is 2,505.166, so
The second treated person is assigned a weight of 50 because the model predicts a treatment probability of only 0.02 for someone who was treated. The low ESS signals that this weighted comparison is highly concentrated. Such a sample pattern is a warning about practical overlap and model fit; it does not by itself prove a population-level violation of positivity. The analyst should inspect the underlying covariate pattern, treatment-group score distributions, and sensitivity of the target estimand before deciding what analysis is defensible.
A Disciplined Comparison Workflow
A defensible comparison treats candidate models as alternatives for achieving the target design, rather than as a contest for the largest predictive score. Use the same diagnostics for each method, and decide in advance how balance, overlap, and weight stability will inform the choice. If candidate models all fail important diagnostics, selecting the least poor one does not make the design adequate.
Define treatment, the target estimand, the eligible population, and the pre-treatment covariates. Apply the same rules to every candidate.
Fit HDPS, lasso, and flexible candidates with an evaluation strategy that avoids assessing in-sample predictions as if they were out-of-sample. Apply comparable tuning discipline.
Calculate covariate-level SMDs and inspect imbalanced features and blocks. Examine nonlinearities or interactions when they are relevant to the design.
Compare score distributions by treatment, inspect extreme scores and weight tails, and calculate ESS. Identify which observations receive the largest weights.
Prefer a candidate that achieves substantively adequate balance without unacceptable weight concentration, subject to the causal assumptions. If none does, report the limitation and consider whether the estimand or design must change.
The rationale is direct: weighting aims to make measured covariate distributions comparable for the target contrast. SMDs assess that objective on observed features; overlap and weight summaries assess whether the comparison is supported by the observed treatment assignment patterns. Predictive discrimination answers a different question: how well the model separates treatment groups. Consequently, good discrimination is neither necessary nor sufficient for good weighted balance.
Common Pitfalls and Limitations
- Choosing the largest AUC. This rewards treatment ranking, not necessarily balance or stable weights. Compare weighted covariate distributions and the score and weight tails as well.
- Reporting only an average SMD. A small average can hide a few severely imbalanced or clinically important features. Show the distribution of absolute SMDs and identify the largest residual imbalances.
- Treating a diagnostic cutoff as a theorem. An SMD below 0.1 is a useful convention, not proof of exchangeability, absence of bias, or adequate balance for every feature and estimand.
- Assuming flexibility always improves the analysis. Flexible learners can capture complex assignment patterns, but can also overfit and produce extreme scores. Cross-fitting addresses a source of overfitting in evaluation; it does not create overlap or guarantee causal validity.
- Using outcome results to pick a favorable score. Searching candidate models for the largest or most significant estimated effect can make the result data-dependent and undermine interpretation. Base design choices on prespecified covariate, overlap, and weight diagnostics, not a preferred treatment effect.
- Ignoring the population changed by poor overlap. Restricting to a region of common support or trimming weights can alter the target population or estimand. State what population the revised analysis addresses rather than presenting it as an unchanged ATE.
HDPS, lasso, and flexible learners are not universally ordered from inferior to superior. HDPS offers a structured way to prioritize database features; lasso offers sparse regularized modeling within a specified feature representation; flexible learners can represent complex treatment-assignment patterns. Their relative performance depends on the data, candidate features, tuning, and target population. Balance, overlap, and weight stability provide complementary evidence for choosing among them, but none can verify that all confounders were measured or that the causal assumptions hold.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A flexible learner has AUC 0.90, while a lasso model has AUC 0.76. The flexible learner also has more covariates with large absolute SMDs and a much lower ESS. Is the flexible learner automatically preferable? Why?
- A weighted prevalence is 0.30 in treated people and 0.20 in controls. The fixed pre-weighting standard deviation is 0.50. Calculate the SMD.
- For weights \(1,1,1,1\), calculate ESS. What does the result illustrate?
- Two candidate scores have similar balance, but one has several estimated propensities very close to zero or one. Name two additional diagnostics to inspect before choosing it.
Solutions. (1) No. AUC describes treatment ranking, not causal balance or stability; the worse balance and lower ESS are warnings that require investigation. (2) \(\operatorname{SMD}=(0.30-0.20)/0.50=0.20\), which exceeds the commonly used 0.1 flag. (3) \(\operatorname{ESS}=(1+1+1+1)^2/(1^2+1^2+1^2+1^2)=16/4=4\), equal to the sample size because all weights are equal. (4) Inspect treatment-specific score distributions and weight quantiles or maxima; also calculate ESS and examine which observations carry extreme weights.