Why Propensity Model Fit Is Not Balance

Prediction and Balance Are Different Targets

A treatment model predicts treatment assignment from measured covariates. A weighting analysis uses the model’s predicted probabilities to reweight the observed treatment groups. These operations are connected, but they do not ask the same question. Predictive fit asks whether the model assigns probabilities that describe treatment assignment well; balance asks whether weighting has made the measured covariate distributions comparable across treatment groups.

The distinction matters because inverse-probability weights depend on the numerical values of predicted probabilities, not just on how well the model ranks people by treatment likelihood. A model may have a strong likelihood or discrimination measure and still give weights that leave important covariates imbalanced. Conversely, a model can balance covariates well even if it does not classify treatment assignment especially accurately.

The population balancing claim requires the actual conditional probability \(e(L)\), not merely a fitted probability that performs well on some prediction metric. For a causal interpretation of the resulting weighted comparison, additional assumptions are needed: consistency, conditional exchangeability given \(L\), and positivity. In particular, positivity requires \(0<e(L)<1\) on the covariate support relevant to the target population. For finite-variance weighted summaries, stronger bounds away from zero and one, together with suitable moment conditions, are often useful.

Why the True Propensity Score Balances Covariates

The key mathematical connection is an expectation identity. It applies to any integrable function of the measured covariates, not just to a single covariate mean. The result is about population weighted distributions; it does not promise exact balance in a finite sample.

Proof. By iterated expectation and the definition of \(e(L)\),

For the untreated group, \(E(1-A\mid L)=1-e(L)\), so the same calculation gives

Setting \(h(L)=1\) shows that each group’s expected unnormalized weight is one. Taking \(h(L)\) to be a covariate, an indicator of a covariate-defined set, or another integrable function shows that both weighted groups reproduce the covariate distribution in the full population. In data, estimated probabilities and random sampling make the empirical weighted distributions only approximate this identity. \(\square\)

This identity explains what correct propensity probabilities achieve. It does not say that maximizing a treatment-model likelihood guarantees correct probabilities, nor that any measure of predictive success verifies the identity’s conditions. A fitted model can miss the conditional treatment probabilities even when its overall predictions appear plausible.

Three Examples of Prediction and Balance Diverging

Why Ranking Metrics Cannot Certify Weighting Probabilities

The second mathematical point formalizes what the examples illustrate. AUC depends on pairwise score ordering, whereas inverse-probability weights depend on probability magnitudes. If \(q\) is a score and \(g\) is strictly increasing, replacing \(q\) by \(g(q)\) leaves every strict ordering unchanged. Ties are unchanged as well. Consequently, the AUC is unchanged.

Derivation. For an independent treated score \(Q_1\) and untreated score \(Q_0\), define

Strict increase gives \(Q_1>Q_0\) exactly when \(g(Q_1)>g(Q_0)\), and \(Q_1=Q_0\) exactly when \(g(Q_1)=g(Q_0)\). Both probabilities in the definition are unchanged. But the treated weight changes from \(1/q\) to \(1/g(q)\), and the untreated weight changes from \(1/(1-q)\) to \(1/\{1-g(q)\}\). Unless the transformation leaves the probabilities unchanged, these weights generally differ. Since weighted covariate summaries use those magnitudes, unchanged AUC does not imply unchanged balance. \(\square\)

A transformation of probabilities need not remain a calibrated probability model. That is precisely why a ranking score and a treatment probability should not be treated as interchangeable: weighting requires probability values that represent the conditional assignment mechanism, not merely a useful ordering.

What to Check and What Can Go Wrong

Treatment-model likelihood, classification accuracy, and AUC summarize prediction from different angles. None is a direct test of post-weighting balance. Balance diagnostics instead compare weighted covariate means, higher moments, or distributions between treatment groups, including across follow-up when the analysis is longitudinal. For a longitudinal treatment history, the weights involve products of conditional treatment probabilities at successive times; errors in those probabilities can compound. The population identity still motivates checking the resulting weighted histories directly.

Calibration itself needs careful interpretation. A prediction can be calibrated on average, as in the first example, while omitting covariate structure that matters for balance. Calibration across coarse groups also only establishes agreement at that resolution; residual differences within a group can remain. Direct balance assessment complements, rather than replaces, examination of the treatment model and the causal assumptions.

A common mistake is to reason, “The model predicts treatment well, so inverse weighting must have removed confounding.” That conclusion does not follow: prediction metrics may emphasize ranking or average fit, while weighting depends on accurate conditional probabilities and overlap. The correction is to inspect post-weighting covariate balance and weight behavior, and to treat poor balance as evidence that the fitted weighting procedure has not achieved its intended measured-covariate target. Good balance, in turn, does not prove that unmeasured confounding is absent or that positivity and exchangeability hold.

The principal limitation is that balance is evaluated for observed covariates and chosen summaries. Two weighted groups can match on means but differ in tails, nonlinear features, or interactions; distributional checks address some of these gaps but cannot establish equality of every feature from finite data. In longitudinal analyses, balance also depends on treatment history and time, so an overall summary can conceal imbalance at particular follow-up times.