Why Fitted Scores Do Not Guarantee Exact Sample Balance

Why a Propensity Score Is Not a Balance Guarantee in Every Data Set

The propensity score has a powerful population property: if units are grouped by their true probability of receiving treatment given their covariates, treatment assignment is independent of those covariates within each exact score value. In practice, however, the score is estimated from a finite sample. The fitted values may not equal the true probabilities, and even a well-fitted model does not force treated and control units to have identical covariate distributions. The difference between a population theorem and a finite-sample guarantee is important whenever balance is assessed or a score-based procedure is planned.

Let \(A\) indicate treatment, with \(A=1\) for treated and \(A=0\) for controls, and let \(X\) denote the pretreatment covariates. The true propensity score is \(e(X)=P(A=1\mid X)\). This tutorial focuses on what the balancing property says, why fitting a model does not make it exact in the sample, and how model form and grouping scores affect the discrepancy.

What the Population Theorem Says

The result is a statement about the true conditional treatment probability, not about the quality of a particular fitted model. For any set \(B\) of covariate values and any score value \(t\) with \(0<t<1\), conditioning on \(e(X)=t\) gives

The same calculation for controls gives

Dividing the first expression by \(P(A=1\mid e(X)=t)=t\), and the second by \(P(A=0\mid e(X)=t)=1-t\), shows that the conditional probability of \(X\in B\) is the same in each treatment group. Applying this to all sets \(B\) establishes equality of the covariate distributions. For continuous scores, the statement is understood using conditional distributions rather than literal positive-probability point masses.

The result concerns the true score in the population. For causal comparisons, one also needs the relevant causal identification assumptions, such as conditional exchangeability given \(X\) and positivity. The balancing theorem itself does not establish those assumptions, and it does not turn an estimated score into a true one.

Why Estimated Scores Do Not Enforce the Theorem

In a data set, the true \(e(X_i)\) is unknown. An analyst fits a model and obtains \(\hat e_i\), an estimated score for unit \(i\). The theorem does not say that \(A\) and \(X\) are independent conditional on \(\hat e_i\). Estimation error, model form, and finite-sample variation all matter. Even a score model that captures the average treatment rate can miss covariate patterns relevant to assignment.

There is a useful but limited finite-sample property of ordinary, unpenalized logistic maximum likelihood. Suppose the fitted model has linear predictor \(\beta^\mathsf{T}Z_i\), where \(Z_i\) is the vector of included model terms, and \(\hat e_i\) is the fitted logistic probability. If a finite maximum-likelihood estimate exists, its score equations are

If \(Z_i\) contains an intercept, then \(\sum_i A_i=\sum_i\hat e_i\). If it also contains a covariate as a linear term, the corresponding equation makes its observed treated total equal its fitted-probability-weighted total, \(\sum_i A_iX_i=\sum_i\hat e_iX_i\). These are calibration equations involving the fitted probabilities. They do not generally imply that the treated and control sample means of \(X_i\) are equal. They also provide no such constraint for omitted covariates, nonlinear functions, or interactions. Penalization, separation, and predictions made out of fold can alter or remove these in-sample likelihood equations.

Score Grouping Can Reintroduce Imbalance

Even if scores are known correctly, replacing exact score values with broad intervals can lose the balancing property. Units with different scores can have different treatment probabilities and different covariate distributions. Pooling them changes the mixture of covariate values in the treated and control groups. Coarse grouping is sometimes used for practical reasons, but it is an approximation, not the exact theorem.

Diagnostics and Alternatives

Balance must be checked on the covariates that matter, rather than inferred from the appearance of fitted scores alone. A useful diagnostic compares treated and control distributions after the intended score-based adjustment or grouping. Examine covariate means and proportions, but also consider spread, quantiles, nonlinear transformations, and interactions when they are substantively relevant. A balance check only for variables included as main effects can overlook imbalance in their squares or joint patterns.

For a quantitative covariate, a standardized mean difference is one descriptive summary: the treated-minus-control mean difference divided by a standard deviation used for scaling. For a binary covariate, the corresponding difference in proportions is often easy to interpret directly. These summaries describe observed imbalance; they are not tests that prove the groups are equivalent. Inspecting score overlap and the number of treated and control units in each score region is also important, since a nominally narrow bin with very few units cannot support a stable comparison.

A fitted-score approach is not the only way to assess or pursue comparability. Directly inspecting covariate balance makes the target of the diagnostic explicit; a richer model can represent nonlinearities or interactions, while a more restrictive model may be easier to estimate but leave patterns unmodeled. Stratifying on scores trades residual differences from coarse grouping against sparse strata. None of these choices substitutes for checking the resulting covariate distributions. In particular, a good prediction or calibration metric for treatment does not, by itself, establish covariate balance.

Out-of-fold prediction can help limit overfitting, but scores predicted for held-out units do not satisfy the training-sample logistic likelihood equations for those units. That is not a defect: those equations are not a general balance guarantee. It means balance should be diagnosed using the predictions and the sample in which they will be used. Uncertainty from estimating scores is a separate issue; the immediate point is that neither fitting nor cross-fitting makes finite-sample balance automatic.

Common Mistake

A common but incorrect argument is: “The propensity model converged and its fitted probabilities are calibrated, so treated and control units must be balanced.” Convergence means the fitting algorithm found an estimate under the specified model; calibration may mean certain aggregate predictions agree with observed totals. Neither statement establishes equality of covariate distributions. The correction is to treat the balancing theorem as a population property of the true score, and to examine covariate balance directly after using estimated scores. An exact balance result for selected model terms, when one applies, should be described narrowly rather than extended to every covariate or to treated-versus-control equality.