Known Versus Estimated Propensity Scores

Why Known and Estimated Scores Must Be Distinguished

A propensity score is defined from the treatment assignment mechanism, not from a particular fitted model. Its population balancing result is exact: among units with the same true score, treatment status is independent of measured baseline covariates. In practice, however, the true treatment mechanism is usually unknown, so analysts estimate scores from data. The resulting fitted values are not automatically the true scores, and even a well-behaved fitting procedure does not promise exact balance in every finite sample.

Keeping those claims separate matters when interpreting matching, weighting, or other procedures that use propensity scores. A balance theorem about a population quantity is not a guarantee about the covariate counts in a particular data set. Balance must be assessed in the data and in the sample actually used for analysis.

The true-score balancing property concerns the conditional distribution of \(X\) among treated and untreated units at a fixed score value. It is not a claim that treated and untreated units have identical covariates overall. The score can vary across the population, and covariate distributions can differ substantially before conditioning on it. For the causal use of the score, positivity—\(0<e(X)<1\) on the covariate values of interest—is also important. The balancing identity itself is about treatment assignment and covariates; causal interpretations additionally require causal identification assumptions such as conditional exchangeability.

Why the True Score Balances Covariates

For any covariate-defined set \(B\), consider units whose true score equals \(s\), where \(0<s<1\). On this set, the probability of treatment is \(s\) for every covariate value. The probability of both belonging to \(B\) and receiving treatment, conditional on the score, is therefore \(s\) times the probability of belonging to \(B\). Dividing by the treatment probability within the score stratum gives the treated covariate distribution. The same calculation for controls cancels the factor \(1-s\).

Since these equalities hold for every covariate-defined set \(B\), they state that \(A\) and \(X\) are independent conditional on the true score: \(A\perp X\mid e(X)\). The argument is a population calculation. It does not say that each realized sample will have precisely equal covariate frequencies within a score stratum.

Why a True Score Does Not Guarantee Sample Balance

The preceding result describes probabilities in a population. A finite data set is one realization of treatment assignments and covariates. Random variation can leave treated and untreated units with noticeably different covariate distributions even when every unit has the same true score. If a procedure further restricts the sample—for example, by retaining only some units for matching—the balance of that selected sample is a separate empirical question.

Why Estimated Scores May Not Balance

An estimated score can differ from the true score for several reasons. The model may omit relevant covariates, impose an inadequate functional form, or be unstable because the sample contains little information in some covariate regions. Even if the model is appropriate, estimated parameters vary from sample to sample. In addition, an in-sample fitted score is calculated using data that include the observed treatment assignments. The fixed-score population argument cannot simply be applied to these data-dependent values as if they were known population probabilities.

Under suitable conditions, a correctly specified and consistently estimated model can make fitted scores approach the true scores as sample size grows. That is an asymptotic statement, not an exact finite-sample guarantee. It does not imply that every fitted-score stratum has identical treated and control covariate distributions, nor that every matching or weighting implementation achieves balance. The quality of the result depends on the model, the data, and the subsequent analysis.

Diagnostics, Comparisons, and Limitations

Balance diagnostics should be applied to the groups that will actually be compared, after any matching, weighting, or restriction. Examine each important baseline covariate directly, not just the distribution of fitted scores. For a binary covariate, compare group proportions and their difference. For a quantitative covariate, compare means and inspect distributional summaries or plots; equal means alone do not rule out differences in spread or shape. Standardized differences can help compare imbalances across covariates measured on different scales, but no single threshold proves that all confounding has been removed.

The main alternatives answer different questions. If the true score were known, the population balancing identity would be exact, but the observed sample could still fluctuate. Using an estimated score makes the method practicable, but introduces model and estimation uncertainty and requires empirical diagnostics. Direct covariate adjustment does not rely on a score to summarize \(X\), but it has its own modeling and design choices. None of these approaches makes unmeasured covariates balanced merely by construction.

A common mistake is to reason, “The propensity score balances covariates by definition, so the fitted scores must have made my sample balanced.” The first clause applies to the true score in the population; the conclusion does not follow for estimated scores in a finite sample. Similar fitted-score distributions are not proof of covariate balance, and covariate balance by itself is not proof that all causal assumptions hold. Diagnostics can reveal problems in measured covariates, but cannot verify the absence of unmeasured confounding.