Why Propensity-Score Balance Does Not Guarantee Covariate Balance

Why a Similar Score Is Not a Balance Certificate

A propensity score compresses a covariate vector into one number. That compression is useful: under the right conditions, conditioning on the true score balances the measured covariates. But seeing similar score distributions in treated and control groups does not, by itself, establish that those conditions hold or that the covariates are balanced in the analyzed sample. A score plot is a diagnostic about the score; covariate balance must still be assessed directly.

Let \(A\) indicate treatment, with \(A=1\) for treated and \(A=0\) for controls, and let \(X\) denote the measured pretreatment covariates. The true propensity score is the conditional treatment probability given those covariates.

Definition: The true propensity score is \(e(X)=P(A=1\mid X)\). A function \(b(X)\) is a balancing score if \(A\) is independent of \(X\) conditional on \(b(X)\). The true propensity score is a balancing score in the population. This result concerns conditioning on the true score; it does not say that similar marginal distributions of estimated scores guarantee balance in a finite sample.

The distinction is important. The balancing-score result concerns the relationship between treatment and covariates within levels of the true score. A comparison of two marginal score histograms does not show whether the score was correctly estimated, whether relevant covariate structure was represented, or whether balance holds for particular covariates and functions of covariates. Nor does visual overlap mean that the distributions are identical.

What the Balancing-Score Result Says

For the true score, \(e(X)\) is determined by \(X\). Conditional on \(X\), treatment has probability \(e(X)\). The law of iterated expectation therefore gives \(P(A=1\mid e(X))=e(X)\). More explicitly, for any set \(B\) of covariate values,

\begin{aligned} P(X\in B,A=1\mid e(X)) &=E\!\left[1_{\{X\in B\}}P(A=1\mid X,e(X))\mid e(X)\right]\\ &=E\!\left[1_{\{X\in B\}}e(X)\mid e(X)\right]\\ &=e(X)P(X\in B\mid e(X)). \end{aligned}

The last equality holds because \(e(X)\) is fixed once we condition on it. Since \(P(A=1\mid e(X))=e(X)\), the joint conditional probability factors into the conditional probabilities for \(X\in B\) and \(A=1\). This is conditional independence: \(A\) is independent of \(X\) given the true score. For causal comparisons, positivity is also needed: the relevant covariate profiles must have a nonzero probability of receiving either treatment. The balancing result alone does not establish that causal effects are identified; it is a statement about treatment and measured covariates.

In an analysis, however, the score is usually estimated, perhaps from a specified model. If that model omits a relevant predictor, represents a nonlinear relationship poorly, or leaves important covariate structure inadequately captured, its fitted score need not be a balancing score. Even a correctly specified score model does not ensure that random variation in the realized sample disappears. And if investigators compare only a coarsened score, such as broad bins, differences within those bins may be invisible.

Worked Examples

Worked Example: Identical Fitted-Score Histograms, Imbalanced Covariate

Suppose a hypothetical analysis uses fitted scores based only on \(X_1\), while an important binary covariate \(X_2\) was omitted from the score model. Among 100 treated units, 50 have a fitted score of 0.20 and 50 have a score of 0.80. The controls have exactly the same score counts. Yet \(X_2=1\) for 40 treated units and 20 controls. Compute the standardized difference for \(X_2\).

The treated and control proportions with \(X_2=1\) are \(\hat p_1=40/100=0.40\) and \(\hat p_0=20/100=0.20\). For a binary covariate, use the pooled standard deviation \(s_{\text{pool}}=\sqrt{[\hat p_1(1-\hat p_1)+\hat p_0(1-\hat p_0)]/2}\). Thus,

s_{\text{pool}} =\sqrt{\frac{0.40(0.60)+0.20(0.80)}{2}} =\sqrt{0.20} =0.4472, \qquad \text{standardized difference} =\frac{0.40-0.20}{0.4472} =0.4472.

The score histograms match exactly, but the standardized difference for \(X_2\) is about 0.447, showing a notable mean imbalance under common diagnostic conventions. The score similarity is not reassuring here: the score model excluded a covariate that differs between groups. A direct covariate balance table or plot would reveal what the score histogram misses. This example uses fitted scores; it does not contradict the balancing-score theorem for the true score.

Worked Example: True Scores Can Match in a Realized Sample While Covariates Differ

Consider a hypothetical randomized study in which every unit has the true propensity score \(e(X)=0.50\). In one realized sample, all 20 treated units happen to have \(X=1\), while all 20 controls have \(X=0\). Compare the observed covariate proportions and calculate the standardized difference.

The treated proportion is \(\hat p_1=20/20=1\); the control proportion is \(\hat p_0=0/20=0\). The pooled standard deviation is

s_{\text{pool}} =\sqrt{\frac{1(0)+0(1)}{2}}=0.

Because both observed groups have no variation in this binary covariate, the standardized difference formula has a zero denominator and is undefined. The raw difference in proportions is nevertheless clear: \(1-0=1\), or 100 percentage points. This unusual but possible randomization outcome illustrates that equal true scores do not force exact balance in every finite sample. The population result is conditional independence; it is not a guarantee about the covariate composition of every realized treated and control group. In practice, report the raw proportions and flag the degeneracy rather than treating an undefined standardized difference as evidence of balance.

Worked Example: Balanced Covariate Means, Imbalanced Interaction

Suppose a hypothetical sample has two binary covariates, \(X_1\) and \(X_2\), and all units have the same true propensity score, 0.50. Among 40 treated units, 20 have profile \((X_1,X_2)=(0,0)\) and 20 have profile \((1,1)\). Among 40 controls, 20 have profile \((0,1)\) and 20 have profile \((1,0)\). Compare the marginal covariate means and the interaction \(X_1X_2\).

For each group, half the units have \(X_1=1\) and half have \(X_2=1\). Therefore, \(\bar X_1=\bar X_2=0.50\) in both groups: the two marginal means are balanced. But \(X_1X_2=1\) only for the \((1,1)\) profile. Its treated mean is \(20/40=0.50\), and its control mean is \(0/40=0\). For this binary interaction, the pooled standard deviation is

s_{\text{pool}} =\sqrt{\frac{0.50(0.50)+0(1)}{2}} =\sqrt{0.125} =0.3536, \qquad \text{standardized difference} =\frac{0.50-0}{0.3536} =1.414.

The marginal means and score are identical, but the interaction has a large observed imbalance. This example shows why checking only individual means can miss differences in joint covariate structure. It also shows that the realized-sample balance claim is not rescued by the common true score. In applications, inspect scientifically relevant interactions and subgroups as well as the marginal covariates.

Why Exact Marginal Equality Is Not the Theorem

It is tempting to read the balancing-score result as saying that treated and control groups should have the same marginal distribution of the true score. That is not what it says: it says that covariate distributions agree between treatment groups conditional on a score value. In the population, the treated and control score distributions are generally different because treatment probability itself depends on the score.

To see this, write \(f(e)\) for the population density of the true score and \(\pi=P(A=1)\) for the treatment prevalence. By conditioning on treatment, the score densities are proportional to treatment probability and control probability, respectively:

f(e\mid A=1)=\frac{e f(e)}{\pi}, \qquad f(e\mid A=0)=\frac{(1-e)f(e)}{1-\pi}.

If these two population densities were exactly equal wherever \(f(e)>0\), cancellation gives \(e/\pi=(1-e)/(1-\pi)\). Multiplying through yields \(e(1-\pi)=\pi(1-e)\), hence \(e=\pi\). Thus exact equality of the full treated and control distributions of the true score in the same population implies a constant score, apart from values with no population support. This is a population identity, not a rule for finite samples or fitted scores. It underlines why “the histograms look alike” is neither the balancing theorem nor a standalone certificate of covariate balance.

How to Diagnose the Discrepancy

Treat score plots as one part of a collection of diagnostics. First check whether the score distributions have adequate overlap for the comparison being made. Then inspect the covariates directly in the same analyzed sample, using summaries suited to their types and distributions. Relevant checks may include mean differences, variance ratios, distributional plots, and diagnostics for interactions or nonlinear terms. The earlier material on these diagnostics provides tools for checking distributional and joint balance; here the central point is to apply them to the covariates rather than infer their results from the score alone.

Clarify which score is being assessed.
Distinguish the true score, which is generally unknown, from fitted scores produced by an estimated model. Similarity of fitted values does not establish that the model captures the relevant treatment-assignment structure.

Check score overlap and its limits.
Look for regions where one group has little or no representation. Overlap is important for feasible comparisons, but overlap alone does not show balance in covariates within the represented range.

Check each covariate directly.
Compare treated and control summaries and distributions for every measured pretreatment covariate, not just the covariates most prominent in a score plot.

Inspect structure beyond marginal means.
Assess meaningful nonlinear terms, interactions, and subgroups. A covariate can have similar means while its distribution or relationship with another covariate differs.

Interpret diagnostics together.
A favorable result on one summary does not cancel an unfavorable result on another. Record which aspects are balanced and which remain problematic, rather than labeling the entire design “balanced” from a single plot.

Common Mistakes, Alternatives, and Limits

Mistake: “The score histograms overlap, so the covariates are balanced.” Overlap describes score support, not direct equality of covariate distributions. Check the covariates and relevant functions themselves.
Mistake: “The balancing-score theorem applies to any fitted score.” The theorem applies to the true propensity score. An estimated score is an approximation, and misspecification can leave residual imbalance.
Mistake: “Balanced means imply balanced covariates.” Means do not characterize full distributions or joint structure. Check spread, tails, nonlinear terms, and interactions where relevant.
Mistake: “A balance threshold proves the design is valid.” A standardized-difference threshold is a useful convention, not a universal boundary between valid and invalid causal inference. Balance diagnostics address measured covariates, not unmeasured confounding.

Alternative approaches answer different diagnostic questions. A score-density plot is useful for visualizing overlap and gross score differences, but compresses covariates and can hide model failures. Direct standardized differences and distributional comparisons show which observed variables differ, but each summary can miss other forms of imbalance. Joint diagnostics can reveal multivariate discrepancies, but may be harder to interpret and do not replace checking scientifically important individual covariates. No single diagnostic establishes causal identification or rules out unmeasured confounding.

Key takeaway: The true propensity score is a balancing score in the population when conditioning on that score. Similar marginal distributions of fitted or realized scores are not proof that measured covariates—or their nonlinear functions and interactions—are balanced in the analyzed sample. Assess overlap and covariate balance separately.

Check Your Understanding

Answer each question, then compare your reasoning with the brief solutions.

What does the balancing-score result say about \(A\), \(X\), and the true propensity score?
A study reports nearly identical fitted-score means but does not show covariate summaries. What can and cannot be concluded?
In a hypothetical sample, a binary covariate has proportions 0.30 in the treated group and 0.10 in controls. Using the pooled-variance convention, calculate its standardized difference.
Why might two groups have equal means for each of two covariates but differ on their interaction?
Does a favorable balance diagnostic rule out confounding by an unmeasured variable? Explain.

Solutions. (1) For the true score \(e(X)=P(A=1\mid X)\), treatment and measured covariates are conditionally independent given the score in the population; this does not promise exact finite-sample balance. (2) Similar means provide only a comparison of score means. They do not establish equal score distributions, fitted-score correctness, or covariate balance; inspect the covariates directly. (3) \(s_{\text{pool}}=\sqrt{[0.30(0.70)+0.10(0.90)]/2}=\sqrt{0.15}=0.3873\), so the standardized difference is \((0.30-0.10)/0.3873=0.5164\). (4) Marginal means do not determine the joint distribution; for example, profiles \((0,0)\) and \((1,1)\) can produce different interaction averages than profiles \((0,1)\) and \((1,0)\), even when both marginal means are 0.50. (5) No. Balance diagnostics assess measured covariates. They cannot show that unmeasured confounding is absent.