Balancing Scores and Conditional Covariate Distributions

What Does It Mean for a Score to Balance Covariates?

A score is useful for adjustment only if its groups have a clear relationship to the baseline covariates that influence treatment assignment. The key idea is conditional balance: after fixing a score value, the distribution of measured baseline covariates should be the same among treated and untreated units. This is a statement about distributions, not just averages.

Let \(T\) be a binary treatment indicator, with \(T=1\) for treated units and \(T=0\) for untreated units. Let \(X\) denote the vector of measured baseline covariates, and let \(b(X)\) be a function of those covariates. The function may return a single number or a vector. The propensity score \(e(X)=P(T=1\mid X)\) is one particular function introduced in the earlier material on propensity scores; the definition below applies more broadly.

Here, \(\mathcal{L}\) denotes a probability distribution. For a discrete covariate, equality means that every possible covariate pattern has the same conditional probability in the two groups. For continuous covariates, it means the conditional probability distributions agree, not merely that a few summaries agree. The condition concerns the measured covariates in \(X\); it does not establish balance on unmeasured characteristics.

The equality is meaningful only at score values for which treated and untreated units are both represented. If a score value occurs only among treated units, there is no untreated conditional distribution at that value to compare. This is a lack of overlap, not evidence that balance has been demonstrated.

Balance Is a Distributional Condition

Suppose \(X\) includes age, disease severity, and a binary indicator for a prior diagnosis. At a fixed score value, balance requires the treated and untreated groups to have matching conditional distributions for the joint covariate vector. It therefore implies agreement for each covariate separately, but checking each marginal distribution alone may fail to detect differences in how covariates occur together.

For example, two groups could have the same percentage of older patients and the same percentage with a prior diagnosis, yet differ in whether older patients are especially likely to have that diagnosis. Since the joint distribution is part of the condition, such a difference is a balance failure. Conversely, equality of means alone is insufficient: distinct distributions can have the same mean.

The example shows what balance means; it does not imply that all functions of covariates balance. Whether a particular score satisfies the definition depends on the conditional distributions it produces. The next example shows how a score can group units together while leaving their covariate mix different.

Why Matching Means Is Not Enough

A diagnostic that compares only means can miss substantial distributional differences. Consider one score stratum in hypothetical data. The treated covariate values are \(0,0,2,2\), while the untreated values are \(-1,1,1,3\). Both groups have mean 1:

But the distributions are not equal. For instance, all treated values lie between 0 and 2, whereas the untreated group includes values of \(-1\) and 3. The mean comparison does not detect these tail differences. A balance assessment should therefore examine distributional features relevant to the covariates: for example, category proportions, spread, quantiles, empirical distribution functions, and important joint patterns. No single summary proves equality of entire distributions in finite samples, but multiple checks can reveal problems hidden by a mean comparison.

A Short Justification of the Equivalent Statements

Conditional independence gives a precise way to express the distributional comparison. By definition, \(T\perp X\mid b(X)\) means that, after conditioning on \(b(X)\), learning the treatment status does not change the conditional distribution of \(X\). For any measurable set \(A\) of covariate values and score value \(u\) with both groups represented, this requires:

If this equality holds for every such set \(A\), the two conditional distributions agree. Conversely, if the conditional distributions agree, then the conditional probability of every event involving \(X\) is the same across treatment groups at that score value, which is precisely conditional independence of \(T\) and \(X\) given \(b(X)\). The definition is thus stronger than checking any finite list of averages or proportions.

How to Assess Balance and Recognize Failure

In a data analysis, the population condition is not directly observable: analysts have a finite sample. Balance diagnostics use the observed data to look for discrepancies within score groups or across score values represented in both treatment groups. The checks should reflect the structure of \(X\), including categorical variables, continuous variables, and important combinations of covariates.

Imbalance can appear as different category proportions, shifted quantiles, unequal tails, or different combinations of covariates within a score group. Sparse data can also make sample diagnostics unstable: apparent equality in a small stratum is weak evidence about the underlying distributions. Balance diagnostics concern measured covariates and cannot reveal whether unmeasured variables are balanced.

Relationship to Other Ways of Conditioning

Conditioning on the full covariate vector \(X\) compares units with the same covariate values. A balancing score instead groups covariate values according to a function \(b(X)\); by definition, those groups must retain the same conditional distribution of \(X\) across treatment groups. If a proposed score does not meet that condition, conditioning on it alone does not provide balance as defined here.

This distinction is about the balance property, not a guarantee that any particular analysis identifies a causal effect. Balance is defined for measured baseline covariates. It does not, by itself, address unmeasured confounding, establish that all relevant covariates were measured, or resolve lack of overlap. Nor does a finite-sample diagnostic establish exact equality of population distributions. Those are separate considerations in causal analysis.

A common mistake is to say, “The treated and untreated groups have the same average age within each score group, so the score balances age.” The reasoning is incomplete: equal means do not imply equal distributions. The correction is to describe the result narrowly—mean age is similar—and assess other features of the age distributions and the other covariates. A second mistake is to compare covariate distributions only across the full treated and untreated samples. Overall groups can differ because they contain different proportions of score values, even when their covariate distributions agree within score values; conversely, an overall comparison can hide within-score discrepancies.