Residual Confounding Within Propensity-Score Strata

Why Sharing a Stratum Is Not the Same as Being Balanced

Subclassification groups units with similar propensity scores and compares treated and control outcomes within those groups. But a stratum is an interval, not a single propensity-score value. If treatment probabilities still vary inside that interval, treated and control units can have different covariate distributions there. When those covariates predict the outcome, the within-stratum comparison may retain confounding.

The key point is not that subclassification fails to be useful. It is that grouping on a score coarsens information: membership in the same group does not generally make treatment independent of the covariates. To understand the remaining bias, distinguish the stratum label from the propensity score values within that stratum.

Definition: Let \(A\in\{0,1\}\) indicate treatment, \(X\) denote measured pretreatment covariates, and \(e(X)=P(A=1\mid X)\) be the propensity score. Let \(S=s(X)\) be a subclassification stratum formed from the score. Residual confounding within strata is the remaining difference in covariate-related potential outcomes between treated and control units within a stratum, because treatment assignment is not generally independent of \(X\) given only \(S\).

For the causal interpretation below, assume consistency (the observed outcome is \(Y=Y(A)\)), no interference between units, and conditional exchangeability, \(Y(a)\perp A\mid X\), for \(a=0,1\). Also require positivity: \(0<e(X)<1\) for covariate values in the target population. These assumptions concern treatment and potential outcomes; dividing the score into strata does not itself establish them.

Deriving the Within-Stratum Imbalance

For a function \(h(X)\), such as a covariate or a prognostic function of covariates, compare its mean among treated and control units in a stratum \(S=s\). Write \(m_s=E[e(X)\mid S=s]\). By the definition of the propensity score, weighting the covariate distribution by the chance of treatment gives:

E[h(X)\mid A=1,S=s] =\frac{E[e(X)h(X)\mid S=s]}{m_s}, \qquad E[h(X)\mid A=0,S=s] =\frac{E[(1-e(X))h(X)\mid S=s]}{1-m_s}.

These expressions require \(0<m_s<1\). Subtracting the control mean from the treated mean gives a useful exact identity:

\begin{aligned} &E[h(X)\mid A=1,S=s]-E[h(X)\mid A=0,S=s]\\ &=\frac{E[e(X)h(X)\mid S=s]-m_sE[h(X)\mid S=s]} {m_s(1-m_s)}\\ &=\frac{\operatorname{Cov}(h(X),e(X)\mid S=s)} {m_s(1-m_s)}. \end{aligned}

The second equality follows because \(E[e(X)h(X)\mid S=s]-E[e(X)\mid S=s]E[h(X)\mid S=s]\) is the conditional covariance. Thus, a covariate is imbalanced in a stratum when it covaries with the propensity score there. Its direction depends on that covariance: a covariate that tends to be larger where the score is higher will, on average, also tend to be larger among treated units.

This derivation also identifies a sufficient condition for no residual covariate imbalance: if \(e(X)\) is constant within a stratum, then the covariance is zero for every \(h(X)\). In that case \(A\) is independent of \(X\) conditional on the stratum. A weaker, outcome-specific condition is that the prognostic function of \(X\) has zero covariance with \(e(X)\) within the stratum. Neither condition follows merely from assigning units to the same interval.

How Imbalance Becomes Outcome Bias

Let \(m_0(X)=E[Y(0)\mid X]\) be the conditional mean outcome without treatment, and let \(\tau(X)=E[Y(1)-Y(0)\mid X]\) be the conditional treatment effect. Under consistency and conditional exchangeability, the observed treated-control outcome contrast in a stratum can be written as:

\begin{aligned} &E[Y\mid A=1,S=s]-E[Y\mid A=0,S=s]\\ &=E[\tau(X)\mid A=1,S=s] +E[m_0(X)\mid A=1,S=s] -E[m_0(X)\mid A=0,S=s]. \end{aligned}

The final two terms are the residual-confounding component: treated and control units can have different expected untreated outcomes because their covariate distributions differ. The first term is the treatment effect averaged over treated units in the stratum. If effects vary with \(X\), that average need not equal the stratum’s population average treatment effect. To isolate residual confounding, suppose the treatment effect is constant, \(\tau(X)=\tau\). Then the contrast is \(\tau\) plus the difference in mean \(m_0(X)\). In particular, if \(m_0(X)=\beta h(X)\), the residual bias is \(\beta\) times the imbalance in \(h(X)\) derived above.

Key result: With a constant treatment effect, the within-stratum treated-control contrast equals the causal effect plus the treated-control difference in the mean untreated potential outcome. Balance in selected covariate means is not enough if important outcome predictors are nonlinear functions of those covariates.

Worked Example: A Binary Prognostic Covariate

Consider one hypothetical stratum in which half the population has \(Z=0\) and half has \(Z=1\). Suppose the true propensity scores are \(e(0)=0.30\) and \(e(1)=0.60\); both values fall in the same score interval. Check the treated-control imbalance in \(Z\), then calculate the outcome contrast if \(E[Y(0)\mid Z]=2Z\) and the treatment effect is constantly 1 outcome unit.

The stratum’s mean propensity score is \(m_s=(0.30+0.60)/2=0.45\). The treated mean of \(Z\) is:

E[Z\mid A=1,S=s] =\frac{E[e(X)Z\mid S=s]}{m_s} =\frac{(0.60)(0.50)}{0.45} =\frac{2}{3}.

Among controls, the corresponding mean is:

E[Z\mid A=0,S=s] =\frac{E[(1-e(X))Z\mid S=s]}{1-m_s} =\frac{(0.40)(0.50)}{0.55} =\frac{4}{11}.

The covariate difference is \(2/3-4/11=10/33\approx0.3030\). Since the untreated outcome mean is \(2Z\), the difference in untreated outcomes is \(2(10/33)=20/33\approx0.6061\) outcome units. Adding the constant treatment effect, the observed contrast is \(1+20/33=53/33\approx1.606\) outcome units. The score overlap within the stratum did not make the groups exchangeable there.

Worked Example: Calculating Bias From a Nonlinear Outcome

Consider a hypothetical coarse stratum containing \(Z=0,1,2\). Suppose the treated proportions at these values are \(0.40,0.20,0.40\), respectively, and the control proportions are \(4/15,7/15,4/15\). Assume every unit has \(Y(0)=Z^2\) and \(Y(1)=Z^2+1\). Calculate the within-stratum contrast and its residual bias.

For treated units, the mean of \(Z^2\) is:

E[Z^2\mid A=1,S=s] =(0)(0.40)+(1)(0.20)+(4)(0.40)=1.80.

For controls, the mean is:

E[Z^2\mid A=0,S=s] =\frac{(0)(4)+(1)(7)+(4)(4)}{15} =\frac{23}{15}\approx1.533.

So the difference in untreated outcomes is \(1.80-23/15=4/15\approx0.2667\) outcome units. The true treatment effect is exactly 1 outcome unit for every unit. The observed treated mean is \(1.80+1=2.80\), while the control mean is \(23/15\). Therefore, the within-stratum contrast is:

2.80-\frac{23}{15} =\frac{42}{15}-\frac{23}{15} =\frac{19}{15} \approx1.267\text{ outcome units}.

The contrast exceeds the true effect by \(4/15\approx0.2667\) outcome units. In this construction, the treated and control distributions both place mass on all three covariate values, so the difference does not require a lack of covariate support.

Why Mean Balance Can Miss the Problem

A balance check can look reassuring if it examines only a covariate’s mean. Yet the outcome may depend on that covariate nonlinearly, as in the previous example. The next hypothetical illustration shows that even exactly equal means of \(Z\) can coexist with imbalance in \(Z^2\).

Worked Example: Equal Covariate Means, Unequal Prognosis

Within a hypothetical stratum, let the treated distribution of \(Z=-1,0,1\) be \(0.25,0.50,0.25\), and the control distribution be \(0.10,0.80,0.10\). Suppose \(Y(0)=Z^2\) and the treatment effect is constantly 2 outcome units. Both groups have mean \(Z=0\). What is the outcome contrast?

For treated and controls alike, \(E[Z]=0\). But their mean squared covariate values differ:

E[Z^2\mid A=1,S=s]=0.25(1)+0.50(0)+0.25(1)=0.50, \qquad E[Z^2\mid A=0,S=s]=0.10(1)+0.80(0)+0.10(1)=0.20.

The untreated-outcome difference is \(0.50-0.20=0.30\) outcome units. The observed contrast is therefore \(2+0.30=2.30\) outcome units, although the true treatment effect is 2. This example is compatible with propensity scores inside a common interval: with equal overall group sizes, the implied scores at \(Z=-1,0,1\) are approximately \(0.714,0.385,0.714\), respectively. Equal covariate means alone do not establish balance in the outcome-relevant distribution.

Diagnostics, Alternatives, and Limits

Evaluate balance within each stratum rather than relying only on the overall balance or on the fact that groups share score intervals. Useful checks include treated-control differences in baseline covariate means, variances, and distributional summaries; transformations such as squares or clinically justified interactions; and the propensity-score range and spread among units in each stratum. Prioritize covariates known to predict the outcome. These checks describe measured covariates and cannot demonstrate that unmeasured confounding is absent.

Warning sign: treated and control score distributions occupy visibly different portions of the same stratum. This is direct evidence that the interval has not made their assignment probabilities alike.
Warning sign: a covariate has a small mean difference but an important variance or nonlinear-function difference. Assess outcome-relevant functions, not only raw means.
Warning sign: a stratum contains very few units in one treatment group. A small estimated imbalance can then be imprecise, and the within-stratum outcome contrast may be unstable.
Interpretation limit: balance on observed covariates is a diagnostic, not a test of conditional exchangeability. It cannot verify assumptions about unmeasured covariates or potential outcomes.

The main alternatives address the coarsening in different ways. Using the propensity score continuously in weighting avoids replacing its values with stratum labels, but requires attention to the weights and their target population. Covariate adjustment within strata can address remaining measured imbalance if its outcome model is appropriate. Matching on the score also seeks closer comparisons, though it can discard units or leave imbalance. These methods have their own assumptions and diagnostics; none repairs unmeasured confounding by itself.

A practical response to detected imbalance is to revise the analysis and recheck balance, for example by representing important covariate functions more directly or using a method that does not rely on coarse groups. There is no universal balance diagnostic that guarantees negligible bias: even exact balance in a selected set of summaries may leave differences in other outcome predictors. Finite-sample variability and propensity-score estimation error also affect empirical subclassification results, in addition to the population mechanism derived here.

Common Mistake

Common mistake: “The treated and control units are in the same propensity-score stratum, so their covariates are balanced and their outcome contrast is causal.” The correction is that stratum membership only places score values in the same interval. Unless the score is effectively constant within that interval, treated and control covariate means can differ; if those covariates predict the outcome, the contrast can retain bias. Check within-stratum balance and state the causal assumptions separately.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

Within one stratum, \(e(X)\) is constant. What does the imbalance identity imply for any function \(h(X)\)?
Suppose \(m_s=0.50\) and \(\operatorname{Cov}(h(X),e(X)\mid S=s)=0.02\). Find the treated-control difference in the mean of \(h(X)\).
With a constant treatment effect, the untreated-outcome mean is three times \(h(X)\). If treated units have a mean \(h(X)\) that is 0.10 higher than controls, what is the residual bias?
Why can equal treated and control means of \(Z\) fail to show adequate balance when the outcome depends on \(Z^2\)?

Solutions. (1) The covariance is zero, so the identity gives a zero difference for every \(h(X)\), provided \(0<m_s<1\). (2) The difference is \(0.02/[0.50(1-0.50)]=0.02/0.25=0.08\) units of \(h\). (3) The untreated-outcome difference is three times 0.10, so residual bias is 0.30 outcome units. (4) Equal means constrain only the first moment; the groups can still have different variances or distributions, and hence different mean values of \(Z^2\), which can create an untreated-outcome difference.