Why Stratum-Specific Effects Need Careful Interpretation
Subclassification groups people by ranges of their propensity scores, then compares treated and control outcomes within each group. Those within-stratum comparisons can reveal an important feature that a single overall estimate can hide: the average effect may differ across strata. For example, treatment may have a larger average effect among people with high estimated treatment probabilities than among people with low estimated probabilities.
A difference between stratum-specific estimates is not automatically evidence of effect heterogeneity. Estimates vary partly because of sampling uncertainty, and a crude treated-control comparison can also change because the treated and control groups contain different proportions of people from each stratum. These are distinct issues: effect heterogeneity concerns differences in effects; composition concerns differences in who is represented.
This is a comparison of averages for different groups, not a claim that every person in one stratum has a larger individual treatment effect than every person in another. Nor does it show that the propensity score itself causes the effect to differ. The strata are groups defined by baseline characteristics summarized by the score.
When a Within-Stratum Contrast Has a Causal Interpretation
Within stratum \(k\), the observed mean outcome among treated people can be compared with the observed mean among controls. To identify \(\tau_k\) with that difference, require consistency (the observed outcome equals the potential outcome under the treatment actually received), no interference between individuals, exchangeability of treatment and potential outcomes within the stratum, and a positive probability of receiving either treatment level in the stratum. In notation, exchangeability is \((Y(1),Y(0))\perp A\mid S=k\), and positivity requires \(0<P(A=1\mid S=k)<1\).
Under these conditions, the observed treated mean identifies \(E[Y(1)\mid S=k]\), and the control mean identifies \(E[Y(0)\mid S=k]\). Thus their difference identifies \(\tau_k\). In a propensity-score analysis, however, placing people in the same estimated-score stratum does not guarantee exchangeability. Covariates can remain imbalanced within a stratum, leaving residual confounding. The earlier material on Subclassification on the Propensity Score motivates checking balance within strata rather than treating the grouping itself as proof of comparability.
The stratum-specific effect is defined for the people in that stratum. If the target population changes, the weighting of the stratum effects in an overall effect can change even when every \(\tau_k\) remains the same. The interpretation of subgroup effects and the interpretation of an overall effect are therefore related but not interchangeable.
Worked Example: Reading Two Stratum-Specific Effects
Suppose a hypothetical study compares a program with usual care. The outcome is symptom-free days during a fixed follow-up period; higher values are better. Propensity-score subclassification produces two strata. The table gives the observed sample sizes and mean outcomes.
| Stratum | Treated: n | Treated mean | Control: n | Control mean | Within-stratum contrast |
|---|---|---|---|---|---|
| Lower score | 30 | 8.0 days | 60 | 6.0 days | 2.0 days |
| Higher score | 70 | 5.0 days | 40 | 4.0 days | 1.0 day |
In the lower-score stratum, the estimated contrast is \(8.0-6.0=2.0\) symptom-free days. In the higher-score stratum, it is \(5.0-4.0=1.0\) day. If the causal identification assumptions hold within both strata, these estimate stratum-specific average effects. The point estimates suggest a larger effect in the lower-score stratum, by \(2.0-1.0=1.0\) day.
The table alone does not establish that the effects truly differ. We would also need to assess uncertainty in both estimates and in their difference. Nor can we infer causality merely from the arithmetic: exchangeability within each stratum, adequate treatment and control representation, consistency, and no interference remain necessary. In particular, the unequal cell sizes do not prove a positivity violation, but very sparse cells would make estimates unstable and could signal inadequate overlap.
Effect Heterogeneity Versus Stratum Composition
A crude contrast compares all treated people with all control people without first making their stratum distributions the same. If those distributions differ, the crude contrast combines outcome differences within strata with differences in the strata represented in the two groups.
Let \(p_{1k}=P(S=k\mid A=1)\) and \(p_{0k}=P(S=k\mid A=0)\), and let \(\mu_{ak}=E[Y\mid A=a,S=k]\). The crude treated-control difference is \(\sum_k p_{1k}\mu_{1k}-\sum_k p_{0k}\mu_{0k}\). Adding and subtracting \(\sum_k p_{1k}\mu_{0k}\) gives the exact decomposition below.
This identity is arithmetic; by itself it does not make either component causal. Under the identification assumptions, the within-stratum contrasts can represent the \(\tau_k\). The composition component remains a consequence of the differing stratum distributions and control outcome means. It can increase, decrease, or reverse the crude contrast. A difference in crude outcomes is therefore not, by itself, evidence that treatment effects vary.
Worked Example: A Crude Contrast Reversed by Composition
Consider a hypothetical binary outcome measured as a score, with higher values better. There are two strata, and both have positive within-stratum treatment contrasts.
| Stratum | Treated mean \(\mu_{1k}\) | Control mean \(\mu_{0k}\) | Treated proportion \(p_{1k}\) | Control proportion \(p_{0k}\) |
|---|---|---|---|---|
| Lower score | 12 | 10 | 0.20 | 0.80 |
| Higher score | 8 | 5 | 0.80 | 0.20 |
The within-stratum contrasts are \(12-10=2\) points in the lower-score stratum and \(8-5=3\) points in the higher-score stratum. If exchangeability and the other identification assumptions hold within each stratum, both estimated effects are positive, with a larger point estimate in the higher-score stratum.
The crude treated mean is $$ 0.20(12)+0.80(8)=2.4+6.4=8.8 \text{ points}. $$ The crude control mean is $$ 0.80(10)+0.20(5)=8.0+1.0=9.0 \text{ points}. $$ Therefore the crude contrast is \(8.8-9.0=-0.2\) points, even though the contrast is positive in both strata.
The decomposition confirms why: $$ \text{within-stratum component}=0.20(2)+0.80(3)=2.8 \text{ points}, $$ $$ \text{composition component}=(0.20-0.80)(10)+(0.80-0.20)(5)=-6+3=-3.0 \text{ points}, $$ $$ 2.8+(-3.0)=-0.2 \text{ points}. $$ The treated group contains many more people from the higher-score stratum, where the control mean is lower. That distributional difference outweighs the positive within-stratum contrasts in the crude comparison. This example illustrates composition, not evidence that one stratum's treatment effect is negative.
Assessing Whether Estimated Effects Differ
To assess heterogeneity, examine contrasts between stratum-specific estimates along with their uncertainty. It is not enough to observe that one estimate is statistically significant and another is not: that comparison does not test whether the estimates differ. The relevant quantity is a difference such as \(\hat{\tau}_j-\hat{\tau}_k\), with its own standard error and interval.
If estimates from two disjoint strata are independent, an approximate standard error for their difference is \(\sqrt{SE(\hat{\tau}_j)^2+SE(\hat{\tau}_k)^2}\). If they are not independent, the covariance must also be accounted for. A formal interaction analysis is another way to assess whether effects differ across groups, but it still depends on a suitable model and valid causal identification. Subgroup comparisons can be imprecise, especially with many strata or small treatment and control samples in some strata.
Worked Example: Quantifying Uncertainty in a Difference
Suppose hypothetical estimated effects on a continuous outcome are \(4.0\) points in one stratum, with \(SE=1.0\) point, and \(1.5\) points in another, with \(SE=0.8\) point. Assume for this calculation that the estimates are independent and approximately normally distributed.
The estimated difference is \(4.0-1.5=2.5\) points. Its standard error is $$ SE(\hat{\tau}_1-\hat{\tau}_2)=\sqrt{1.0^2+0.8^2} =\sqrt{1.64}=1.281 \text{ points}. $$ The approximate test statistic for no difference is \(z=2.5/1.281=1.952\), giving a two-sided \(p\)-value of about \(0.0509\). A 95% confidence interval is $$ 2.5\mathbin{\pm}1.96(1.281) =2.5\mathbin{\pm}2.510 =(-0.010,\ 5.010) \text{ points}. $$
At the 0.05 significance level, this calculation does not provide sufficient evidence to reject equal effects: the two-sided \(p\)-value is just above 0.05, and the interval includes zero. The estimated difference is substantial in point terms, but its uncertainty is also substantial. This inference is about the difference in stratum-specific estimates; causal interpretation additionally requires valid identification within each stratum.
Diagnostics, Alternatives, and Limitations
Before interpreting apparent heterogeneity, check whether the comparisons are credible and whether the differences are more than noise.
- Check covariate balance within strata. Remaining differences in baseline covariates can confound the within-stratum contrasts.
- Check treatment and control representation. Very small cells or strata with only one treatment group yield unstable estimates or prevent a within-stratum comparison.
- Inspect uncertainty, not just point estimates. Wide intervals and multiple subgroup comparisons make apparent patterns easy to overinterpret.
- Distinguish the estimand from the sample distribution. A crude contrast can change when the proportions of strata differ, even if the within-stratum effects do not.
Stratum-specific estimates make effect variation visible, while a pooled contrast summarizes outcomes under the stratum mix present in the groups being compared. A regression model with treatment-by-covariate interactions offers a model-based alternative for describing effect modification; it can be more concise, but its results depend on the model specification and the chosen scale. Neither approach eliminates confounding by itself. With a binary outcome, for example, differences on a risk-difference scale need not match differences on a ratio scale, so the scale must be stated when describing heterogeneity.
A common mistake is to say, “The overall treated mean is lower, so treatment must harm people in at least one stratum.” That conclusion does not follow: as the composition example showed, a crude contrast can be negative while all within-stratum contrasts are positive. The correction is to compare outcomes within strata, examine their uncertainty and identification assumptions, and separately account for how stratum proportions differ.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- In one stratum, the treated mean is 15 points and the control mean is 11 points. What is the estimated within-stratum contrast, and what additional conditions support a causal interpretation?
- Two stratum-specific estimates are 2.0 and 2.5 points. Does their difference alone establish effect heterogeneity? Explain.
- Suppose treated and control groups have different proportions in each stratum. Can their crude contrast be interpreted as a simple average of the stratum-specific effects without further qualification?
- Why is “significant in one stratum but not significant in another” not, by itself, a valid test that effects differ?
Solutions. (1) The contrast is \(15-11=4\) points. A causal interpretation requires consistency, no interference, exchangeability within the stratum, and positive probabilities of both treatment levels there. (2) No. The point estimates differ by \(2.5-2.0=0.5\) points, but uncertainty in that difference must be assessed before concluding that the underlying effects differ. (3) No. Different stratum proportions can contribute a composition component to the crude contrast; the within-stratum contrasts and the target population's stratum weights must be distinguished. (4) The significance decisions concern each estimate relative to zero, not the difference between estimates. Test or form an interval for the difference itself, accounting for its standard error and any covariance.