What a Within-Stratum Difference Measures
Propensity-score stratification divides individuals into groups with similar estimated probabilities of receiving treatment. Once the strata have been formed, a direct calculation is available in each group: find the mean outcome among treated individuals, find the mean among untreated individuals, and subtract the second mean from the first. This produces a separate observed contrast for every stratum. It does not, by itself, combine those contrasts into a single population effect.
Let \(T_i\) indicate treatment for individual \(i\), with \(T_i=1\) for treated and \(T_i=0\) for untreated. Let \(Y_i\) be that individual's observed outcome, and let \(S_i=k\) indicate membership in propensity-score stratum \(k\). A larger \(Y_i\) is not necessarily better; the meaning of the sign depends on the outcome and the chosen subtraction order. Throughout this tutorial, the contrast is treated minus untreated.
The word “observed” matters. The calculation compares outcomes actually recorded for treated and untreated individuals in the same stratum. Calling \(\hat{\Delta}_k\) a causal effect requires additional assumptions; stratum membership alone does not prove that the groups are comparable in all relevant respects.
Calculation and Causal Interpretation
For a binary treatment, define \(Y_i(1)\) and \(Y_i(0)\) as the potential outcomes for individual \(i\) under treatment and no treatment. Only one is observed for each individual. The causal average effect in stratum \(k\), for the individuals in that stratum, is \(E[Y(1)-Y(0)\mid S=k]\). The observed difference estimates that quantity only under assumptions.
The key requirements include consistency (the observed outcome equals the potential outcome under the treatment actually received), no interference between individuals, and conditional exchangeability within the stratum: \(Y(1),Y(0)\) are independent of \(T\) given \(S=k\). Both treatment groups must also occur in the stratum, so the two means can be estimated. In observational data, conditional exchangeability is an untestable assumption about measured and unmeasured confounding. Moreover, grouping people by estimated propensity scores only approximates adjustment: individuals with similar scores can still differ in covariates that predict the outcome.
The identification logic is short. Under consistency, the observed outcome for treated individuals is \(Y(1)\), and the observed outcome for untreated individuals is \(Y(0)\). Under conditional exchangeability within stratum \(k\), treatment status does not distinguish the potential-outcome distributions within that stratum. Therefore:
The equality is a population identification statement under the assumptions, not a guarantee that two finite-sample means equal the causal effect exactly. The calculated \(\hat{\Delta}_k\) is a sample estimate, subject to sampling variability and possible residual confounding. If exchangeability holds only conditional on the full covariate vector \(X\), it does not automatically follow that it holds conditional on the coarser stratum label \(S\).
Worked Example: Two Strata and a Negative Difference
Suppose a hypothetical study measures a symptom score after treatment; lower scores indicate fewer symptoms. Individuals have already been assigned to low- and high-propensity-score strata. The recorded scores are:
| Stratum | Treated scores | Untreated scores |
|---|---|---|
| Low propensity | 8, 10, 9 | 11, 12, 10 |
| High propensity | 6, 7, 8 | 10, 9, 11 |
In the low-propensity stratum, the treated mean is \((8+10+9)/3=27/3=9\) score points. The untreated mean is \((11+12+10)/3=33/3=11\) score points. Thus,
In the high-propensity stratum, the treated mean is \((6+7+8)/3=21/3=7\) points, and the untreated mean is \((10+9+11)/3=30/3=10\) points. Therefore,
Both observed contrasts favor treatment in the descriptive sense that the treated group has a lower mean symptom score. These calculations do not establish that treatment caused either reduction. A causal interpretation would additionally require the assumptions above to be credible within each stratum. The results are two stratum-specific differences; no overall effect has been calculated.
Work Carefully From Counts and Sums
In practice, a data table may present counts and outcome totals rather than every individual outcome. The same definition applies: divide each group's outcome sum by its own count, then subtract. Do not use a common denominator for the two means, and do not subtract totals without accounting for group sizes.
Worked Example: Unequal Group Sizes Within Strata
Consider hypothetical recovery-day outcomes, where fewer days indicate faster recovery. The table gives the number of people and the total recovery days in each treatment group. The strata were formed before these outcome summaries were calculated.
| Stratum | Treated: count | Treated: total days | Untreated: count | Untreated: total days |
|---|---|---|---|---|
| A | 4 | 24 | 5 | 40 |
| B | 3 | 27 | 4 | 44 |
| C | 5 | 60 | 2 | 28 |
For stratum A, the treated mean is \(24/4=6\) days and the untreated mean is \(40/5=8\) days. The difference is \(6-8=-2\) days. For stratum B, the means are \(27/3=9\) and \(44/4=11\) days, giving \(9-11=-2\) days. For stratum C, the means are \(60/5=12\) and \(28/2=14\) days, giving \(12-14=-2\) days.
All three within-stratum observed differences are \(-2\) recovery days. The unequal counts do not change how each mean is calculated: each group has its own denominator. These results still do not specify an overall treatment effect, nor do equal observed differences demonstrate that the effect is truly homogeneous across strata.
Diagnostics and Failure Modes
Before interpreting a stratum-specific difference, check that the arithmetic is defined and that the comparison has useful support. For every stratum, verify the treated and untreated counts, confirm the outcome units and coding, and inspect whether a very small group is making its mean unstable. A stratum with one treated person has a computable mean, but that mean is just one observed outcome and may be highly sensitive to that person's value.
An empty treatment group is a more fundamental problem. If \(n_{1k}=0\), the treated mean is undefined; if \(n_{0k}=0\), the untreated mean is undefined. In either case, a within-stratum difference cannot be computed from the observed data. Do not treat the missing mean as zero or silently borrow a mean from another stratum. The absence of one group indicates a lack of within-stratum overlap and limits what these data can tell us about a treatment contrast there.
Also examine whether the strata actually balance important covariates. Propensity-score strata are useful only insofar as they make treated and untreated individuals more comparable. Compare covariate distributions between treatment groups within strata, not merely the overall propensity-score ranges. Noticeable within-stratum imbalance warns that residual confounding may remain. This is a diagnostic, not proof that all confounding has been removed if the measured covariates appear balanced.
Record \(n_{1k}\) and \(n_{0k}\) for every stratum. A mean requires a positive count.
Divide the outcome total by the corresponding treated or untreated count; retain the outcome's units.
Use treated mean minus untreated mean, and interpret the sign in light of whether larger outcomes are favorable or unfavorable.
Report an observed within-stratum contrast; describe it causally only if the identification assumptions are defensible.
Worked Example: An Empty Treatment Group
In a hypothetical study of days with a target symptom, lower is better. Within a low-score stratum, the treated outcomes are \(5\), and the untreated outcomes are \(7,8,9\). Within a middle stratum there are no treated individuals and the untreated outcomes are \(6,7,8\). Within a high stratum, treated outcomes are \(4,5,6\), and there are no untreated individuals.
For the low stratum, \(\bar{Y}_{1,\text{low}}=5/1=5\) days and \(\bar{Y}_{0,\text{low}}=(7+8+9)/3=24/3=8\) days. Thus \(\hat{\Delta}_{\text{low}}=5-8=-3\) days.
For the middle stratum, \(n_{1,\text{middle}}=0\), so the treated mean would require division by zero. There is no observed within-stratum difference. For the high stratum, \(n_{0,\text{high}}=0\), so the untreated mean is undefined and again no difference can be calculated. The correct report is that a contrast is estimable in the low stratum, but not in the middle or high stratum from these data. A causal effect for those strata cannot be recovered by assigning a numerical value to the missing mean.
How This Calculation Relates to Other Approaches
An unadjusted comparison subtracts the overall untreated mean from the overall treated mean. It is easy to calculate, but it does not account for differences in the distribution of propensity-score strata between the treatment groups. Within-stratum comparisons make those comparisons separately and reveal whether contrasts vary across the strata. They do not, on their own, specify how those separate contrasts should be combined into an overall target effect.
Matching instead constructs comparisons between treated and untreated individuals according to a matching rule, while weighting changes individuals' contributions through weights. Regression adjustment models the outcome in relation to treatment and covariates. Each approach uses different structure and has its own assumptions and diagnostics. The calculation here is deliberately narrower: it reports the two group means and their difference within each already-defined stratum. It neither matches individuals nor applies weights nor fits an outcome model.
A common mistake is to say, “The propensity scores are similar in this stratum, so the difference must be causal.” Similar estimated treatment probabilities do not guarantee balance in every confounder, correct propensity-score estimation, or conditional exchangeability. The correction is to report the arithmetic as an observed contrast, check within-stratum covariate balance and group support, and state the assumptions required for a causal interpretation. Another mistake is to use a stratum-specific difference as though it were automatically the overall average treatment effect. It is not: it pertains to a particular stratum, and an overall estimand requires an explicit combination rule and target population.
Check Your Understanding
Answer each question, then compare with the short solutions below.
- In one stratum, treated outcomes are \(3,5,4\), and untreated outcomes are \(6,5,7,6\). Calculate both means and the treated-minus-untreated difference.
- For an outcome where larger values are better, a stratum-specific difference is \(-1.5\) outcome units. What does the sign mean descriptively?
- Why is a within-stratum difference undefined when the stratum contains no untreated individuals?
- Does similar propensity-score membership within a stratum by itself establish that the observed difference is causal? Explain.
Solutions: (1) The treated mean is \((3+5+4)/3=12/3=4\); the untreated mean is \((6+5+7+6)/4=24/4=6\). The difference is \(4-6=-2\) outcome units. (2) The treated group's observed mean is 1.5 units lower than the untreated group's observed mean; because larger is better, the descriptive contrast favors the untreated group. (3) The untreated mean requires division by the untreated count, which is zero, so it cannot be estimated from observations in that stratum. (4) No. Similar propensity scores do not guarantee exchangeability or eliminate residual confounding; a causal interpretation requires defensible assumptions, including conditional exchangeability within the stratum.