Common Errors in Subclassification Aggregation

Why Aggregation Is Part of the Estimand

Subclassification produces a treatment-effect estimate for each propensity-score stratum. The overall estimate is not determined merely by those stratum estimates: it also depends on how they are weighted. Equal-weighting strata, weighting them by their share of the full population, and weighting them by their share of the treated population answer different questions when stratum effects vary.

This tutorial focuses on common aggregation errors. It assumes that treatment is binary, with \(A=1\) for treated and \(A=0\) for control, and that the analysis has formed \(K\) strata, indexed by \(k=1,\ldots,K\). Let \(Y(1)\) and \(Y(0)\) denote the potential outcomes under treatment and control. For a target population with stratum proportions \(q_k\), the average treatment effect in that population is the average of the stratum-specific effects using those same proportions.

Definition: Let \(\tau_k=E[Y(1)-Y(0)\mid S=k]\) be the treatment effect in stratum \(k\), where \(S\) denotes stratum membership. For a target population whose proportion in stratum \(k\) is \(q_k\), with \(\sum_{k=1}^K q_k=1\), the corresponding aggregated effect is \(\tau(q)=\sum_{k=1}^K q_k\tau_k\). In a sample, an estimated aggregate is \(\hat{\tau}(q)=\sum_{k=1}^K q_k\hat{\tau}_k\), where \(\hat{\tau}_k\) is the estimated effect in stratum \(k\).

The weights are part of the estimand, not a cosmetic calculation choice. For the average treatment effect (ATE) in the full study population, \(q_k\) is the proportion of that population in stratum \(k\). For the average treatment effect among the treated (ATT), \(q_k\) is the proportion of treated individuals in stratum \(k\). A different target population calls for its own stratum proportions.

Why Target-Population Weights Are Required

The weighting rule follows directly from the definition of an average over a population partitioned into strata. In the derivation below, \(q_k=P(S=k)\) refers to the stratum proportions in the target population. The identity applies to the target estimand; causal interpretation of each stratum effect additionally requires the relevant identification assumptions.

\begin{aligned} \tau(q) &= E[Y(1)-Y(0)]\\ &= \sum_{k=1}^{K}P(S=k)\,E[Y(1)-Y(0)\mid S=k]\\ &= \sum_{k=1}^{K}q_k\tau_k. \end{aligned}

The second line partitions the target population into mutually exclusive strata: the overall mean is the sum of each stratum’s mean contribution, weighted by the probability of belonging to that stratum. Replacing \(q_k\) with \(1/K\) estimates an equally weighted average of stratum effects. That equals the target-population effect only when the target stratum proportions are equal, or when the stratum effects happen to make the different weighted averages coincide.

In applications, \(\tau_k\) is not directly observed. A common estimate is \(\hat{\tau}_k=\bar{Y}_{1k}-\bar{Y}_{0k}\), the treated-minus-control outcome mean within stratum \(k\). Interpreting these contrasts causally requires consistency, exchangeability of treatment assignment within the relevant strata, and treatment positivity within strata. In particular, creating broad propensity-score strata may not remove all within-stratum confounding; stratification does not guarantee exchangeability simply because it was performed.

Name the target.
Decide whether the estimand concerns the full study population, treated individuals, or another specified population.

Calculate its stratum proportions.
Use the number or proportion of target-population members in each stratum. Check that the weights sum to 1.

Aggregate the stratum effects.
Multiply each stratum effect estimate by its target weight, then add the weighted contributions.

Check interpretation and diagnostics.
Report the target and weights, inspect stratum-specific estimates and sample sizes, and verify that treated and control observations support each within-stratum contrast.

Worked Aggregations

Worked Example: Equal Weighting Changes the ATE

Suppose a hypothetical study has two strata. Stratum 1 contains 100 of 400 individuals and has a treated-minus-control mean outcome difference of 10 units. Stratum 2 contains 300 individuals and has a difference of 2 units. Assume, for this calculation, that the within-stratum contrasts identify the stratum effects.

The full-population stratum proportions are \(q_1=100/400=0.25\) and \(q_2=300/400=0.75\). The ATE estimate is therefore

\hat{\tau}_{ATE} =0.25(10)+0.75(2) =2.5+1.5 =4.0\text{ units}.

An equal-weight calculation instead gives \((10+2)/2=6.0\) units. It gives the small stratum the same influence as the stratum containing three times as many target-population members. That 6.0-unit result is not the full-population ATE for these data; it is the equally weighted average of the two stratum effects. Equal weighting would be appropriate only for a target population with equal proportions in these strata, or for an explicitly defined equal-stratum estimand.

Worked Example: Using Treated-Population Weights for the ATT

Use the same hypothetical stratum effects, 10 units in stratum 1 and 2 units in stratum 2. Now suppose the study has 80 treated individuals in stratum 1 and 30 treated individuals in stratum 2. The ATT targets treated individuals, so its weights come from the treated group, not from everyone in the study.

There are \(80+30=110\) treated individuals. Their stratum proportions are \(q_{1,1}=80/110\approx0.7273\) and \(q_{1,2}=30/110\approx0.2727\). Thus,

\hat{\tau}_{ATT} =\frac{80}{110}(10)+\frac{30}{110}(2) =\frac{800+60}{110} \approx7.818\text{ units}.

Using the full-population proportions instead would give \(0.25(10)+0.75(2)=4.0\) units, the ATE calculation from the preceding example. That is not the ATT aggregation: it gives most of the weight to stratum 2 even though most treated individuals are in stratum 1. The two answers differ because they target different populations, not because one set of weights is universally correct.

Worked Example: Pooled Means Can Reverse the Stratified Contrast

Consider hypothetical data with two equally sized strata, each containing 100 individuals. In stratum 1, 90 are treated and 10 are controls; their mean outcomes are 10 and 8 units, respectively. In stratum 2, 10 are treated and 90 are controls; their mean outcomes are 20 and 18 units. Within each stratum, the treated-minus-control difference is \(2\) units.

For the full population, the stratum weights are \(100/200=0.5\) each. The standardized estimate is

\hat{\tau}_{ATE} =0.5(10-8)+0.5(20-18) =0.5(2)+0.5(2) =2.0\text{ units}.

But the pooled treated mean uses 90% weight on stratum 1 and 10% on stratum 2: \((90/100)(10)+(10/100)(20)=11\) units. The pooled control mean uses 10% weight on stratum 1 and 90% on stratum 2: \((10/100)(8)+(90/100)(18)=17\) units. Their unadjusted difference is \(11-17=-6\) units. This is not the standardized ATE: it compares groups with different stratum compositions. The arithmetic reversal does not establish a negative within-stratum effect; both stratum contrasts are positive. A pooled difference and a target-standardized contrast use different weights and can answer different questions.

Common Errors and How to Diagnose Them

Giving every stratum equal weight by default. This silently changes the target to an equal-stratum average. Compare the weights used with the proportions in the population named by the estimand.
Using the full-sample proportions for every estimand. Full-sample proportions are appropriate for a full-population ATE, not automatically for the ATT. Calculate proportions in the target population, such as the treated group for the ATT.
Pooling treated and control means before standardizing. The treated and control groups may have different stratum distributions. Their pooled mean difference therefore does not generally equal a weighted average of within-stratum effects using one common target distribution.
Averaging effects without stating the target. A numeric weighted average is not fully interpreted until the population represented by the weights is clear.
Ignoring empty or sparse treatment cells. If a stratum contains no treated individuals or no controls, its within-stratum contrast cannot be estimated from the observed data. Very small cells can make estimates unstable. Do not conceal the problem by silently dropping the stratum or changing the weights; report the issue and reassess the estimand and analysis.

A practical audit is to write down the target population, list each stratum’s target count, divide by the total target count, and verify that the resulting weights sum to one. Then compute each weighted contribution separately. Inspecting \(\hat{\tau}_k\) alongside its sample sizes can reveal whether the aggregate is driven by one small, imprecisely estimated stratum. A valid-looking set of weights does not itself establish that the stratum contrasts are causally identified.

How Aggregation Relates to Other Approaches

Direct standardization makes the target distribution explicit: estimate stratum-specific outcomes or effects, then average them using the target population’s stratum proportions. Inverse probability weighting instead uses propensity-score-based weights to construct a weighted comparison for a specified target estimand. Regression standardization models outcomes and averages predictions over a target population. These approaches differ in how they adjust for covariates and how they represent the target, but each still requires a clear estimand and appropriate assumptions.

Subclassification offers a transparent view of effect variation across score strata, but it replaces a continuous score with groups. A single aggregate can obscure heterogeneity, and finite strata may leave residual differences in covariates within groups. Weight selection fixes neither residual confounding nor a lack of treated or control observations in important strata. Standard errors and uncertainty calculations must also reflect the estimation procedure; the point-estimate formula alone does not provide a confidence interval.

Common Mistake: “The Strata Are the Same Size, So Equal Weighting Is Fine”

The relevant sizes are those in the target population, not necessarily the sizes in an analysis table, a matched subset, or one treatment arm. Equal weighting is justified for a particular estimand only when the target distribution has equal stratum proportions, or when it produces the same numerical average by coincidence. Always identify the target first, then derive the weights from it.

Key takeaway: Aggregate stratum-specific effects with the proportions of strata in the population the estimand targets. Equal weights, full-sample weights, and treated-group weights are not interchangeable; choosing among them defines which average effect is being estimated.

Check Your Understanding

For each question, identify the appropriate calculation before checking the solutions below.

Two strata have estimated effects of 5 and 1 units. They contain 25% and 75% of the full population. What is the full-population aggregate?
In a two-stratum study, 40% of treated individuals are in stratum 1 and 60% are in stratum 2. The stratum effects are 3 and 7 units. What is the ATT aggregate?
When is an equal-weighted average of stratum effects appropriate as a target-population effect?
Why can the pooled treated-minus-control mean difference differ in sign from all the within-stratum differences?
A stratum contains controls but no treated individuals. Can its treated-minus-control stratum contrast be estimated from the observed data? What should an analyst avoid doing?

Solutions. (1) \(0.25(5)+0.75(1)=1.25+0.75=2.0\) units. (2) \(0.40(3)+0.60(7)=1.2+4.2=5.4\) units. The weights are treated-group proportions because the target is the treated population. (3) It is appropriate when the target population has equal proportions in the strata, or when equal weighting happens to give the same value as the target-weighted average. (4) The pooled treated and control means can use different stratum proportions. Their difference then mixes within-stratum contrasts with differences in stratum composition, allowing a reversal. (5) No. The treated mean, and therefore the within-stratum contrast, is not observed for that stratum. The analyst should not silently drop the stratum or alter the weights while presenting the result as the original target effect; the limitation and any change in target must be made explicit.