Comparing Weighted and Subclassification Estimates

Why Compare These Estimates?

Inverse-probability weighting and propensity-score subclassification can both estimate an average treatment effect (ATE), but they do not necessarily produce the same number in a finite sample. Comparing them is useful because the difference can reveal how much the result depends on using individual-level propensity scores versus averages within coarser strata. It can also expose differences in the population weights the calculations apply.

The comparison is meaningful only when the estimand and analysis population are held fixed. Here the target is the ATE for the study population: the average difference between each unit’s potential outcome under treatment and its potential outcome under control. Assume a binary treatment \(A_i\), observed outcome \(Y_i\), pretreatment covariates \(X_i\), and propensity score \(e(X_i)=P(A_i=1\mid X_i)\). The observed outcome is \(Y_i=A_iY_i(1)+(1-A_i)Y_i(0)\).

For both estimates to have a causal interpretation, the relevant identifying assumptions are consistency, conditional exchangeability \(Y(a)\perp A\mid X\) for \(a=0,1\), and positivity \(0<e(X_i)<1\) for covariate values in the target population. Consistency connects observed outcomes to potential outcomes; exchangeability says that, conditional on the measured covariates, treatment assignment is independent of the potential outcomes; positivity requires a possibility of either treatment at each relevant covariate value. The propensity-score model must also estimate scores adequately for the weighting procedure. Subclassification requires its strata to control confounding sufficiently well; grouping scores into strata does not guarantee that covariates are balanced within them.

Two Estimators and Their Target Weights

Let \(S_i=k\) indicate that unit \(i\) belongs to stratum \(k\). Write \(n_k\) for the number of units in stratum \(k\), \(n_{1k}\) and \(n_{0k}\) for its treated and control counts, and \(\bar{Y}_{1k}\) and \(\bar{Y}_{0k}\) for the corresponding observed outcome means. With all strata represented in both treatment groups, the usual sample ATE subclassification estimate is

The stratum effect is weighted by \(n_k/n\), the fraction of the study population in that stratum. Thus, this estimator targets the study population’s stratum distribution, not an equally weighted average across strata unless the strata happen to have equal sizes.

One common ATE inverse-probability-weighted estimator is the Horvitz–Thompson form:

In this formula, each treated outcome receives weight \(1/\hat{e}(X_i)\), and each control outcome receives weight \(1/[1-\hat{e}(X_i)]\). Division by \(n\) puts the weighted contrast on the study-population scale. A normalized, or Hájek, version instead divides the weighted treated and control outcome totals by their respective total weights before taking the difference. The two versions can differ in finite samples, so a comparison should state which version is used.

The weighting formula applies the target-population weight at the individual level. Subclassification first forms an outcome contrast in each stratum, then applies the stratum’s population share. This distinction matters when estimated scores vary substantially within strata, when strata have unequal sizes, or when treatment effects differ across strata.

Worked Comparison on One Hypothetical Dataset

When the Estimates Coincide

A useful special case gives a direct algebraic link between these methods. Suppose each unit in stratum \(k\) receives the same estimated score, and that score is the observed treatment fraction in that stratum: \(\hat{e}_i=n_{1k}/n_k\) for units in stratum \(k\). Then the Horvitz–Thompson IPW estimate is exactly the subclassification estimate.

To see why, consider the treated contribution from stratum \(k\). There are \(n_{1k}\) treated units, so dividing each treated outcome by \(n_{1k}/n_k\) multiplies their total by \(n_k/n_{1k}\). Dividing the resulting sum by the full sample size \(n\) gives \(n_k/n\) times the treated mean. The same reasoning applies to the control contribution, using \(1-\hat{e}_i=n_{0k}/n_k\). Subtracting these two terms and summing across strata produces the subclassification formula:

Taking the difference for each stratum and summing gives \(\hat{\tau}_{\mathrm{IPW}}=\hat{\tau}_{\mathrm{sub}}\). This is an algebraic identity under the stated score choice, not a general guarantee that independently fitted individual-level scores will reproduce subclassification. The identity also applies to the unnormalized Horvitz–Thompson estimator as written; normalized weighted means need not have the same finite-sample equality.

Unequal Stratum Sizes Determine the Target Weights

The stratum weights are part of the estimand calculation, not a cosmetic choice. For the study-population ATE, a stratum containing 75% of the study units receives three times the weight of one containing 25%, regardless of how many strata are used. Equal weighting across strata would instead target an artificial population that gives every stratum the same share.

Diagnostics, Interpretation, and Limitations

When estimates disagree, begin by checking that both calculations use the same outcome scale, analysis sample, target population, and estimand. Then inspect the mechanisms that can create differences:

• Within-stratum balance: Subclassification controls confounding only to the extent that relevant covariates are balanced within each stratum. Similar scores alone do not prove that every covariate is balanced.
• Individual weights: IPW can be strongly affected by observations with large inverse-probability weights. Check the range and distribution of weights and whether a small number of observations dominate the weighted totals.
• Stratum contributions: Report each stratum’s size, treated and control counts, estimated contrast, and contribution to the overall estimate. A large change in one stratum can explain much of the difference.
• Score specification: Compare the score estimates used for weighting with the score groupings used for subclassification. Coarse strata discard within-stratum score variation; weighting retains it.
• Uncertainty: A difference between point estimates does not establish that the methods estimate different causal effects. Compare uncertainty using an inference procedure appropriate to each estimator; if using a bootstrap, repeat the propensity-score estimation and analysis steps within each resample.

Treatment-effect heterogeneity can make the role of weights especially visible. If stratum-specific effects differ, changing the weights changes their average. Conversely, even when effects are constant, finite-sample variation, imperfect score estimation, and residual confounding within strata can lead to differing estimates. Neither estimate should be selected merely because it is larger, smaller, or more appealing.

Both approaches have limitations. Subclassification can leave residual confounding within strata and its estimate can depend on how strata are formed. IPW uses individual scores but can be sensitive to extreme weights and model misspecification. In either case, causal interpretation still relies on the identifying assumptions; computation alone cannot establish exchangeability or positivity. The introductory material on propensity-score subclassification and ATE weighting supplies the procedures, while this comparison emphasizes that their weights encode how the target population is represented.

Common Mistake