Compare Outcome Regression and Propensity Weighting

Why Compare Two Adjustment Approaches?

For an observational comparison, people with different exposure histories may also differ in characteristics that predict the outcome. Regression adjustment and propensity weighting address measured differences in different ways: regression models the outcome, while propensity weighting reweights people according to their probability of receiving the exposure they actually received. Comparing them is useful, but the two estimates are meaningful as a comparison only when their target population and effect measure are aligned.

We use \(A=1\) for exposure and \(A=0\) for no exposure, \(Y\) for the outcome, and \(X\) for measured pre-exposure covariates. Let \(Y(1)\) and \(Y(0)\) denote the potential outcomes for a person under exposure and no exposure. For a binary outcome, this tutorial uses the population-average risk difference as the effect measure.

Definition: The average treatment effect (ATE) in a target population is \(E[Y(1)-Y(0)]\): the population mean outcome under exposure minus the population mean outcome under no exposure. The propensity score is \(e(X)=P(A=1\mid X)\), the probability of exposure given measured covariates. The outcome regression function is \(m_a(X)=E[Y\mid A=a,X]\), the mean observed outcome at exposure level \(a\) and covariate values \(X\).

The ATE is causal only under assumptions that connect observed outcomes to potential outcomes. In particular, consistency requires that a person's observed outcome equals the potential outcome corresponding to their actual exposure. Conditional exchangeability requires \((Y(1),Y(0))\) to be independent of \(A\) given \(X\); in words, the measured covariates must suffice to remove confounding. Positivity requires a nonzero probability of each exposure level at covariate profiles represented in the target population. These assumptions are substantive: balance checks and model fit do not establish that there is no unmeasured confounding.

Two Ways to Estimate the Same Population Effect

With outcome regression, estimate the mean outcome at each exposure level for each covariate profile, then average those predictions over the target population's covariate distribution. This is also called regression standardization. For an ATE, the calculation is

\widehat{\mathrm{ATE}}_{\mathrm{OR}} = \frac{1}{n}\sum_{i=1}^{n} \left\{\widehat m_1(X_i)-\widehat m_0(X_i)\right\}.

Here, \(\widehat m_a(X_i)\) is the fitted outcome model's predicted mean for person \(i\) if their exposure were set to \(a\). The prediction is made twice for each person, once at \(A=1\) and once at \(A=0\), and the contrasts are averaged. The model need not be a particular kind of regression, but its predictions must estimate the relevant conditional outcome means adequately.

Propensity weighting instead gives greater weight to people whose observed exposure was less likely given their covariates. For an ATE, the inverse-probability weights are \(1/e(X_i)\) for exposed people and \(1/[1-e(X_i)]\) for unexposed people. A normalized, or Hájek, estimate of the risk under each exposure level is

\widehat\mu_1^{\,W} = \frac{\sum_i A_iY_i/\widehat e(X_i)} {\sum_i A_i/\widehat e(X_i)}, \qquad \widehat\mu_0^{\,W} = \frac{\sum_i (1-A_i)Y_i/[1-\widehat e(X_i)]} {\sum_i (1-A_i)/[1-\widehat e(X_i)]}, \qquad \widehat{\mathrm{ATE}}_W=\widehat\mu_1^{\,W}-\widehat\mu_0^{\,W}.

The normalized weighted means estimate the two outcome risks in a reweighted population. With a correctly specified propensity model, adequate positivity, and the causal identification assumptions, weighting aims to make the measured covariate distribution comparable across exposure groups for an ATE analysis. The estimate still depends on the propensity scores being estimated well enough; very small or large scores can produce very large weights.

Why Weighting Can Recover a Population Mean

The weighting idea follows from a simple conditional expectation. For people with covariates \(X\), the probability of being exposed is \(e(X)\). Multiplying an exposed person's outcome by \(1/e(X)\) offsets the lower representation of exposed people in covariate profiles where exposure is uncommon.

E\left[\frac{AY}{e(X)}\middle|X\right] = \frac{E[AY\mid X]}{e(X)} = \frac{P(A=1\mid X)E[Y\mid A=1,X]}{e(X)} = m_1(X).

The second equality uses the fact that \(AY=0\) when \(A=0\), so \(E[AY\mid X]=P(A=1\mid X)E[Y\mid A=1,X]\). The last equality substitutes \(P(A=1\mid X)=e(X)\). Averaging over the target population's covariate distribution gives \(E[m_1(X)]\), the mean outcome under exposure when the identification assumptions hold. The corresponding calculation for unexposed people uses \((1-A)/[1-e(X)]\) and recovers \(E[m_0(X)]\). Their difference is the ATE.

This argument explains the population target, not a guarantee that finite-sample estimates will agree exactly. Estimated scores, limited sample sizes, model choices, and the realized data all affect the estimates.

Worked Example: Matching ATE Estimates in Two Covariate Strata

Suppose a hypothetical sample of 200 people has two covariate strata, \(X=0\) and \(X=1\), with 100 people in each. In stratum \(X=0\), 20 are exposed and 80 are unexposed; their outcome counts are 2 and 16, respectively. In stratum \(X=1\), 80 are exposed and 20 are unexposed; their outcome counts are 24 and 8. Assume for this calculation that the propensity scores are \(0.20\) in stratum 0 and \(0.80\) in stratum 1. The target population contains equal proportions from the two strata.

The observed risks within the strata are \(2/20=0.10\) and \(24/80=0.30\) among exposed people, and \(16/80=0.20\) and \(8/20=0.40\) among unexposed people. Standardizing both exposure-specific risks to the equal-stratum target gives

\widehat\mu_1^{\,\mathrm{OR}}=0.50(0.10)+0.50(0.30)=0.20, \qquad \widehat\mu_0^{\,\mathrm{OR}}=0.50(0.20)+0.50(0.40)=0.30,

so the standardized risk difference is \(0.20-0.30=-0.10\), or 10 percentage points lower risk under exposure in this hypothetical estimate.

For weighting, the exposed outcome total weighted by inverse propensity is \(2/0.20+24/0.80=10+30=40\). The total exposed weight is \(20/0.20+80/0.80=100+100=200\). Thus the weighted exposed risk is \(40/200=0.20\). For the unexposed group, the weighted outcome total is \(16/0.80+8/0.20=20+40=60\), and the total weight is \(80/0.80+20/0.20=100+100=200\). The weighted unexposed risk is \(60/200=0.30\), giving a weighted risk difference of \(0.20-0.30=-0.10\).

The agreement is expected here: both calculations use the same target distribution, and the stratum-specific risks and propensity scores line up exactly. The assumed positivity condition is satisfied because both exposure probabilities are strictly between 0 and 1 in both strata. The arithmetic does not verify consistency or conditional exchangeability; those remain assumptions about the hypothetical study and covariates.

Check That the Estimands Match

Propensity weighting does not always target the ATE. A common alternative is weighting for the average treatment effect among the treated (ATT), which asks about the effect in the population that actually received exposure. The regression predictions can also be averaged over that treated population. ATE and ATT can differ when exposure effects vary across covariate profiles, even if both approaches are implemented correctly.

Worked Example: ATE and ATT Need Not Be Equal

Consider a hypothetical target population that is split equally between two covariate strata. Suppose the causal risk difference is \(-0.10\) in stratum \(X=0\) and \(-0.30\) in stratum \(X=1\). Assume the exposed population is 20% in stratum 0 and 80% in stratum 1.

The ATE averages the two stratum effects using the full population's 50%/50% distribution:

\mathrm{ATE}=0.50(-0.10)+0.50(-0.30)=-0.05-0.15=-0.20.

The ATT averages those same effects using the exposed population's 20%/80% distribution:

\mathrm{ATT}=0.20(-0.10)+0.80(-0.30)=-0.02-0.24=-0.26.

The estimates differ by 0.06, or 6 percentage points, because they average heterogeneous effects over different populations. The difference is not evidence that one calculation failed. Before comparing numerical results, state the target population and effect measure for each.

Interpreting Disagreement Between Estimates

When regression standardization and weighting target the same ATE but give different estimates, the disagreement is a reason to investigate, not a rule for choosing a winner. Both methods rely on conditional exchangeability, consistency, and positivity. They also make different modeling demands: regression standardization relies on the outcome model's conditional means, whereas propensity weighting relies on the propensity model and can be sensitive to extreme weights.

Worked Example: A Same-Target Disagreement

Suppose two analyses of the same hypothetical sample target the same population-average risk difference and adjust for the same measured covariates. Regression predictions, averaged over the target sample, give a risk of 0.212 under exposure and 0.288 under no exposure. The standardized risk difference is

$$ 0.212-0.288=-0.076. $$

The propensity-weighted calculation gives an exposed weighted outcome total of 21.6 with total exposed weight 108, and an unexposed weighted outcome total of 27.9 with total unexposed weight 90. Therefore,

\widehat\mu_1^{\,W}=\frac{21.6}{108}=0.200, \qquad \widehat\mu_0^{\,W}=\frac{27.9}{90}=0.310, \qquad \widehat{\mathrm{ATE}}_W=0.200-0.310=-0.110.

The point estimates differ by \(-0.110-(-0.076)=-0.034\), or 3.4 percentage points. The discrepancy alone does not establish which estimate is closer to the causal effect, and neither estimate proves that unmeasured confounding is absent. A careful comparison checks that the target population and outcome scale truly match, examines covariate balance after weighting, and checks whether poor overlap or very large weights make the weighted estimate unstable. It also recognizes that the outcome and propensity models may represent their respective relationships imperfectly.

Common Mistakes and Practical Meaning

“The methods disagree, so one must be wrong.” Finite-sample variation and model dependence can produce different estimates. Investigate the cause; do not choose one just because its result is preferred.
Comparing different targets. An ATE and an ATT answer different questions. Specify the population distribution over which effects are averaged before comparing estimates.
Treating balance as proof of no confounding. Balance checks concern measured covariates. They cannot show that all confounders were measured or that the causal assumptions hold.
Ignoring extreme weights. Propensity scores near 0 or 1 can give some observations enormous influence. Positivity and overlap matter for stability as well as identification.
Reading a coefficient as the standardized ATE. A regression exposure coefficient need not equal a population-average risk difference. Standardization explicitly predicts both exposure scenarios for the target population and averages their contrast.

These approaches are useful complements: one models outcomes directly, while the other constructs a weighted comparison. Agreement can be reassuring about sensitivity to these modeling choices, but it does not establish causal validity. Disagreement can reveal sensitivity worth reporting, but does not by itself identify which assumptions or model features are responsible.

Check Your Understanding

Answer each question in terms of the target population, outcome scale, and assumptions.

In one sentence, distinguish regression standardization from propensity weighting.
What does the ATE average over, and how can that target differ from the ATT?
Name the three causal identification assumptions stated in this tutorial. Which one is not established by a measured-covariate balance check?
Why can a propensity score near zero create an unstable weighted estimate for the exposed group?
Two estimates target the same ATE but disagree. Give two possible reasons and explain why the disagreement alone cannot identify the correct estimate.