Workshop: Analyze an Observational Treatment Comparison

From an Observational Comparison to a Defined Question

A treatment comparison is not interpretable until its question is precise. Which treatment strategies are being compared? What outcome is measured, and when? For which population is the contrast intended? In observational data, people receiving different treatments may also differ in health or other baseline characteristics. An unadjusted difference can therefore mix a treatment contrast with differences that existed before treatment.

This workshop follows a hypothetical cohort of 200 eligible clinic patients. Some received a new program and others received usual care. The outcome, measured 12 weeks later, is improvement in a symptom score, in points; larger improvements are better. Baseline severity is classified as high or low. The example is deliberately small in its structure so that each adjustment calculation can be checked by hand. It is not evidence about a real program.

For this workshop, the target is the full eligible cohort represented by these 200 patients, and the estimand is the cohort ATE in symptom-improvement points at 12 weeks. Choosing the ATE does not make it identifiable by itself. A causal interpretation requires assumptions. In particular, treatment and outcome must be defined consistently, measured baseline covariates must suffice to control confounding, and both treatments must be possible within each relevant covariate pattern. We will assess the last condition for the measured severity groups, but cannot establish from these data alone that every confounder has been measured.

Worked Example: Define the Estimand and Inspect the Crude Difference

The table gives the hypothetical patient counts and mean symptom improvements within each treatment and severity group. The severity measurements are baseline measurements, taken before treatment.

The cohort contains \(48+30=78\) high-severity patients and \(32+90=122\) low-severity patients. The new-program group has a larger share of high-severity patients: \(48/80=0.600\), compared with \(30/120=0.250\) in usual care.

The crude mean improvement in the program group is \( (48\times7+32\times11)/80=(336+352)/80=8.60\) points. For usual care it is \( (30\times5+90\times7)/120=(150+630)/120=6.50\) points. The crude difference is \(8.60-6.50=2.10\) points in favor of the program. This is a descriptive difference between the observed groups, not yet an estimate that isolates treatment from their different severity composition.

The target remains the average effect for all 200 patients, not merely the effect for those who received the program. In this example, the target population’s severity proportions are \(78/200=0.390\) high and \(122/200=0.610\) low. Those proportions will provide a common baseline distribution for standardizing both treatments.

Check Overlap Before Adjusting

The propensity score is the probability of receiving treatment given measured baseline covariates. Here, use baseline severity \(C\) alone, with \(C=1\) for high severity and \(C=0\) for low severity. The estimated score in each category is the observed proportion receiving the program:

Both estimated probabilities are strictly between 0 and 1. Each severity category contains patients in both treatment groups, so there is empirical overlap for this measured covariate. This does not show that overlap holds for all relevant patient characteristics or that the exchangeability assumption is true. If a category contained only treated patients or only controls, the data would not support an observed comparison within that category without extrapolation or a change in target population.

A balance check asks whether the groups have comparable measured baseline covariates after adjustment. Before adjustment, the high-severity proportions are 0.600 and 0.250, a difference of 0.350. One descriptive standardized mean difference (SMD) for this binary covariate divides that difference by the pooled standard deviation of the binary indicator:

The covariate is clearly imbalanced in this example. An SMD describes the magnitude of a covariate difference on a standardized scale; it is not a significance test, and a small SMD would not prove that confounding has been removed. Balance must be assessed for relevant measured covariates, not just the one chosen for this illustration.

Worked Example: Standardize to the Target Population

Direct standardization estimates the mean outcome under each treatment by applying the same target-population covariate distribution to the treatment-specific outcome means. Here, the target distribution is 0.390 high severity and 0.610 low severity. Applying it first to the program outcomes gives

Applying the same distribution to usual-care outcomes gives

The standardized contrast is \(9.44-6.22=3.22\) points. Under the identification assumptions, this estimates the cohort ATE. The result differs from the crude contrast of 2.10 points because the crude groups had different severity distributions. Standardization compares the treatment-specific means as if both groups had the cohort’s same severity mix.

The within-severity observed contrasts are 7 minus 5, or 2 points, among high-severity patients, and 11 minus 7, or 4 points, among low-severity patients. The ATE averages these contrasts using the target population’s severity proportions: \(0.390(2)+0.610(4)=3.22\) points. This calculation also makes clear why the target population matters when effects differ across covariate groups.

Propensity Weighting and the Balance Check

For an ATE analysis, inverse-probability weights are \(1/\hat e(C)\) for treated patients and \(1/[1-\hat e(C)]\) for controls. These weights give more influence to patients whose observed treatment was less likely given their measured covariates. With the two estimated scores above, the treated high-severity weight is \(1/0.6154=1.625\), and the treated low-severity weight is \(1/0.2623=3.8125\). The control weights are \(1/(1-0.6154)=2.600\) for high severity and \(1/(1-0.2623)=1.356\) for low severity.

The balance mechanism can be seen directly. In a severity category with \(N_c\) patients, let \(n_{1c}\) receive the program and \(n_{0c}\) receive usual care. If the estimated score is \(n_{1c}/N_c\), the weighted treated count is \(n_{1c}/(n_{1c}/N_c)=N_c\). The weighted control count is \(n_{0c}/(n_{0c}/N_c)=N_c\), because \(1-\hat e(c)=n_{0c}/N_c\). Thus the two weighted groups have matching severity totals in this example.

Weighting and standardization target the same ATE here, but they construct the comparison differently. Weighting changes how patients contribute to group means; standardization predicts outcomes under both treatments and averages those predictions over the target covariate distribution. Each approach depends on measured covariates and adequate modeling or estimation. Neither protects against unmeasured confounding merely by producing balanced measured covariates.

Compare Methods and Keep the Target Population Clear

An outcome-regression analysis models the outcome using treatment and baseline covariates. For these data, a saturated model with treatment, severity, and their interaction can represent each of the four observed cell means. Standardization from that model predicts each patient’s mean outcome once setting \(A=1\) and once setting \(A=0\), then averages the predictions over the target cohort. The predictions are the cell means in the table, so the resulting standardized means are 9.44 and 6.22 points, and the ATE estimate is 3.22 points.

The agreement between this outcome-regression estimate and the propensity-weighted estimate is reassuring as a comparison of these two analyses, but it is not a validation of their assumptions. Both use the same hypothetical data and the same measured severity covariate. In a more complex application, different estimates may prompt investigation of model specification, overlap, weight instability, or differences in the target population. The earlier material on comparing outcome regression and propensity weighting gives the methods in broader terms; here the numerical comparison shows what they target in one example.

The ATE is not the only possible estimand. For example, the average treatment effect among the treated (ATT) averages \(Y(1)-Y(0)\) among patients who actually received the program. In this cohort, 60% of program patients have high severity and 40% have low severity. The observed program mean is \(0.600(7)+0.400(11)=8.60\) points. The usual-care outcomes standardized to the treated group’s severity mix are \(0.600(5)+0.400(7)=5.80\) points. Thus the ATT estimate is \(8.60-5.80=2.80\) points. It differs from the ATE of 3.22 points because the target severity distribution differs and the within-severity contrasts are not equal.

This distinction is practical, not merely terminological. A conclusion about the effect for the whole eligible cohort should use the ATE target and corresponding adjustment. A conclusion about patients who received the program should use an ATT target. A numerical estimate cannot be interpreted correctly without saying which population it averages over.

Worked Example: State a Calibrated Conclusion

Suppose the analysis report includes the crude contrast, the ATE estimates, and the checks above. A careful summary would be:

“Among the 200 hypothetical eligible patients, the crude observed difference in mean 12-week symptom improvement was 2.10 points, favoring the new program. After standardizing both treatment groups to the cohort’s baseline-severity distribution, the estimated average difference was 3.22 points. Propensity weighting gave the same estimate and balanced the measured severity indicator in this example. A causal interpretation requires that the treatment strategies are well defined, that measured baseline covariates adequately control confounding, that treatment overlap is adequate for the target cohort, and that the other identification assumptions hold. This analysis cannot rule out bias from unmeasured differences between patients.”

The wording gives the outcome, time, units, direction, population, and adjusted estimate. It distinguishes the observed crude comparison from a causal interpretation conditional on assumptions. It does not claim that the treatment caused an improvement for every patient, that the data prove assumptions, or that matching one measured covariate establishes overall comparability.

Common Pitfalls and a Practical Sequence

A useful analysis sequence is to define the contrast before fitting or weighting, inspect whether the data support it, and then report what the adjustment can establish. The following steps keep those decisions visible:

• Mistake: treating the crude difference as the treatment effect. The reasoning is that a difference in group means directly answers the causal question. The correction is that the crude contrast compares groups with potentially different baseline characteristics; here their high-severity proportions are 0.600 and 0.250.
• Mistake: treating a balanced measured covariate as proof of no confounding. Weighting balances severity here because the score was constructed from severity, but unmeasured confounders may remain and other covariates may still be imbalanced in a fuller analysis.
• Mistake: describing an adjusted association as certain causation. Adjustment does not demonstrate exchangeability or consistency. State the estimate and the assumptions needed for its causal interpretation, and do not claim that observational data establish those assumptions.
• Mistake: reporting an estimate without its target. The ATE of 3.22 points and ATT of 2.80 points answer different questions. Name the target population and outcome units whenever reporting the estimate.