Why a Workflow Matters
A propensity-score analysis is not a single calculation. It is a chain of choices: define the target effect, select pretreatment covariates, estimate treatment probabilities, construct weights or strata, calculate an effect, and assess whether the analysis behaved as intended. If any link is ambiguous, another analyst may be unable to reproduce the result—or may calculate a different result while using the same data.
A reproducible workflow makes those choices explicit and records them before interpreting the estimate. It does not make an inadequate design valid by itself. In particular, it cannot establish that all confounders were measured or that the causal assumptions hold. Its purpose is to make the design and calculations inspectable, consistent with the target estimand, and repeatable.
For causal interpretation, the analysis typically requires consistency (the observed outcome equals the potential outcome under the treatment received), conditional exchangeability \(Y(1),Y(0)\perp A\mid X\), and conditional positivity \(0<P(A=1\mid X)<1\) in the target population. The score model must also estimate treatment probabilities well enough for the chosen design. These assumptions are substantive and statistical requirements, not boxes that a software output can certify. Conditional positivity and its diagnostics require particular attention, but are treated in more detail in the later material on positivity.
A Prespecified Analysis Sequence
The sequence below is useful whether the final design uses inverse probability weights, propensity-score strata, or both as prespecified alternatives. Decisions made after viewing outcome differences can change the analysis and should be reported as such, rather than presented as if they were planned in advance.
Specify the eligible population, treatment contrast, outcome, follow-up time, and whether the target is the average treatment effect (ATE), average treatment effect among the treated (ATT), or another explicitly defined effect. State units, such as a mean difference in days.
Record eligibility rules, exclusions, missing-data handling, the treatment indicator, outcome, and pretreatment covariates. Explain the substantive basis for covariate selection. Do not include variables measured after treatment as if they were baseline confounders.
Specify the model form, covariate coding, transformations, interactions, fitting method, and software details. Save each unit's estimated score and treatment status, along with the model specification and analysis data version.
For weighting, use weights that correspond to the stated target. For subclassification, define the score cut points and identify the target-population weights used to aggregate stratum effects. Record any exclusions or modifications and their rationale.
Inspect score distributions and covariate balance under the proposed design; review extreme weights, effective sample size, and empty or sparse treatment groups in strata. If the design is not credible, document the problem and any revised analysis rather than silently changing the procedure.
State the exact estimator and standard-error or interval procedure, including any clustering or score-estimation procedure relevant to the analysis. Report the estimate in context with uncertainty, diagnostics, limitations, and enough computational detail for reproduction.
The order matters. For example, choosing ATE weights and then describing the result as an effect among treated units is not a harmless relabeling: the weights define a target population. Likewise, model fitting alone is not a complete analysis. The workflow should preserve a record of both the score model and the effect calculation.
Weighting: A Calculation That Can Be Reproduced
For binary treatment, common unstabilized inverse-probability weights are \(1/e(X_i)\) for treated units and \(1/\{1-e(X_i)\}\) for control units when targeting the ATE. A normalized, or Hájek, estimate compares the weighted outcome means:
The first ratio is the weighted mean outcome among treated units and the second is the weighted mean among controls. Under the causal assumptions and adequate estimation, the contrast targets the ATE. Normalizing each group by its own sum of weights makes this a ratio estimator; it is not the same finite-sample calculation as an unnormalized Horvitz–Thompson estimator.
Worked Example: ATE Weighting With Saved Scores
Suppose a hypothetical six-person data set has the following observed outcomes and estimated propensity scores. The outcome is a score in points. Assume for this arithmetic example that the causal assumptions are plausible for the specified covariates and the displayed scores are the analysis scores.
| Unit | Treatment \(A_i\) | \(\hat e(X_i)\) | Outcome \(Y_i\) (points) |
|---|---|---|---|
| 1 | 1 | 0.80 | 10 |
| 2 | 1 | 0.50 | 8 |
| 3 | 1 | 0.20 | 9 |
| 4 | 0 | 0.80 | 4 |
| 5 | 0 | 0.50 | 5 |
| 6 | 0 | 0.20 | 6 |
For the ATE, treated weights are \(1/\hat e(X_i)\), giving \(1.25, 2,\) and \(5\). Control weights are \(1/\{1-\hat e(X_i)\}\), giving \(5, 2,\) and \(1.25\). Each treatment group has total weight \(8.25\). The treated weighted mean is
The control weighted mean is
Thus the normalized ATE estimate is \(8.909-4.545=4.364\) points. This is an adjusted estimate for the specified target under the stated assumptions; these six hypothetical observations alone do not establish those assumptions or provide a reliable uncertainty estimate. A reproducible report would also state how the scores were estimated and how uncertainty was calculated.
Stratification: Preserve the Target in Aggregation
In subclassification, units are grouped into strata based on estimated propensity scores. Within each stratum, calculate a treatment-control contrast, then aggregate those contrasts using weights that represent the chosen target population. Equal weighting of strata is justified only when equal stratum weights are appropriate for that target. Record the cut points, group counts, stratum-specific estimates, and aggregation weights so the final result can be rebuilt.
Worked Example: Aggregating Stratum Effects for a Target Population
A hypothetical analysis defines two score strata. The target population contains 80 units in the first stratum and 20 in the second. The estimated treatment effects are 2 points and 6 points, respectively. Assume each stratum has both treatment groups and the stratum-specific contrasts are considered appropriate for aggregation.
The target-population shares are \(80/100=0.80\) and \(20/100=0.20\). Therefore, the aggregated estimate is
Giving the strata equal weight would instead yield \(\{2+6\}/2=4\) points. That calculation targets an equally weighted average of the two stratum effects, not the stated population, in which the first stratum is four times as large. The workflow should therefore retain both stratum-specific results and the target weights, rather than reporting only the aggregated number.
Make the Estimand and the Weights Agree
The same estimated scores can produce different estimators because different estimands call for different weights. For the ATT, treated units represent the target group and are usually left with weight 1, while control units receive odds weights \(\hat e(X_i)/\{1-\hat e(X_i)\}\). The contrast is then the treated mean outcome minus the weighted control mean. This does not make ATT weights a variation of ATE weights with no interpretive consequence: it changes the target population.
Worked Example: Changing From the ATE to the ATT
Use the hypothetical scores and outcomes in the first example, but now target the ATT. The treated mean is the ordinary treated-group mean, \((10+8+9)/3=9\) points. The three controls receive odds weights \(0.80/0.20=4\), \(0.50/0.50=1\), and \(0.20/0.80=0.25\). Their weighted mean is
The ATT estimate is \(9-4.286=4.714\) points. It differs from the ATE estimate of 4.364 points because the target and weights differ. The arithmetic illustrates the distinction; it does not show that either estimate is more credible without examining assumptions and diagnostics for the corresponding target.
Diagnostics, Documentation, and Failure Modes
Diagnostics belong in the workflow before the effect is interpreted. Examine covariate balance after weighting or within strata, using meaningful summaries rather than relying only on whether a significance test is nonsignificant. Review the score distributions by treatment group, the largest weights, and the effective sample size. For weights \(w_i\), a common effective sample size summary is \((\sum_i w_i)^2/\sum_i w_i^2\); a much smaller value than the raw sample size indicates that a small number of observations dominate the weighted calculation. These checks can flag instability, but passing them does not prove no unmeasured confounding or validate the causal assumptions.
A reproducible record should include the analysis population and exclusions; definitions and timing of treatment, outcome, and covariates; missing-data procedures; score model formula and coding; software and version or saved code; score and weight summaries; stratum definitions; the exact estimator; uncertainty method; balance and stability diagnostics; and any deviations from the prespecified plan. When a bootstrap or other procedure is used, document which stages are repeated, including whether the propensity model is re-estimated in each replicate.
Common failures include fitting the score model but not saving its specification, using one weight formula for every estimand, aggregating strata equally without justification, reporting only the final contrast, and changing the model or exclusions until the outcome estimate is attractive. A revised design may be warranted, but report what prompted it, what changed, and whether it was exploratory. Do not imply that balance on measured covariates confirms exchangeability.
Weighting and subclassification are alternative design strategies, not interchangeable labels for the same computation. Weighting uses individual-level weights and can represent a target directly, but extreme weights can make results unstable. Subclassification summarizes within-score-group comparisons and makes aggregation visible, but coarsening scores can leave residual imbalance within strata. In either approach, the target estimand, diagnostics, and uncertainty procedure must be stated. A regression adjustment or matching analysis may also be considered, but is not a substitute for documenting the design actually used.
Check Your Understanding
Answer each question, then compare with the solutions below.
- What must be specified before choosing ATE or ATT weights, and why?
- In a two-stratum target population, the stratum shares are 0.70 and 0.30 and the effects are 1 and 5 days. Calculate the aggregated effect.
- In a hypothetical ATT analysis, the treated mean outcome is 12 units. Two controls have outcomes 8 and 10 and propensity scores 0.60 and 0.20. Using ATT odds weights, calculate the weighted control mean and ATT estimate.
- Name two diagnostics that can reveal instability or poor design, and state one limitation of diagnostics.
Solutions. (1) Specify the target population and estimand, along with the treatment contrast and outcome. ATE and ATT weights represent different target populations, so choosing weights first risks interpreting the result as the wrong effect. (2) The target-weighted contrast is \(0.70(1)+0.30(5)=0.70+1.50=2.20\) days. (3) The control odds weights are \(0.60/0.40=1.5\) and \(0.20/0.80=0.25\). The weighted control mean is \(\{1.5(8)+0.25(10)\}/(1.5+0.25)=14.5/1.75=8.286\) units. The ATT estimate is \(12-8.286=3.714\) units. (4) Examples include post-weighting covariate balance, extreme weights, effective sample size, and score distributions by treatment. Diagnostics can reveal observable problems, but satisfactory results do not establish exchangeability or rule out unmeasured confounding.