Why Compare Two Transport Estimators?
A randomized trial can identify treatment effects for its participants without directly identifying the effect in a different target population. If the trial and target differ in covariates that modify treatment effects, the trial’s marginal effect may not be the target population’s effect. Two common approaches use trial outcomes and target covariate information to address this problem: inverse-odds weighting, which reweights trial participants, and outcome-model standardization, which predicts outcomes under each treatment and averages those predictions over the target population.
The methods can produce similar estimates when they use compatible assumptions and models, but they do not do the same calculation. Comparing their results is useful for understanding which modeling choices matter and whether the trial has adequate representation of target covariate patterns. The comparison does not, by itself, establish that either estimate is correct.
Assumptions and the Two Calculations
The comparison is meaningful only if the trial results can identify the target quantities. Assume consistency, so each participant’s observed outcome under the received treatment equals the corresponding potential outcome. Assume treatment assignment in the trial is exchangeable conditional on \(X\), meaning that within covariate strata treatment assignment is independent of the potential outcomes. In a randomized trial this is supported by its randomization scheme; the assignment probabilities may nevertheless vary across strata. Treatment positivity requires \(e_a(x)>0\) in covariate strata represented in the target.
For transport, assume conditional mean exchangeability across trial participation: for each treatment level \(a\), the mean potential outcome at \(X=x\) is the same in trial participants and in the target population. This condition allows the trial’s conditional outcome means to stand in for the target’s conditional means. Sampling positivity also matters: target strata must be represented in the trial, so \(q(x)>0\) wherever \(p(x)>0\). The covariates must be measured comparably in trial and target data.
Under these assumptions, the target mean outcome under treatment level \(a\) is the target-distribution average of the conditional trial mean \(\mu_a(x)=E(Y\mid A=a,X=x,S=1)\). The target average treatment effect on the risk-difference or mean-difference scale is the difference between those two target means.
Standardization estimates each \(\mu_a(x)\) from trial outcomes, predicts the outcome under \(a\) for target covariate values, and averages those predictions using the target covariate distribution. In a finite target sample, the sum is replaced by the average over its members. With categorical covariates and a separate outcome mean in each stratum, this is direct standardization.
For weighting, a trial participant assigned to level \(a\) in stratum \(x\) receives an arm-specific weight proportional to \(p(x)/[q(x)e_a(x)]\). The factor \(p(x)/q(x)\) corrects the trial-to-target covariate distribution difference; dividing by \(e_a(x)\) accounts for the fact that the covariate distribution in a treatment arm also depends on the trial’s assignment probabilities. A common constant multiplier for all weights in an arm does not change its normalized weighted mean.
Worked Trial-to-Target Comparison
Worked Example: Two Risk Strata and Different Assignment Probabilities
Hypothetical data. A trial has 1,000 participants: 600 in a lower-risk stratum and 400 in a higher-risk stratum. The target population is 30% lower-risk and 70% higher-risk. In the trial, the treatment probabilities are 0.25 in the lower-risk stratum and 0.60 in the higher-risk stratum. The outcome is a binary adverse event. The trial’s observed counts and event rates are:
| Risk stratum | Trial count | Treated count | Treated events | Control count | Control events |
|---|---|---|---|---|---|
| Lower risk | 600 | 150 | 9 | 450 | 45 |
| Higher risk | 400 | 240 | 60 | 160 | 48 |
The counts match the stated assignment probabilities: \(150/600=0.25\) and \(240/400=0.60\). The stratum-specific treated event risks are \(9/150=0.06\) and \(60/240=0.25\); the control risks are \(45/450=0.10\) and \(48/160=0.30\). For this example, take these observed stratum rates as the conditional trial means.
Standardization. The target treated risk is the sum of the stratum-specific treated risks multiplied by target proportions:
The standardized target control risk is:
Thus the standardized risk difference is \(0.193-0.240=-0.047\), or 4.7 percentage points fewer adverse events under treatment in the target population, as estimated from these hypothetical data.
Arm-specific weighting. In the lower-risk stratum, the treated weight is \(0.30/(0.60\times0.25)=2.00\). In the higher-risk stratum it is \(0.70/(0.40\times0.60)=2.917\). Therefore the weighted treated risk is:
For controls, the assignment probabilities are 0.75 in the lower-risk stratum and 0.40 in the higher-risk stratum. The control weights are \(0.30/(0.60\times0.75)=0.6667\) and \(0.70/(0.40\times0.40)=4.375\). The weighted control risk is:
The weighted risk difference is \(0.193-0.240=-0.047\), matching standardization here. The agreement follows because both calculations use the same target proportions and the same stratum-specific trial means; it is not a guarantee that estimates from arbitrary models or finite samples will match.
Why the Assignment Probability Belongs in the Weight
The comparison above illustrates a key point: weighting each arm by \(p(x)/q(x)\) alone is generally insufficient when treatment assignment depends on \(X\). Among trial participants assigned to arm \(a\), the stratum distribution is proportional to \(q(x)e_a(x)\), not simply \(q(x)\). Multiplying that distribution by \(p(x)/q(x)\) leaves stratum mass proportional to \(p(x)e_a(x)\). The arm-specific factor \(1/e_a(x)\) removes this assignment-induced difference.
The cancellation can be seen directly for an arm-specific weighted mean. In the population version, the expected weighted outcome numerator in arm \(a\) is proportional to:
The corresponding expected weighted denominator is proportional to \(\sum_xp(x)=1\), so their ratio is the target mean \(\psi_a\). This argument requires both trial participation and treatment assignment positivity in the target-relevant strata. It also relies on the conditional outcome mean \(\mu_a(x)\) being transportable to the target.
For example, using only \(p(x)/q(x)\) for treated participants in the worked data would leave treated stratum masses proportional to \(0.30(0.25)=0.075\) and \(0.70(0.60)=0.420\). After normalizing, those masses are about 0.1515 and 0.8485, rather than the target proportions 0.30 and 0.70. The resulting treated risk would be approximately \((0.1515)(0.06)+(0.8485)(0.25)=0.2212\), not 0.193. This discrepancy is caused by failing to account for the stratum-specific treatment probability.
Diagnostics: Balance and Weight Concentration
A transported point estimate should be accompanied by diagnostics. For weighting, inspect whether the weighted covariate distribution in each treatment arm reproduces the target distribution. Also examine the range and concentration of the weights: a few very large weights can make the estimate sensitive to a small number of trial outcomes. For standardization, inspect whether the trial contains outcome information across the covariate patterns represented in the target and whether the outcome model’s functional form is plausible. Neither set of checks can prove the identifying assumptions.
Worked Example: Checking Balance and Effective Sample Size
Use the trial counts and arm-specific weights from the previous hypothetical example. The treated lower-risk stratum contributes total weight \(150(2.00)=300\), and the higher-risk stratum contributes \(240(2.917)\approx700\). Thus the treated weighted distribution is 30% lower-risk and 70% higher-risk, matching the target. For controls, the corresponding weighted totals are \(450(0.6667)\approx300\) and \(160(4.375)=700\); their weighted distribution also matches the target.
A common summary of weight concentration is the effective sample size, defined for weights \(w_i\) by \((\sum_iw_i)^2/\sum_iw_i^2\). It is not a replacement for a variance calculation, but a much smaller effective sample size than the actual arm size signals that the weighted estimate relies heavily on a subset of participants.
For treated participants, the sum of weights is 1,000. The sum of squared weights is \(150(2.00^2)+240(2.917^2)\approx2,641.7\), so:
There are 390 treated participants, so the effective sample size is lower than the arm’s actual size. For controls, the squared-weight sum is \(450(0.6667^2)+160(4.375^2)\approx3,262.5\), giving:
The control arm contains 610 participants, but its effective sample size is about 306.5. In particular, the higher-risk control weight of 4.375 is large relative to the lower-risk control weight of 0.6667. This does not invalidate the estimate automatically, but it warns that uncertainty may be substantial and that estimates could be sensitive to a small number of higher-risk control outcomes.
Model Choice and Disagreement
With correctly specified conditional outcome means and the same target covariate distribution, standardization targets the same arm-specific means as weighting under the stated assumptions. In practice, however, standardization requires an outcome model, while weighting requires estimates of the trial and target covariate distributions and the trial assignment probabilities. Model misspecification, sparse covariate patterns, estimated nuisance quantities, and finite-sample variability can all make the estimates differ.
Sensitivity analysis can compare plausible outcome models while keeping the target distribution fixed. A change in the transported estimate is evidence that conclusions depend on modeling choices; it does not identify which model is correct. Predictions in target covariate regions with little trial support deserve particular scrutiny, even if a model returns a numerical value there.
Worked Example: Sensitivity to the Outcome Model
Suppose a hypothetical target population is divided into four covariate groups with proportions 0.20, 0.30, 0.30, and 0.20. Two plausible fitted outcome models produce the following predicted event risks. These values are illustrative predictions, not results from an actual study.
| Target group | Target proportion | Model A treated risk | Model A control risk | Model B treated risk | Model B control risk |
|---|---|---|---|---|---|
| 1 | 0.20 | 0.05 | 0.08 | 0.08 | 0.10 |
| 2 | 0.30 | 0.10 | 0.15 | 0.13 | 0.17 |
| 3 | 0.30 | 0.20 | 0.28 | 0.25 | 0.29 |
| 4 | 0.20 | 0.30 | 0.40 | 0.40 | 0.42 |
Under Model A, the standardized treated risk is \(0.20(0.05)+0.30(0.10)+0.30(0.20)+0.20(0.30)=0.160\). The standardized control risk is \(0.20(0.08)+0.30(0.15)+0.30(0.28)+0.20(0.40)=0.225\). The resulting risk difference is \(0.160-0.225=-0.065\), or 6.5 percentage points fewer events under treatment.
Under Model B, the treated risk is \(0.20(0.08)+0.30(0.13)+0.30(0.25)+0.20(0.40)=0.210\). The control risk is \(0.20(0.10)+0.30(0.17)+0.30(0.29)+0.20(0.42)=0.242\). The risk difference is \(0.210-0.242=-0.032\), or 3.2 percentage points fewer events under treatment.
The two models give different transported risks and different effect estimates, despite using the same target proportions. A useful response is to examine how each model represents covariates and treatment interactions, compare its predictions with observed trial outcomes where support is adequate, and report the sensitivity. Choosing the more favorable result would not resolve the underlying modeling uncertainty.
Using the Comparison Responsibly
Weighting and standardization offer complementary ways to express the same target-population estimand. When their estimates are close, that agreement can be reassuring about robustness to some analytic choices, but shared assumptions can still be wrong. When they disagree, investigate whether they use the same target distribution, treatment-assignment probabilities, covariates, and outcome scale; then inspect weight concentration and outcome-model predictions. Do not treat agreement as proof of transportability or disagreement as proof that one method has failed.
Both estimates are causal only under the identification assumptions, including conditional transportability and positivity. If target covariate patterns are absent from the trial, changing between weighting and standardization does not create the missing outcome information. The interpretation should remain tied to the specified target population and effect scale: here, a difference in target adverse-event risks, measured in percentage points.
- Common mistake: omit treatment assignment from arm-specific weights. Using \(p(x)/q(x)\) alone assumes the covariate mix within each treatment arm already matches the trial’s overall covariate mix. With assignment probabilities that vary by \(X\), it does not. Use weights proportional to \(p(x)/[q(x)e_a(x)]\) for arm \(a\).
- Common mistake: call agreement proof. Weighting and standardization can share the same incorrect transport assumptions or fail in the same unsupported target strata. Agreement is a diagnostic, not a validation of those assumptions.
- Common mistake: interpret effective sample size as the actual number of observations. It is a summary of weight concentration, not a literal count of independent participants and not a substitute for uncertainty estimation.
Check Your Understanding
Use the assumptions and calculations in this tutorial to answer the following.
- Why is \(p(x)/q(x)\) alone generally insufficient as the weight for an arm-specific mean when \(e_a(x)\) varies across strata?
- In the two-stratum example, what target treated and control risks result from standardization, and what is their risk difference?
- What does a lower effective sample size than the actual arm size suggest about a weighted estimate?
- If weighting and standardization disagree, name two modeling or data features you would check before interpreting the difference.
- Why does close agreement between the two estimates not establish that the transported effect is identified?