Why a Randomized Trial May Not Answer the Target Question
Randomization can support a causal comparison among people enrolled in a trial, but the treatment effect for those participants need not equal the effect for another eligible population. The populations may differ in characteristics that modify the treatment effect. A useful way to generalize a trial result is therefore to estimate conditional effects in the trial and average them over the covariate distribution of the population of interest.
This approach depends on two distinct sources of information. Randomization supports the causal comparison between treatment arms within the trial. A separate assumption about trial participation connects the trial’s conditional effects to those in the target population. Neither assumption substitutes for the other.
The target population must be specified before interpreting the result: for example, all eligible patients in a defined health system, rather than all patients everywhere. The target effect also refers to a particular treatment contrast and outcome. If the outcome is measured in points, the average treatment effect is in points; for a binary outcome, the difference in mean outcomes is a risk difference.
Assumptions That Connect the Trial to Its Target
Suppose the trial randomly assigns treatment, and let \(A=1\) denote treatment and \(A=0\) denote control. The identification argument uses the following conditions. They concern the target estimand and the relationship between trial and target, not merely the fact that the study is randomized.
Consistency and well-defined treatment: a participant who receives \(A=a\) has observed outcome \(Y=Y^a\), with treatment versions sufficiently well defined for the comparison. Outcomes and covariates must be measured comparably across trial and target settings. Interference between individuals is assumed absent or appropriately handled.
Trial randomization: within the trial, assignment is independent of the potential outcomes conditional on \(X\) and any randomization strata \(R\). This identifies \(E[Y^a\mid X=x,R=r,S=1]\) where treatment positivity holds; to identify \(E[Y^a\mid X=x,S=1]\), average these means over \(P(R=r\mid X=x,S=1)\). If assignment is independent of the potential outcomes given \(X\) alone, the latter mean is identified directly.
Conditional exchangeability across participation: for each treatment \(a\) and relevant \(x\), \(E[Y^a\mid X=x,S=0]=E[Y^a\mid X=x,S=1]\). After conditioning on \(X\), the mean potential outcome in the target population is the same as in trial participants. The covariates must capture the differences needed for this equality, including relevant effect modification.
Positivity: target covariate values must have representation in the trial, and each treatment being compared must be possible at those values. For discrete \(X\), if \(P(X=x\mid S=0)>0\), then \(P(S=1\mid X=x)>0\), and \(P(A=a\mid X=x,S=1)>0\) for each treatment level \(a\) of interest. For continuous or more general \(X\), require the target covariate distribution to be absolutely continuous with respect to the trial covariate distribution, \(\mathcal{L}(X\mid S=0)\ll\mathcal{L}(X\mid S=1)\), and require \(P(A=a\mid X,S=1)>0\) for each treatment level of interest for target-almost every \(X\).
The participation condition is an equality of conditional means, sufficient for identifying the target mean effect. It does not require the trial and target populations to have the same covariate distribution. Indeed, their distributions may differ; the target distribution is explicitly used in the calculation. Randomization alone does not establish conditional exchangeability across participation, since people who enroll may differ from nonparticipants even after treatment is randomized.
Identification by Averaging Conditional Trial Outcomes
For discrete \(X\), the target mean under treatment \(a\) can be written as a weighted average over covariate strata. Each weight is the proportion of the target population in that stratum. The conditional outcome mean comes from the trial.
The first equality is the law of total expectation within the target population. The second uses conditional exchangeability across participation. The third uses trial randomization together with consistency: within the trial and covariate stratum, the mean observed under assignment to \(a\) identifies the corresponding potential-outcome mean. Subtracting the expression for \(a=0\) from that for \(a=1\) gives the target effect:
Thus, the procedure combines a trial-based conditional treatment contrast in each stratum with the target population’s stratum proportions. For continuous covariates, the sum is replaced by an integral over the target covariate distribution. In practice, trial conditional means and target covariate proportions are estimated from data, so the resulting quantity is an estimate of the identified target effect, not a guaranteed exact value.
Worked Example: Generalizing a Binary-Outcome Trial
Consider a hypothetical randomized trial of a treatment intended to prevent a binary adverse outcome. Let \(X\) indicate baseline severity, with \(X=L\) for lower severity and \(X=H\) for higher severity. Assume the trial includes both strata and randomizes treatment within them. The following are hypothetical observed event counts; the within-stratum event proportions are trial estimates of conditional outcome means.
| Severity stratum | Treatment: events / participants | Control: events / participants | Target proportion |
|---|---|---|---|
| Lower, \(L\) | 4 / 40 = 0.10 | 8 / 40 = 0.20 | 0.75 |
| Higher, \(H\) | 20 / 40 = 0.50 | 28 / 40 = 0.70 | 0.25 |
Suppose the target population has 75% lower-severity and 25% higher-severity individuals. The estimated target risk under treatment is
The estimated target risk under control is
Therefore, the estimated target risk difference, treatment minus control, is \(0.200-0.325=-0.125\), or a 12.5 percentage-point lower risk under treatment. This is an estimate for the specified target population, under the stated assumptions. It is not automatically the effect for every population, and the observed event proportions have sampling uncertainty.
The trial itself has equal numbers in each severity stratum. Its crude observed risks are \((0.10+0.50)/2=0.300\) for treatment and \((0.20+0.70)/2=0.450\) for control, a difference of \(-0.150\). That crude estimate averages over the trial’s observed stratum mix, not the target mix. The difference between \(-0.150\) and \(-0.125\) illustrates why the target covariate distribution matters.
Population Composition and Treatment-Effect Modification
When the conditional treatment effect varies with \(X\), different covariate distributions can yield different average effects even if the conditional effects are identical across trial and target populations. The next example is stated in terms of population conditional means, not realized sample averages. That distinction matters: randomization does not force any particular finite sample’s observed mean to equal its expected value.
Worked Example: Trial and Target Effects Can Differ
Suppose a hypothetical trial has two covariate strata, \(X=0\) and \(X=1\). Assume its conditional potential-outcome means are as follows, and that conditional exchangeability across participation holds. The target population has \(P(X=0\mid S=0)=0.80\) and \(P(X=1\mid S=0)=0.20\). The trial population has proportions 0.20 and 0.80, respectively. The values below are stipulated conditional population means for illustration; they are not inferred merely from treatment allocation counts.
| Stratum | \(E[Y^1\mid X,S=1]\) | \(E[Y^0\mid X,S=1]\) | Conditional effect |
|---|---|---|---|
| \(X=0\) | 0.10 | 0.30 | -0.20 |
| \(X=1\) | 0.50 | 0.80 | -0.30 |
The target treatment mean is \((0.10)(0.80)+(0.50)(0.20)=0.08+0.10=0.18\). The target control mean is \((0.30)(0.80)+(0.80)(0.20)=0.24+0.16=0.40\). Thus the target risk difference is \(0.18-0.40=-0.22\).
For comparison, averaging these same conditional means over the trial’s covariate distribution gives a trial-population treatment mean of \((0.10)(0.20)+(0.50)(0.80)=0.42\), and a control mean of \((0.30)(0.20)+(0.80)(0.80)=0.70\). The corresponding trial-population contrast is \(0.42-0.70=-0.28\). These are population averages, or equivalently expected arm means under randomization; a realized trial sample’s arm means need not equal them exactly. The target and trial effects differ because the conditional effect is \(-0.20\) in one stratum and \(-0.30\) in the other, and the populations put different weights on those strata.
Working with a Continuous Covariate
The same logic applies when \(X\) is continuous, although the target distribution is then averaged over rather than summed across a short list of strata. A model can summarize the trial’s conditional means, but its form must be suitable over the target covariate range. The following calculation assumes the displayed conditional mean functions correctly describe the relevant trial means and that participation exchangeability holds.
Worked Example: Averaging Conditional Means Over a Target Distribution
In a hypothetical study, let \(X\) be a baseline score. Suppose trial conditional mean models are \(E[Y^1\mid X=x,S=1]=50+2x\) and \(E[Y^0\mid X=x,S=1]=46+x\), where \(Y\) is measured in points. Assume target participants have \(X=2,6,10\) with proportions 0.20, 0.50, and 0.30. The target distribution is within the trial’s covariate support, and conditional exchangeability is assumed.
The treatment means at those covariate values are \(54\), \(62\), and \(70\) points. Therefore, the target mean under treatment is \((0.20)(54)+(0.50)(62)+(0.30)(70)=10.8+31.0+21.0=62.8\) points. The control means are \(48\), \(52\), and \(56\) points, giving \((0.20)(48)+(0.50)(52)+(0.30)(56)=9.6+26.0+16.8=52.4\) points. The estimated target mean difference is \(62.8-52.4=10.4\) points. The calculation averages the two conditional outcome means over the target composition; it does not treat the trial’s covariate distribution as the target distribution.
What the Result Does—and Does Not—Establish
A generalized trial effect is credible only to the extent that the identification conditions are credible. Randomization protects the treatment comparison within the enrolled trial, but cannot by itself guarantee that trial participants represent nonparticipants in potential outcomes conditional on measured \(X\). Important unmeasured effect modifiers or outcome predictors that differ between populations can undermine the participation condition. Poorly measured covariates, incompatible treatment versions, and differences in outcome definition can also compromise the interpretation.
The target covariate distribution must refer to the population named by the estimand. If the target proportions are wrong, or if the target definition changes, the calculated average can change. Likewise, if a target covariate stratum has no trial participants, the trial cannot supply its conditional outcome mean from the observed data without additional modeling or assumptions. Such a gap is not fixed simply by the fact that treatment was randomized in the trial.
- Randomization answers an internal question: what causal contrast is identified among trial participants?
- Participation exchangeability answers a transport question: can trial conditional potential-outcome means stand in for those of target-population members with the same \(X\)?
- Target weighting answers a population question: how should the conditional means be combined for the specified target population?
The central result is a conditional-to-marginal translation: identify treatment-specific conditional means in the randomized trial, invoke the stated participation assumption, and average those means over the target covariate distribution. The result pertains to that target and those assumptions; it is not a blanket claim that the trial effect applies everywhere.
Check Your Understanding
Use the definitions and assumptions in this tutorial to answer the following.
- In one sentence each, state what trial randomization identifies and what conditional exchangeability across participation contributes.
- A target population is 60% in stratum \(X=0\) and 40% in \(X=1\). Trial conditional treatment risks are 0.15 and 0.45 in those strata. Calculate the standardized target risk under treatment.
- Why can a randomized trial’s marginal effect differ from the target-population effect even when the conditional effects are transportable?
- Explain why conditional mean exchangeability across participation is not an implication of treatment randomization.
- What positivity problem arises if a covariate stratum occurs in the target population but not in the trial?