Tutorials › Causal Inference › Worked Example of Trial-to-Population Generalization

Transportability and Generalizability · Tutorial 945 of 972

Worked Example of Trial-to-Population Generalization

Learn how to combine treatment effects estimated within trial subgroups with target-population covariate data to calculate and interpret a target average treatment effect.

Advanced 9 min read

What You'll Learn

  • Define a target-population average treatment effect using potential outcomes.
  • Identify the assumptions needed to transport conditional effects from a randomized trial.
  • Estimate subgroup treatment effects from trial outcome data.
  • Standardize subgroup effects to the target population’s covariate distribution.
  • Interpret how a different target mix can change the overall effect.
  • Recognize when the trial data do not support the required calculation.

From a Trial Effect to a Target-Population Effect

A randomized trial can estimate treatment effects well for its participants, yet those participants may differ from the population to which a decision-maker wants to apply the results. If treatment effects vary across covariate groups, the trial’s overall average treatment effect may not equal the average effect in that target population. A direct way to address this is to estimate effects within relevant covariate groups and average them using the target population’s group frequencies.

This tutorial works through that calculation. The central task is not to reweight trial participants; it is to combine estimated conditional effects with the target distribution of covariates. The resulting estimate answers a specific question: what would the average treatment effect be for the target population, if conditional treatment effects were transportable from the trial?

Definition: Let \(Y(1)\) and \(Y(0)\) denote the outcomes an individual would have under treatment and control, respectively. Let \(S=1\) indicate membership in the randomized trial and \(S=0\) membership in the target population outside the trial. For covariates \(X\), the target-population average treatment effect is $$ \text{ATE}_{T}=E[Y(1)-Y(0)\mid S=0]. $$ The conditional treatment effect at covariate value \(x\) is \(E[Y(1)-Y(0)\mid X=x,S=s]\). Generalization uses trial estimates of these conditional effects and averages them over the target distribution of \(X\).

For discrete covariates with possible values \(x\), the standardization formula is

$$ \text{ATE}_{T} = \sum_x E[Y(1)-Y(0)\mid X=x,S=1]P(X=x\mid S=0). $$

Each subgroup effect is measured in the outcome’s units. Its target-population frequency is a proportion, so the sum is also in the outcome’s units. For continuous covariates, the corresponding operation averages the conditional effect function over the target covariate distribution rather than summing over a finite set of categories.

What Must Hold for the Calculation to Identify the Target Effect?

Randomization supports treatment-effect estimation within the trial, but it does not by itself guarantee that trial effects apply outside the trial. The standardization formula requires assumptions about treatment assignment, participation, measurement, and available covariate support.

  • Consistency and a well-defined intervention. A participant’s observed outcome under the treatment actually received equals the corresponding potential outcome. The treatment and control conditions must be sufficiently well-defined for those potential outcomes to be meaningful.
  • No interference. One individual’s outcome is not affected by other individuals’ treatment assignments, or any relevant interference is handled by the design and estimand.
  • Randomization within the trial. Treatment assignment is independent of the potential outcomes within the trial, with a positive probability of receiving each trial arm in each covariate group used for estimation.
  • Conditional effect transportability. Within levels of the measured covariates \(X\), the mean treatment effect is the same in the trial and target population: $$ E[Y(1)-Y(0)\mid X=x,S=1] = E[Y(1)-Y(0)\mid X=x,S=0]. $$
  • Sampling positivity for the target groups. Every covariate group that occurs in the target population must also occur in the trial, so its conditional effect can be estimated there.
  • Target covariate distribution is known or estimated appropriately. The target group proportions used in the calculation must describe the intended target population.

Conditional effect transportability is a substantive assumption, not a consequence of randomization. The covariates must capture the differences relevant to how treatment effects vary between trial and target populations. Importantly, the formula does not require equality of the trial and target outcome means within each covariate group. It requires equality of the conditional treatment effects. Outcome levels can differ while treatment contrasts remain transportable.

Worked Calculation from Trial Subgroups

Worked Example: Standardizing Effects on a Continuous Outcome

Hypothetical data. A randomized trial evaluates a program intended to increase symptom-free days over a fixed follow-up period. The effect modifier \(X\) is age group: younger than 65 years or 65 years and older. The table gives the numbers of participants and the total symptom-free days in each trial cell.

Age groupArmParticipantsTotal symptom-free daysMean days
Younger than 65Control1006006.0
Younger than 65Treatment1009009.0
65 years and olderControl803204.0
65 years and olderTreatment805126.4

Step 1: Estimate trial conditional effects. In the younger group, the treatment mean is \(900/100=9.0\) days and the control mean is \(600/100=6.0\) days. The estimated effect is \(9.0-6.0=3.0\) additional symptom-free days. In the older group, the treatment mean is \(512/80=6.4\) days and the control mean is \(320/80=4.0\) days, giving an estimated effect of \(6.4-4.0=2.4\) additional days.

The trial is stated to be randomized, with participants in both arms of each age group, so the trial cell contrasts estimate the age-specific treatment effects under the randomization and consistency assumptions. Suppose, additionally, that the age-specific effects are transportable and that the target covariate data are representative.

Step 2: Use the target age distribution. Suppose the target population is 70% younger than 65 and 30% aged 65 years or older. Standardize the subgroup effects:

$$ \widehat{\text{ATE}}_{T} = (0.70)(3.0\text{ days})+(0.30)(2.4\text{ days}) = 2.10\text{ days}+0.72\text{ days} = 2.82\text{ days}. $$

Under the stated assumptions, the estimated target-population average effect is 2.82 additional symptom-free days over the follow-up period. This is an average contrast for the target population, not a prediction that every individual gains 2.82 days.

Why Standardization Uses the Target Mix

The calculation follows from the law of total expectation. Because \(X\) partitions the target population into covariate groups, the target average effect is the weighted average of the conditional target effects, with weights equal to the target group probabilities:

$$ E[Y(1)-Y(0)\mid S=0] = \sum_x E[Y(1)-Y(0)\mid X=x,S=0]P(X=x\mid S=0). $$

Under conditional effect transportability, replace each conditional target effect with its trial counterpart. This gives the standardization formula:

$$ E[Y(1)-Y(0)\mid S=0] = \sum_x E[Y(1)-Y(0)\mid X=x,S=1]P(X=x\mid S=0). $$

Thus, the formula combines two distinct sources of information: the trial supplies conditional treatment effects, and target covariate data supply the weights. If the target distribution differs from the trial distribution and effects differ across covariate groups, the trial’s unstandardized average effect generally answers a different question.

Worked Example: Standardizing a Risk Difference

Hypothetical data. A randomized trial studies whether a preventive intervention reduces the risk of an adverse event. In a lower-risk covariate group, 12 of 120 control participants and 6 of 120 treated participants experience the event. In a higher-risk group, the corresponding counts are 36 of 120 under control and 24 of 120 under treatment. Assume randomization within the trial, and suppose the conditional risk differences are transportable to the target population.

In the lower-risk group, the estimated control risk is \(12/120=0.10\) and the treatment risk is \(6/120=0.05\). The risk difference, treatment minus control, is \(0.05-0.10=-0.05\), or 5 percentage points lower risk under treatment.

In the higher-risk group, the control risk is \(36/120=0.30\), and the treatment risk is \(24/120=0.20\). The risk difference is \(0.20-0.30=-0.10\), or 10 percentage points lower risk under treatment.

Suppose target data indicate that 25% of the target population is in the lower-risk group and 75% in the higher-risk group. Then:

$$ \widehat{\text{ATE}}_{T} = (0.25)(-0.05)+(0.75)(-0.10) = -0.0125-0.075 = -0.0875. $$

The estimated target-population risk difference is \(-0.0875\), or 8.75 percentage points lower risk under treatment. The negative sign follows the defined contrast, treatment minus control. This is an average absolute risk difference for the target population; it is not a relative reduction of 8.75%.

Target Composition Can Change the Overall Effect

Standardization is not just a technical adjustment. When effects differ across groups, changing the group proportions changes the population average. The next calculation makes that dependence explicit and shows why a trial-wide average should not automatically be reported as the target effect.

Worked Example: A Different Target Mix Changes the Average

Hypothetical data. In a randomized trial, the estimated effect on a favorable outcome is \(+4\) units in the younger group and \(-2\) units in the older group. Suppose each group makes up half the trial. The trial’s overall effect is therefore:

$$ (0.50)(4)+(0.50)(-2)=2-1=1\text{ unit}. $$

Now suppose a target population contains 50 younger and 70 older individuals, for a total of 120. The younger-group proportion is \(50/120=5/12\), and the older-group proportion is \(70/120=7/12\). Using the trial’s conditional effects:

$$ \widehat{\text{ATE}}_{T} = \left(\frac{5}{12}\right)(4) + \left(\frac{7}{12}\right)(-2) = \frac{20}{12}-\frac{14}{12} = 0.50\text{ unit}. $$

The estimated target average is \(0.50\) outcome unit, rather than the trial average of \(1\) unit. The difference comes from the target population’s larger share of the group with a negative estimated effect. This calculation assumes the two conditional effects transport to the target and that both groups are represented in the trial; the target counts provide the target mix, not new treatment-effect evidence.

Practical Checks Before Reporting the Estimate

A numerical result is only as credible as the inputs and assumptions behind it. Before interpreting a standardized estimate, verify that the trial and target covariates use compatible definitions, that all target groups have trial representation, and that the target proportions sum to one. Also check that the outcome scale and contrast direction are consistent across groups.

The conditional effects need not all be estimated precisely for the arithmetic to be valid, but uncertainty in subgroup estimates and target proportions affects uncertainty in the final estimate. A point estimate such as 2.82 days does not by itself establish that the effect differs from zero. Statistical uncertainty requires an appropriate interval or test that accounts for how the conditional effects and target distribution were estimated.

If an important effect modifier is unmeasured, or if the conditional effects differ between trial and target populations even after conditioning on the available \(X\), the standardized estimate may not identify the target effect. More exact arithmetic cannot repair a failure of transportability. Likewise, if a target group has no trial counterparts, its conditional effect cannot be recovered by this formula from the trial alone.

Common Mistake

Common mistake: “The trial was randomized, so its overall average treatment effect must apply to the target population.” Randomization supports comparisons of treatment arms among trial participants; it does not make the trial’s covariate distribution match the target’s, nor does it guarantee equal effects across those distributions. The correction is to estimate effects within relevant covariate groups, justify conditional effect transportability and overlap, and average those effects using the target population’s covariate frequencies.
Key takeaway: A trial-to-population standardized effect is a target-weighted average of trial-estimated conditional effects. Its interpretation depends on the target covariate distribution and on the assumption that conditional effects transport across trial and target populations.

Check Your Understanding

Use the definitions and calculations above to answer these questions.

  1. A trial estimates effects of 2 units and 5 units in two covariate groups. The target proportions are 0.60 and 0.40. What is the standardized target average effect?
  2. Why does randomization within a trial not, by itself, establish conditional effect transportability?
  3. For a binary outcome, a standardized treatment-minus-control risk difference is \(-0.06\). State its interpretation in percentage points.
  4. What positivity condition is needed when standardizing trial effects to target covariate groups?
  5. Can the standardization formula require trial and target outcome means to be equal within every covariate group? Explain what equality it does require.
Need help applying this method to your own data?

Our team supports biostatistics, statistical programming and pharmacometric analyses. Send us a request and we will reply by email.

Request an analysis
Clinical Biostats