Treatment-Effect Heterogeneity in Intermediate Outcomes

Why heterogeneity matters at an interim selection look

An overall MAMS selection rule typically retains or drops an experimental arm using information summarized across the trial population. That summary can conceal meaningful differences: an arm may have a large intermediate-outcome effect in one baseline subgroup and little effect in another. The overall rule still makes one decision for the arm, but the effect being summarized depends on the subgroup mix and the outcome scale.

The central distinction is between a subgroup-specific treatment effect and a difference in subgroup outcomes. A subgroup may have a better prognosis under both treatment and control, yet have the same treatment effect as another subgroup. Heterogeneity concerns variation in the treatment contrast, not variation in baseline outcome levels alone.

For a time-to-event final endpoint, \(Y\) can be restricted mean survival time (RMST) through a fixed horizon \(\tau\): \(Y=\min(T,\tau)\). Then \(\theta_{Y,g}\) is a difference in expected survival time through \(\tau\), measured in time units. Using an RMST contrast avoids interpreting a hazard ratio as if it were a difference in average survival time. The intermediate outcome may instead be binary, in which case a risk difference is measured in probability units.

Assumptions and the overall effect

A causal interpretation of the subgroup contrasts requires treatment assignment to be randomized within the relevant trial population, consistency (the observed outcome under the assigned treatment equals the corresponding potential outcome), and a well-defined treatment and outcome. Each subgroup used for inference must have participants assigned to both treatment and control; this is the positivity requirement. The subgroup should be defined using baseline information, not a variable affected by treatment.

For an RMST contrast, follow-up and censoring must also permit estimation through \(\tau\). In particular, censoring must be independent of the event time conditional on the variables used in the analysis, or a suitable adjustment must be made; there must be adequate follow-up support through the chosen horizon. For a simple comparison of intermediate-outcome means or risks, the outcome must be observed or missingness addressed under stated assumptions.

Let \(\pi_g=P(G=g)\) be the proportion of the target population in subgroup \(g\). When subgroup effects are expressed on the same additive scale, the population-average treatment effect is the prevalence-weighted average of the subgroup effects. The weights must refer to the population the selection rule is intended to serve.

This identity follows by conditioning the expected potential outcome under each treatment on \(G\): \(E[I(a)]=\sum_g\pi_gE[I(a)\mid G=g]\), and likewise for control. Subtracting the two expectations gives the weighted sum of the subgroup-specific differences. It is an identity for a specified target population, not a claim that every subgroup has the same effect. If trial subgroup proportions differ from the target population, the target weights should be applied explicitly rather than allowing the trial’s accidental mix to define the estimand.

How heterogeneity affects an arm-level rule

Suppose a prespecified interim rule retains an experimental arm when its estimated overall intermediate-outcome effect exceeds a threshold \(c\). The threshold and effect must use the same scale. If \(c=0.10\) for a binary-outcome risk difference, the threshold is 10 percentage points. A pooled rule applies to the arm as a whole: it does not automatically create different treatment assignments or decisions for different subgroups.

An overall decision can be appropriate when the target estimand is the population-average effect. It may nevertheless conceal which subgroups contribute to that average. Conversely, making subgroup-specific retain/drop decisions changes the decision structure and can lead to different populations receiving further study. Such decisions need to be specified and justified in the design; they cannot be inferred just by inspecting subgroup estimates after the fact.

The example also shows that the target-population mix matters. If the prevalence of subgroup 1 changes, both weighted averages can change, potentially changing whether an arm crosses the threshold. A threshold calibrated for one population should not be transferred to another without checking how its estimand and operating characteristics change.

Estimating heterogeneity and judging uncertainty

A simple way to describe heterogeneity for a continuous intermediate outcome is a treatment-by-subgroup interaction model. With two subgroups, let \(A=1\) denote the experimental arm and \(A=0\) control, and let \(G=1\) identify subgroup 1. A linear model with an additive treatment effect has the form

Under this model, the treatment contrast in subgroup 0 is \(\beta_A\), and in subgroup 1 it is \(\beta_A+\beta_{AG}\). The interaction coefficient \(\beta_{AG}\) is the difference between those contrasts. For binary outcomes, the same idea can be expressed on a risk-difference scale by modeling subgroup-specific risks and subtracting them; a logistic interaction instead describes variation on the log-odds scale, which need not correspond to the same heterogeneity conclusion on the risk-difference scale.

An estimated interaction is uncertain, especially when subgroups are small. A noisy subgroup estimate can lie above a retention threshold by chance, or below it despite a favorable underlying effect. Therefore, display subgroup estimates with uncertainty intervals and report the overall estimate separately. A nonsignificant interaction test does not prove that effects are equal; it may simply have little power to detect a difference.

Diagnostics, alternatives, and limitations

At an interim look, useful diagnostics are direct and tied to the selection estimand:

• Report the number assigned to each treatment within every prespecified subgroup, along with missing outcomes and event counts. Sparse cells produce imprecise contrasts.
• Show subgroup-specific treatment contrasts and uncertainty intervals beside the target-population weighted estimate. Check that the weights sum to one and match the intended target population.
• State the scale for heterogeneity. A difference in risk differences, mean differences, or RMST differences answers a different question from an interaction on an odds or hazard scale.
• Check whether subgroup definitions were fixed before examining interim results. Data-driven cut points and multiple exploratory comparisons can make apparent variation unstable.
• For survival outcomes, inspect whether follow-up and censoring support the chosen horizon in each subgroup. A pooled estimate can be well supported while a subgroup RMST estimate is not.

Three summaries are often useful, but they answer different questions. A pooled estimate describes the average effect for a specified population and is aligned with an overall arm-level threshold. Subgroup-specific estimates describe variation and may be imprecise. An interaction estimate summarizes the difference between subgroup effects on a chosen scale; it does not itself decide which arm to retain. A table can show all three without substituting one for another.

A common mistake is to see a favorable overall estimate and conclude that the treatment benefits every subgroup. The correction is to interpret the pooled contrast as a weighted average and inspect subgroup contrasts before making any claim about uniformity. The opposite mistake is to declare that effects differ because one subgroup estimate crosses a threshold and another does not. Different threshold outcomes do not, by themselves, demonstrate a statistically or clinically meaningful difference; the interaction and its uncertainty address that comparison more directly.

Subgroup analysis has limits. Prespecification does not guarantee adequate sample size, and a small number of observed subgroup estimates cannot establish a general pattern of effect variation. The average effect depends on both the chosen scale and target-population weights. Finally, variation in intermediate effects alone does not determine variation in final effects: the two endpoints must be assessed on their own scales and with the appropriate evidence. An intermediate-based arm-selection decision should be described as the result of its stated rule, not as proof of benefit for every subgroup.