Treatment Effects That Change Over Calendar Time

Why a Changing Outcome Is Not Necessarily a Changing Treatment Effect

In a response-adaptive trial, allocation probabilities can change as outcomes accumulate. Enrollment time can therefore be associated with which treatment a participant receives. At the same time, prognosis may improve or worsen over calendar time: clinical practice can change, eligibility can shift, or the population reaching the trial can differ. A treatment-arm outcome trend may reflect this background drift, a changing treatment effect, or both.

The key comparison is not whether outcomes among treated participants change from early to late enrollment. It is whether the treatment-versus-control contrast changes over enrollment time. Potential outcomes make that distinction precise: for each enrollment time, compare the expected outcome under treatment with the expected outcome under control for the population defined at that time.

Definition: Let \(T_i\) denote participant \(i\)'s enrollment time, and let \(Y_i(a,t)\) be that participant's potential outcome if assigned treatment \(a\) at time \(t\), where \(a=1\) denotes the experimental treatment and \(a=0\) denotes control. Define the time-specific mean potential outcome as \(\mu_a(t)=E[Y_i(a,t)\mid T_i=t]\). The time-specific average treatment effect is \(\tau(t)=\mu_1(t)-\mu_0(t)\). The control mean \(\mu_0(t)\) describes outcome levels under control at time \(t\); its change over time is baseline outcome drift. A change in \(\tau(t)\) is treatment-effect variation on the specified outcome scale.

The word “baseline” here refers to the control potential outcome, not necessarily a measurement recorded before treatment. If larger outcomes are better, a positive \(\tau(t)\) favors treatment. If smaller outcomes are better, a negative \(\tau(t)\) may favor treatment. The sign must therefore be interpreted using the outcome’s units and direction.

The Decomposition Across Two Enrollment Times

The distinction follows from an identity. Since \(\mu_1(t)=\mu_0(t)+\tau(t)\), the change in the treated potential-outcome mean between two times is the sum of the change in the control mean and the change in the treatment effect.

\begin{aligned} \mu_1(t_2)-\mu_1(t_1) &= \{\mu_0(t_2)+\tau(t_2)\}-\{\mu_0(t_1)+\tau(t_1)\}\\ &= \{\mu_0(t_2)-\mu_0(t_1)\}+\{\tau(t_2)-\tau(t_1)\}. \end{aligned}

Thus, a rise in the treatment-arm mean does not by itself show that treatment has become more effective. The control mean might have risen by the same amount, leaving the treatment contrast unchanged. Conversely, a stable treatment-arm mean could conceal a changing effect if the control mean changes in the opposite direction.

Key result: Compare \(\tau(t)=\mu_1(t)-\mu_0(t)\) at early, middle, and late enrollment times to describe treatment-effect variation. Compare \(\mu_0(t)\) across those times to describe control-outcome drift. Changes in either quantity can occur without changes in the other.

Worked Examples

Worked Example: Improving Outcomes but a Constant Treatment Effect

Suppose a hypothetical trial records a favorable continuous outcome, measured in points. In three enrollment periods, the mean outcomes are:

Enrollment period	Control mean	Treatment mean	Treatment contrast
Early	5	8	8 − 5 = 3 points
Middle	8	11	11 − 8 = 3 points
Late	11	14	14 − 11 = 3 points

The treatment-arm mean rises from 8 to 14 points, an increase of \(14-8=6\) points. But the control mean also rises by \(11-5=6\) points. The time-specific contrasts are \(8-5=3\), \(11-8=3\), and \(14-11=3\) points. The observed pattern is consistent with improving outcomes under both assignments and a constant 3-point treatment effect, not an improving treatment effect.

These hypothetical means illustrate the estimand; in an actual trial they would be estimates subject to sampling variability. Equal contrasts in a small dataset would not prove that the population treatment effect is exactly constant.

Worked Example: Baseline Drift and a Declining Treatment Effect

Now suppose another hypothetical trial has the following mean outcomes, again with higher scores favorable:

Enrollment period	Control mean	Treatment mean	Treatment contrast
Early	10	13	13 − 10 = 3 points
Middle	13	15	15 − 13 = 2 points
Late	16	16	16 − 16 = 0 points

From early to late enrollment, the control mean increases by \(16-10=6\) points. The treatment mean increases by \(16-13=3\) points. The change in the treatment effect is \(0-3=-3\) points. The decomposition checks:

\underbrace{16-13}_{\text{treated-mean change }=3} = \underbrace{16-10}_{\text{control-mean change }=6} + \underbrace{(0-3)}_{\text{effect change }=-3}.

Here, better outcomes later in the trial do not mean that treatment became more beneficial. The control outcomes improved more, while the treatment contrast declined from 3 points to 0 points. Because these are hypothetical period means, the calculations describe the pattern; they do not establish whether the differences exceed random uncertainty.

Worked Example: A Changing Effect Despite Worsening Control Outcomes

Consider a third hypothetical trial measuring an unfavorable outcome, for which lower scores are better. The time-specific means are:

Enrollment period	Control mean	Treatment mean	Treatment contrast
Early	20	19	19 − 20 = −1 point
Middle	18	14	14 − 18 = −4 points
Late	16	9	9 − 16 = −7 points

The control mean falls by \(16-20=-4\) points from early to late, indicating improving outcomes under control. The treatment mean falls by \(9-19=-10\) points. The treatment contrast changes by \((-7)-(-1)=-6\) points, becoming more favorable on this outcome scale. The identity gives \(-10=(-4)+(-6)\): the treatment-arm improvement combines control-outcome improvement with a further 6-point change in the treatment effect.

This example illustrates why the direction of a mean trend alone is not enough. Here, both arms improve, but the treatment-versus-control contrast also changes. Whether the effect variation is statistically distinguishable from random variation requires uncertainty estimates and an analysis appropriate to the trial’s randomization.

Conditions for Interpreting Time-Specific Contrasts

The potential-outcome definitions specify the comparisons of interest. Estimating them from a response-adaptive trial requires attention to how participants are assigned and followed. The following conditions are central to interpreting a contrast as a causal treatment effect within an enrollment period:

Well-defined assignments and outcomes. Treatment and control must have sufficiently consistent meanings, and the outcome must be measured comparably across arms and periods. The observed outcome for a participant assigned \(a\) at time \(t\) must correspond to \(Y_i(a,t)\) (consistency).
Randomized assignment given available information. At each assignment, the randomization must be carried out according to the protocol using the information available at that point. Adaptation can change the probability of assignment, but assignment is not the same as clinician choice based on unrecorded prognosis.
Positive assignment probability. Each treatment being compared must have a nonzero probability of assignment in the relevant enrollment period. If no participants can receive control late in the trial, the late-period treatment-control contrast cannot be estimated from concurrent randomized comparisons without additional assumptions.
No interference between participants. One participant’s assigned treatment should not change another participant’s potential outcome in a way that undermines the individual treatment comparison.
Comparable target populations and follow-up. The meaning of the time-specific population and outcome must be clear. Changes in who enrolls or in follow-up completeness can affect the observed period means, apart from calendar-time changes in prognosis or treatment response.

In practice, time-specific arm means are estimates, not the potential-outcome means themselves. Randomization supports causal comparisons, but finite samples can have chance imbalances, and response-adaptive assignment can leave some period-by-treatment groups small. The allocation procedure and its probabilities must be respected when estimating uncertainty; a raw difference should not be treated as exact evidence of an effect change.

Reading the Pattern and Recognizing Failure Modes

A useful first summary is a table with one row per enrollment period and separate treatment and control means. Calculate the treatment-control contrast in each row, then compare those contrasts. This is a descriptive check on the estimand, not a substitute for inference. A pattern in the point estimates may be too imprecise to support a conclusion about changing effects.

A pooled treatment effect answers a different question: it averages treatment effects over the periods represented in the analysis, with weights determined by the estimand and the analysis. It can be a useful summary when a single average effect is the target, but it does not show whether effects differed over time. A treatment-arm trend alone is even less informative about effect variation because it mixes background outcome change and treatment-effect change.

The time-specific contrast is also scale-dependent. For a continuous outcome, the examples use differences in mean points. For a binary outcome, a difference in event risks and a ratio of risks are distinct effect measures; a pattern that is constant on one scale need not be constant on another. State the scale before describing a treatment effect as changing.

Thin or empty period-by-arm groups: Large uncertainty or no concurrent randomized comparison makes the time-specific contrast unstable or unavailable.
Different follow-up completeness: If later enrollees have less time to experience or record an outcome, apparent temporal differences may reflect measurement or follow-up rather than outcome drift.
Changing enrollment composition: If participant characteristics change across periods, the period-specific contrasts describe the populations enrolled in those periods. A difference between them need not mean that calendar time itself altered treatment response.
Outcome-scale ambiguity: A claim of “increasing benefit” is incomplete unless it specifies the outcome, the favorable direction, and the effect scale.

Common mistake: “The treatment-arm mean is higher late in the trial, so treatment became more effective.” This conclusion does not follow: the control mean may have risen by the same or a greater amount. The correction is to calculate the treatment-control contrast at each enrollment time and compare those contrasts, while accounting for uncertainty and the populations represented.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

In a hypothetical trial, control means are 4, 7, and 10 points early, middle, and late; treatment means are 6, 9, and 12 points. What changes over time?
Control means are 12 points early and 18 points late, while treatment means are 15 and 19 points. Calculate the early and late contrasts and decompose the change in the treatment mean.
Why can a pooled treatment effect fail to reveal treatment-effect variation over enrollment?
In a late period, all participants are assigned treatment and none to control. What prevents direct estimation of the late-period randomized treatment contrast?
Suppose higher values are worse and the treatment-control contrast changes from \(+2\) to \(-1\) point. Which direction did the contrast move, and what additional context is needed to interpret benefit?

Solutions. (1) The contrasts are \(6-4=2\), \(9-7=2\), and \(12-10=2\) points. Both arm means rise by 6 points from early to late, so the pattern shows outcome drift with a constant estimated contrast. (2) The early contrast is \(15-12=3\) points and the late contrast is \(19-18=1\) point. The treatment mean changes by \(19-15=4\) points; this decomposes into a control-mean change of \(18-12=6\) points plus an effect change of \(1-3=-2\) points. (3) A pooled effect averages over periods and can conceal differences among period-specific contrasts; the weighting also determines which average it represents. (4) There is no concurrent randomized control group in that period, so positivity fails for the direct late-period comparison. (5) The contrast decreases by \(-1-(+2)=-3\) points. Because higher values are worse, a negative contrast favors treatment, but interpretation still requires the outcome definition, the effect scale, and uncertainty about both estimates.