Why choosing a trend after seeing outcomes is risky
In an adaptive trial, treatment allocation can change during enrollment, while prognosis may also change with calendar time. An analysis therefore needs a defensible way to represent time. A linear trend, a late-enrollment indicator, and a spline are not interchangeable descriptions: each imposes a different structure on how outcomes vary over time. The difficulty addressed here is not that one particular form is always wrong. It is that choosing among forms, cutoffs, or spline knots because one gives the most favorable treatment result makes the reported result depend on the observed outcomes.
A model can be chosen after inspecting information that does not reveal treatment-specific outcomes—for example, enrollment dates or an external clinical rationale. That is different from trying several models and retaining the one with the largest estimated benefit, smallest p-value, or most favorable interval. The latter is outcome-informed selection. Conventional standard errors and confidence intervals calculated as though the selected model had been fixed in advance do not account for that search.
What the model choice changes
Let \(Y_i\) be the outcome for participant \(i\), \(A_i=1\) denote assignment to the active treatment, \(A_i=0\) denote control, and \(t_i\) denote enrollment time scaled from 0 to 1. For illustration, suppose higher outcome values are better and the target is a treatment contrast in outcome points. A simple adjusted model is
Here, \(\beta_A\) is a common treatment contrast under the model: it assumes the treatment effect does not change with enrollment time, while the mean outcome can have a linear time trend. A late-period indicator instead represents a step in the mean outcome at a specified cutoff. A spline permits a more flexible time pattern, but its shape depends on the chosen knot or knots. If treatment effects themselves are allowed to vary over time, the model also changes which treatment contrast is being estimated; a contrast averaged over the whole trial need not equal the contrast in a selected late window.
These distinctions matter particularly when allocation probabilities vary over time. A time adjustment cannot create information where there is little or no treatment overlap. For the examples below, assume randomization gives both treatments positive assignment probability at the relevant enrollment times, participants’ outcomes are independent conditional on the model variables, and the stated standard errors are valid for the fitted models. Also assume the common-effect models used in the first and third examples are appropriate working descriptions of the target contrast. If these conditions fail, the numerical procedures may not have their stated interpretation even apart from model selection.
Worked Example: Choosing a trend form after comparing treatment estimates
Worked Example: Choosing a Trend Form After Comparing Treatment Estimates
Hypothetical data and assumptions. Consider a trial with 240 participants and a continuous 30-day outcome measured in points, with higher values better. Both treatment groups are represented throughout enrollment, and each participant is randomized according to the trial’s adaptive procedure. Investigators fit two prespecified-in-advance-for-this-calculation model forms to the same data: a common treatment effect with a linear enrollment-time adjustment, and a common treatment effect with a step adjustment for late enrollment, defined as \(t_i\geq 0.70\). Assume the linear form is the intended primary model and that the model-based Wald standard errors are valid under each fitted model.
The linear-trend fit estimates an adjusted treatment contrast of 2.8 points with \(SE=1.2\) points. The late-step fit estimates 4.8 points with \(SE=1.4\) points. To illustrate the selection problem, suppose an analyst looked at both results and chose the form with the larger estimated benefit.
For the linear-trend model, the nominal 95% Wald interval is
For the late-step model, the corresponding nominal interval is
Both nominal intervals exclude zero. The late-step estimate is more favorable and its interval is also higher, but these calculations treat each fitted model as if it had been selected in advance. If the late-step model was chosen because it produced the larger estimate, its nominal interval does not account for that choice. Neither interval, by itself, establishes that the selected trend form is correct. The figures are hypothetical model outputs; the arithmetic shows how to compute the stated intervals, not how to reconstruct them from participant-level observations.
A suitable interpretation is that the two adjustment choices give different estimates of the treatment contrast in this hypothetical trial. The displayed intervals quantify model-based uncertainty conditional on each specified fit, not uncertainty from searching over trend forms and retaining a favorable result. The difference is a reason to examine the prespecified analysis and planned sensitivity models, not a reason to declare the more favorable form the winner.
How selection creates optimistic estimates
The basic effect can be seen without a complex trial model. Suppose two candidate analyses estimate the same fixed treatment effect \(\theta\), and, for a simple illustration, their estimation errors are independent and normally distributed with mean 0 and standard deviation \(\sigma\). Write the estimates as \(\hat{\theta}_1=\theta+Z_1\) and \(\hat{\theta}_2=\theta+Z_2\), where each \(Z_j\) has distribution \(N(0,\sigma^2)\). If the analyst reports the larger estimate, then
Taking expectations, the first two error terms have expectation 0. Since \(Z_1-Z_2\sim N(0,2\sigma^2)\), its expected absolute value is \(2\sigma/\sqrt{\pi}\). Therefore,
This derivation applies to the deliberately simplified case of two independent, equally precise estimates of the same effect. Real candidate models are often correlated, may have different standard errors, and may target different contrasts. The exact amount of selection bias then depends on the search and the dependence among results. The general lesson remains: choosing the largest estimate tends to make it too large on average. Choosing the smallest p-value also makes ordinary, unadjusted p-values and intervals overstate the strength of evidence, because they ignore the process that selected the result.
Worked Example: Selecting a late window changes the target
Worked Example: Selecting a Late Window Changes the Target
Hypothetical data and assumptions. Suppose 240 participants are enrolled in a randomized trial. An analyst considers three nested late-enrollment windows, each with both treatments represented: the last 40% of enrollment contains 96 participants, the last 25% contains 60, and the last 15% contains 36. Assume outcomes are measured in points, higher is better, and the fitted standard errors below are valid for each window-specific randomized contrast.
The estimated active-minus-control contrasts are 3.2 points (SE 1.5) in the last 40%, 5.0 points (SE 2.2) in the last 25%, and 7.0 points (SE 3.2) in the last 15%. These estimates concern different participant subsets. The analyst selects the last 15% because it gives the largest estimated contrast.
For the selected window, the nominal 95% Wald interval is
The interval excludes zero under the stated nominal calculation, but its width reflects substantial uncertainty, and it does not account for choosing this window after comparing three results. More importantly, the selected estimate is a contrast among the last 15% of enrollees, not automatically an estimate of the average treatment effect over all 240 participants. A late window might be scientifically meaningful if defined in advance and justified by the question. It cannot be treated as a neutral adjustment that preserves the original all-enrollee target merely because its result is favorable.
This example involves two issues that should not be conflated: searching over windows can affect inference, and changing the included participants can change the estimand. A careful analysis states both which population or time period the contrast concerns and how that period was chosen.
Worked Example: Choosing a spline knot from the outcome results
Worked Example: Choosing a Spline Knot From the Outcome Results
Hypothetical data and assumptions. In another hypothetical trial, investigators fit a common-treatment-effect model with a linear spline for enrollment time. For a candidate knot \(k\), the time component includes \(t_i\) and \((t_i-k)_+=\max(0,t_i-k)\); the latter allows the time slope to change after \(k\). Assume 180 participants, positive treatment overlap around each candidate knot, independent participant outcomes conditional on the model, and valid model-based standard errors. The target is the common active-minus-control contrast in outcome points.
The candidate knots are \(k=0.40,\ 0.60,\ 0.80\). The resulting hypothetical estimates and standard errors are 3.1 (1.3), 4.2 (1.4), and 3.5 (1.5) points, respectively. If the analyst chooses \(k=0.60\) because it gives the largest estimated contrast, the nominal interval is
This calculation is correct for the stated Wald procedure conditional on the selected fitted model. It is not a selection-adjusted 95% interval for the search over knots. The knot changes how the model represents the outcome-time relationship, and a different knot can change the adjusted treatment estimate. The outcome-informed choice therefore creates a risk of optimistic inference even when all candidate models are plausible and all fits converge.
A prespecified strategy and useful checks
A practical safeguard is to define the primary analysis before treatment-specific outcome patterns are examined. The plan need not pretend that the true time trend is known. Instead, it should specify a defensible primary model and a limited set of sensitivity analyses that probe plausible alternatives. Prespecification is most useful when it records the reason for the choices, not only the final formula.
State whether the target is an all-enrollee treatment effect, a contrast in a specified enrollment period, or a treatment effect allowed to vary with time. A time-window subgroup and an all-trial contrast are not the same target by default.
Choose the time scale, functional form, and any cutoffs or knots using design knowledge, external evidence, or a rationale fixed before examining treatment-specific outcomes. State the treatment-effect structure as well as the time adjustment.
For example, compare the primary linear adjustment with a prespecified step representation and a spline with fixed knots. Explain what feature of plausible drift each alternative tests; do not keep adding models until a preferred result appears.
Inspect outcome patterns over enrollment time, residual patterns, influential observations, and treatment overlap in each period. If a proposed window has very few participants or little representation from one treatment, its contrast may be unstable or poorly supported.
Look for changes in magnitude, uncertainty, or direction across the preplanned models. A different estimate can indicate dependence on modeling assumptions; it does not license replacing the primary model with whichever alternative looks most favorable.
Diagnostics can reveal that a prespecified model is inadequate—for example, systematic curvature in residuals against enrollment time or a few influential late observations. Such a finding should be documented and can motivate an additional analysis, but it does not erase the fact that the model was prompted by observed data. If a new model is added after inspecting outcomes, identify it as exploratory or use an inferential method that accounts for the selection process. A bootstrap that repeats the entire model-selection rule can sometimes quantify selection effects, but only if the rule is clearly defined and the resampling scheme is appropriate for the trial’s randomization and dependence structure.
Common mistake, alternatives, and limits
Wrong reasoning: “We tried linear time, several cutoffs, and several knots. The model with the largest treatment estimate had an interval excluding zero, so it is the best-supported primary analysis.”
Correction: The result was selected using the same outcomes on which its estimate and interval are based. The nominal interval does not include uncertainty from the search, and the selected model may target a different contrast if it restricts the analysis to a late period. Use the prespecified primary model for the primary inference, report planned sensitivity analyses as checks on assumptions, and label outcome-prompted analyses as exploratory unless selection is properly accounted for.
A linear trend is simple and can be stable, but it may miss curvature. A step or window-based approach can represent an abrupt change, but the cutoff must have a defensible rationale and a subgroup contrast may change the target. Splines allow smoother curvature, but knot placement and the number of knots affect flexibility and can themselves become selection opportunities. Flexible modeling is not automatically wrong; unplanned flexibility guided by favorable treatment results is the concern. Likewise, prespecification does not guarantee a correct model. If the primary model poorly represents drift, its estimate may still be biased or inefficient.
No single sensitivity set resolves every limitation. Sparse treatment overlap can make estimates highly dependent on modeling assumptions; a more flexible trend can worsen instability rather than fix it. A change in the treatment effect over time requires a clear definition of the desired summary contrast. Finally, prespecified models can be amended for compelling reasons, but the rationale, timing, and effect on interpretation should be transparent rather than concealed as if the choice had been made in advance.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- An analyst compares four spline knots and reports the knot with the smallest p-value. What is the inferential problem?
- Why might a contrast estimated only among the last 20% of enrollees not answer the same question as an all-enrollee treatment contrast?
- A prespecified linear time model has visible curvature in its residuals. Does that make it appropriate to replace the primary result with whichever spline estimate is largest? Explain.
- Give two design-based or scientific reasons for fixing a knot before examining treatment-specific outcomes.
- In the simplified derivation with two independent estimates, why is the expected selected maximum greater than the true effect?
Solutions. (1) The knot was selected using the outcomes, so the ordinary p-value does not account for searching over four candidates; it can overstate evidence. (2) The late-period contrast concerns a restricted enrollment subset, while the all-enrollee contrast averages over a broader population and time distribution. (3) No. Curvature is a diagnostic that may justify a documented additional analysis, but choosing the largest spline estimate creates further outcome-informed selection; the primary and additional analyses must be distinguished. (4) A knot might be set using an externally documented clinical change date or a fixed enrollment-time feature motivated by design knowledge. (5) The maximum includes a nonnegative absolute-difference term whose expectation is positive, so selecting the larger noisy estimate produces upward bias on average.