Three Questions, Three Statistical Objects
An adaptive trial may change its sample size, allocation, eligible population, or analysis procedure as information accumulates. These changes can affect what is estimated, how precisely it is estimated, and how a claim is tested—but these are not the same thing. A clear analysis begins by asking three separate questions: What clinical quantity is the trial intended to learn about? What calculation estimates that quantity from the observed data? What rule determines whether the evidence is sufficient to reject a null hypothesis?
The estimand is the clinical target. The estimator is a data-based rule for learning about that target. A test combines a statistic with a null distribution and a decision rule. Changing an estimator or test does not automatically change the estimand; conversely, retaining the same analysis code does not guarantee that the clinical target is unchanged. The distinction is especially important when an adaptation changes which participants contribute to the final analysis.
For example, suppose the target is the mean difference in a clinical score at week 12 between assignment to a new treatment strategy and assignment to control, in a fixed target population \(P^*\). If \(Y(1)\) and \(Y(0)\) denote the potential week-12 outcomes under those two treatment conditions, the target can be written as:
The statement is incomplete unless the population, treatment conditions, outcome, time point, and handling of events such as treatment discontinuation are defined. Those specifications determine the clinical question. A sample mean difference, a regression-adjusted contrast, or a standardized contrast might each be an estimator for \(\theta\), provided its assumptions and implementation support that target. A test of \(H_0:\theta=0\) is a further choice: it assesses a specified null about the target, rather than defining the target itself.
How an Estimator Can Preserve a Fixed Target
A useful way to see the distinction is to write an estimator as a rule for combining observed data. Suppose a target population is divided into baseline subgroups \(s=1,\ldots,K\). Let \(\tau_s\) be the treatment effect in subgroup \(s\), and let \(w_s\) be the subgroup’s prespecified proportion in the target population, with \(w_s\geq 0\) and \(\sum_s w_s=1\). The fixed-population estimand is \(\theta=\sum_s w_s\tau_s\). If \(\hat{\tau}_s\) estimates the subgroup effect, a standardized estimator is \(\hat{\theta}=\sum_s w_s\hat{\tau}_s\).
The justification is direct. If each subgroup estimator is unbiased for its corresponding subgroup effect, so that \(E(\hat{\tau}_s)=\tau_s\), then:
Thus the estimator targets the same fixed-population effect even if the trial’s observed subgroup proportions differ from the target weights. This result relies on the subgroup estimates being valid for the subgroup effects and on applying the specified weights. It does not say that any analysis using subgroup data is valid: missing outcomes, model misspecification, nonrepresentative enrollment, or an ill-defined treatment contrast can undermine the estimator’s interpretation.
The test is distinct again. For a two-sided test of \(H_0:\theta=0\), a statistic might be \(T=\hat{\theta}/SE(\hat{\theta})\). Its null distribution and rejection boundary must be justified for the design and analysis actually used. A standard fixed-sample reference distribution cannot simply be assumed valid after a data-dependent adaptation if that adaptation changes the statistic’s null distribution. In that situation, the test may need a design-appropriate calibration. Calibration changes the evidence rule; it does not, by itself, change which clinical quantity \(\theta\) denotes.
Worked Example: A Welch Test for a Fixed Mean Difference
Hypothetical data. In a randomized trial, the endpoint is improvement in a symptom score at week 12, measured in points, with larger values indicating greater improvement. Assume the target is the mean difference in improvement for participants meeting the fixed trial eligibility criteria. There are 23 participants assigned to the new treatment, with \(\bar{x}_1=14.00\) points and \(s_1=10.00\) points, and 24 assigned to control, with \(\bar{x}_0=4.546\) points and \(s_0=11.58\) points. Assume independent participants, randomized assignment, and no influential departures from the sampling model for the group means. The Welch procedure does not require equal population variances; its approximate reference distribution is suitable here under the usual independent-sample conditions and adequate behavior of the group means.
Estimand and estimator. The estimand is the population mean difference in week-12 improvement, in points, for the specified population and treatment contrast. The estimator is the observed difference in sample means:
Test statistic. For \(H_0:\theta=0\), the Welch standard error is:
Therefore \(t=9.454/3.151\approx 3.000\). The Welch–Satterthwaite degrees of freedom are approximately 44.53, using \(\{s_1^2/n_1+s_0^2/n_0\}^2\) divided by \(\{(s_1^2/n_1)^2/(n_1-1)+(s_0^2/n_0)^2/(n_0-1)\}\). The two-sided p-value is approximately \(0.00441\). At \(\alpha=0.05\), this test rejects \(H_0\); the data provide evidence against a zero mean difference under the stated test assumptions. The p-value is the probability, under the null and the reference model, of a statistic at least as extreme as the observed one. It is not the probability that the null hypothesis is true. The estimate remains 9.454 points whether or not the test rejects.
Analysis Changes: Same Target or Different Target?
To audit a proposed change, write down the estimand first, then ask whether the proposed estimator still targets it and whether the test remains calibrated. The distinction is consequential when trial adaptations alter population composition. The introductory material on standardizing effects to a fixed population motivates using fixed weights when the clinical target is fixed; the calculation below makes explicit why the weights matter.
Worked Example: Standardizing Despite a Changed Subgroup Mix
Hypothetical data. The target population is defined to contain 50% of participants in subgroup A and 50% in subgroup B. The clinical outcome is symptom improvement in points, and the target is the fixed-population mean treatment effect. Suppose the estimated treatment effects are 8 points in subgroup A and 2 points in subgroup B. After an adaptive enrollment change, the analyzed sample is 80% subgroup A and 20% subgroup B.
With the prespecified target weights, the standardized estimate is:
If the subgroup effects are validly estimated, this calculation estimates the fixed 50:50 target-population effect. In contrast, simply pooling the subgroup effects according to the adapted sample composition gives:
That 6.8-point figure is a composition-weighted effect for an 80:20 mix, not the stated 50:50 target. It may answer a different clinical question. The change in subgroup mix therefore cannot be treated as a harmless estimator choice if the analysis silently replaces the fixed target weights with observed sample proportions. The numerical estimates alone do not establish precision or statistical significance; those require an appropriate variance estimate and test.
Some changes can leave the target and point estimate intact but change uncertainty or the decision rule. For instance, a revised variance estimator may change \(SE(\hat{\theta})\), while a test calibrated for an adaptive design may use a different reference distribution or rejection boundary than a conventional fixed-sample test. Whether that revision is appropriate depends on the design and its prespecified analysis plan. A change in the test’s calibration is not evidence that the clinical target changed, but an uncalibrated test can make its p-value or error rate unreliable.
Worked Example: Same Estimate, Different Testing Rule
Hypothetical analysis. A trial estimates a treatment effect of \(\hat{\theta}=2.05\) points, with a standard error of 1.00 point, for the same fixed population and week-12 outcome. The statistic is \(Z=2.05/1.00=2.05\). Suppose the design specifies a two-sided test with a standard normal reference distribution. The corresponding p-value is approximately 0.0404.
Under a test with \(\alpha=0.05\), the result rejects \(H_0:\theta=0\), since 0.0404 is less than 0.05. If the applicable, design-calibrated decision rule instead requires a two-sided p-value below 0.025, the result does not reject, since 0.0404 exceeds 0.025. The estimate is 2.05 points in both cases, and the estimand is unchanged. The test rule and resulting decision differ.
This comparison is valid only if the reference distribution and the applicable significance boundary are justified for the design. In an adaptive trial, one must not choose the more favorable boundary after seeing the result and then present it as prespecified. A change in rule needs a defensible calibration and transparent reporting; otherwise, the nominal error rate may not be controlled.
Diagnostics and Common Failure Modes
An estimand–estimator–test audit makes an adaptive analysis easier to inspect. For every final analysis, record the target in clinical terms, the data-based calculation, and the inferential rule separately. Then compare the protocol and analysis versions across adaptations. Useful diagnostic questions include:
- Population: Are the target population and its subgroup weights unchanged, or has the eligible or enrolled population changed the target?
- Treatment contrast: Do the treatment conditions still represent the same strategies, including relevant post-randomization events?
- Outcome: Is the endpoint, measurement time, and summary measure unchanged?
- Estimator: Does the chosen calculation estimate the stated quantity under explicit assumptions? Are missing data, model fit, and subgroup precision plausible?
- Test: Is its null hypothesis about the stated estimand, and is its reference distribution or critical value valid after the adaptation?
A common mistake is to say, “The estimand did not change because the endpoint and treatment arms are still named the same.” That is not enough: changing eligibility can alter the target population, and changing the endpoint’s time point can alter the outcome being summarized. The correction is to compare the complete estimand specification, not just its labels.
A second mistake is to conclude that a new estimator necessarily creates a new estimand. It does not. For example, a model-based estimator and a standardized estimator may target the same fixed-population contrast if both are justified for that target. The correction is to examine the quantity each estimator is intended to estimate and the assumptions connecting the observed data to it.
A third mistake is to treat a statistically significant result as proof that the estimand was preserved. A test addresses evidence against a null under its assumptions; it does not establish that the population, treatment contrast, or outcome specification remained fixed. Likewise, an unchanged estimate does not guarantee a valid p-value after an adaptation. The estimand, estimator, and test each require their own check.
| Object | Question it answers | Example of a change |
|---|---|---|
| Estimand | Which clinical quantity is targeted? | Changing the target population or outcome time point |
| Estimator | How is the target estimated from observed data? | Replacing an unadjusted contrast with a justified standardized estimate |
| Test | How is evidence against a null assessed? | Using a design-calibrated reference distribution or decision boundary |
Why the Distinction Matters
A clinical interpretation belongs to the estimand: it describes the target contrast in the target population and on the specified outcome scale. An estimate is a sample-based result for that target, subject to estimation uncertainty and assumptions. A p-value and rejection decision belong to the test. Keeping these statements separate prevents two opposite errors: interpreting a valid analysis change as a new clinical question, or overlooking a genuine change in the question because the analysis still produces a familiar-looking number.
In an adaptive trial, an adaptation can affect all three objects, or only one or two. Changing sample size may improve precision while preserving the target; changing a standardization method may preserve the target if correctly implemented; changing eligibility may alter the population estimand; and changing the inferential rule may alter the decision without changing the point estimate. The correct conclusion depends on the specification and assumptions, not on the label “adaptive.”
Check Your Understanding
Answer each question by identifying which statistical object is involved. Full solutions follow.
- A trial keeps its week-12 outcome and treatment contrast but changes the target subgroup weights from 50:50 to 70:30. What has changed?
- A fixed target is estimated by subgroup effects combined using prespecified target-population weights. What condition allows the weighted estimator to target the weighted estimand?
- A trial reports \(\hat{\theta}=3.0\) points and \(SE=1.5\) points. Calculate the test statistic for \(H_0:\theta=0\), and name one additional item required to obtain a p-value.
- Can a test’s rejection decision change while the estimand and point estimate remain the same? Explain.
Solutions. (1) The estimand’s target population has changed because its subgroup weights changed; the treatment contrast and outcome alone do not specify the full target. (2) Each subgroup estimator must validly estimate its corresponding subgroup effect, and the estimator must combine them using the fixed target weights: \(E(\sum_s w_s\hat{\tau}_s)=\sum_s w_s\tau_s\). (3) The statistic is \(3.0/1.5=2.0\). A p-value also requires a justified null reference distribution, including relevant degrees of freedom or other design-specific calibration. (4) Yes. A change in a valid significance boundary or null calibration can change the decision while leaving the target and estimate unchanged; the revised test must be justified for the design.