Why Selection Changes the Null Distribution
In a seamless trial, interim results may determine which treatment continues into the confirmatory stage. If the selected treatment’s interim observations are then pooled with its later observations and analyzed as though that treatment had been chosen in advance, the usual pooled test may no longer have its advertised Type I error rate. The issue is not that pooling observations is inherently invalid. It is that the observations used to select a treatment also influence the test statistic.
The effect can arise even when every treatment is ineffective. Among several noisy interim estimates, the largest is likely to be above its true value simply because it won the comparison. Reusing that favorable estimate gives the selected treatment a head start in the final test. This tutorial quantifies that mechanism for two candidate treatments under a simple null model.
The calculation below isolates selection’s effect with standardized treatment-versus-control statistics. Assume the complete null: both candidate treatments have no effect relative to control. Let \(Z_1,Z_2\) be the interim-stage test statistics for the two candidates, and \(W_1,W_2\) their later-stage statistics. For this derivation, assume all four are independent standard normal random variables under the null. This is a simplifying model: in a trial with a shared control group, the interim statistics may be correlated. The mechanism remains, but the numerical calculation must then reflect that dependence.
Suppose the trial selects the candidate with the larger interim statistic, \(J=\arg\max_{j\in\{1,2\}}Z_j\). Assume equal information in the two stages, so the ordinary pooled statistic for that selected candidate is \(T=(Z_J+W_J)/\sqrt{2}\). Under a prespecified one-sided test at \(\alpha=0.025\), the usual critical value is approximately 1.96. If the treatment had been prespecified independently of the interim statistics, its pooled statistic would be standard normal under the null and its rejection probability would be 0.025. After selection, that reference distribution no longer describes \(T\).
Deriving the Rejection Probability
Let \(M=\max(Z_1,Z_2)\). Its cumulative distribution function is \(P(M\le x)=\Phi(x)^2\), because both independent interim statistics must be at most \(x\). Differentiating gives the density \(f_M(x)=2\phi(x)\Phi(x)\). The later-stage statistic for the selected candidate, \(W_J\), is still standard normal and independent of the interim data under these assumptions. Conditional on \(M=x\), the pooled test rejects when \((x+W_J)/\sqrt{2}>1.96\), or equivalently when \(W_J>2.772-x\).
The integral averages the conditional rejection probability over the distribution of the selected interim statistic. The resulting null rejection rate is about 0.05, roughly twice the nominal one-sided level of 0.025. Thus an ordinary pooled test at that level does not control Type I error for this selection-and-reuse procedure. The value is specific to the assumptions and rule stated here; it is not a universal inflation factor.
The source of the shift is visible without evaluating the integral. Since \(M\) is the maximum of two standard normal variables, it tends to be positive even under the null. In fact, its expected value is \(1/\sqrt{\pi}\), approximately 0.564. The later-stage statistic has mean zero, but adding it to an interim statistic selected for being large tends to raise the pooled statistic. Importantly, \(T\) is not simply a standard normal statistic with a fixed mean shift: its distribution reflects the entire selection rule.
Worked Example: Comparing the Nominal and Selected-Treatment Error Rates
Consider the two-treatment model above. Both candidates are null, the trial selects the larger interim \(Z\)-statistic, the stages have equal information, and the pooled test rejects for \(T>1.96\). Evaluate the rejection probability and compare it with the prespecified-treatment rate.
For a treatment specified before observing interim data, its two independent stage statistics each follow \(N(0,1)\). Their standardized pooled statistic is therefore standard normal, giving \(P_{H_0}(T>1.96)=1-\Phi(1.96)\approx 0.0250\).
With selection, the interim statistic in the pooled test is \(M=\max(Z_1,Z_2)\). Its density is \(2\phi(x)\Phi(x)\). Given \(M=x\), the probability of crossing the same pooled-test boundary is \(1-\Phi(1.96\sqrt{2}-x)\), where \(1.96\sqrt{2}\approx2.772\). Substitution gives \[ \int_{-\infty}^{\infty}2\phi(x)\Phi(x) [1-\Phi(2.772-x)]\,dx\approx0.05. \]
Under this model, the chance of a false positive is about 5 in 100 trials, not the nominal 2.5 in 100. The comparison concerns repeated trials conducted under the complete null; it does not mean that any particular observed positive result has a 5% probability of being false.
What the Pooled Test Does to an Observed Result
The next hypothetical example shows how ordinary pooling can produce a small p-value after the selected interim result contributes directly to the final statistic. The values are standardized summaries, not results from an actual trial. For clarity, suppose both stage estimates have standard error 2 points and equal information, and test the selected treatment against a zero treatment effect.
Worked Example: A Favorable Interim Result Carries Forward
At interim, candidate A has an estimated effect of 4.8 points, giving \(Z_A=4.8/2=2.40\). Candidate B has an estimated effect of 2.2 points, giving \(Z_B=2.2/2=1.10\). The prespecified selection rule continues the candidate with the larger interim statistic, so A is selected. In the later stage, A’s estimate is 2.4 points with standard error 2, giving \(W_A=2.4/2=1.20\).
The equal-information pooled estimate for A is the average of the two stage estimates: \((4.8+2.4)/2=3.60\) points. Its standard error is \(2/\sqrt{2}\approx1.414\) points, so the pooled statistic is \[ T=\frac{3.60}{1.414}\approx2.546 =\frac{2.40+1.20}{\sqrt{2}}. \]
The one-sided ordinary normal p-value is \(1-\Phi(2.546)\approx0.00545\). This is below 0.025, so the naive pooled test rejects. The p-value calculation is correct for a treatment and test specified independently of the data used in that test. But A was selected because its interim statistic was the larger one. Under the complete null, such a favorable interim value can occur by chance; treating it as prespecified does not account for that selection.
This example does not establish that the treatment is ineffective, nor does selection guarantee rejection. It illustrates why the ordinary p-value cannot be interpreted as calibrated for the full adaptive procedure. A p-value computed from a statistic has its usual null interpretation only when the assumed null distribution matches how the statistic was generated, including the selection step.
Alternatives, Diagnostics, and Limits
The main alternative for the narrow problem considered here is to avoid carrying selection-affected interim evidence into an ordinary confirmatory test. For example, if the selected treatment is tested using only independent later-stage data, then under the complete null the later-stage statistic for whichever candidate was selected remains standard normal, conditional on the interim selection. This protects the one-sided test’s nominal level in this simplified setting, although it does not recover the precision that reuse might provide.
Other designs can account for selection while retaining interim information, but they require a testing procedure calibrated for the adaptive rule. The defining requirement is that the final null distribution or error-control argument incorporate treatment selection rather than treating the selected treatment as if it had been fixed in advance. The ordinary pooled test is not a substitute for that calibration. When there are multiple confirmatory claims, endpoints, or departures from the complete null, additional multiplicity and error-control questions also arise; the two-treatment calculation does not resolve them.
| Approach | How interim information is handled | Main implication |
|---|---|---|
| Naive pooling | Selected treatment’s interim and later data are pooled; selection is ignored in the test reference distribution. | Can inflate Type I error. |
| Independent later-stage test | Selection uses interim data, but the test statistic uses independent later-stage data only. | In the stated complete-null model, selection does not change the later-stage null distribution; interim information is not reused for the test. |
| Selection-aware confirmatory test | The procedure calibrates inference to the selection and testing rule. | Can use information in a principled way, but requires a method and assumptions suited to the design. |
Diagnostics should start with the actual decision rule, not just the final analysis code. Verify whether the interim endpoint and treatment comparisons used for selection enter the final statistic; document which patients contribute to each stage; and check whether treatment selection and stagewise estimates are independent under the assumed null. A simulation under a specified null can help evaluate the full procedure: repeatedly generate data under the null, apply the selection rule, perform the planned final test, and estimate the rejection proportion. The simulation must reproduce the allocation, shared controls, stage sizes, and decision rules; otherwise it may miss the dependence that matters.
A common warning sign is a protocol that describes treatment selection as adaptive but reports a conventional pooled p-value as if no selection occurred. Conversely, seeing a favorable interim statistic does not itself prove that the final test is invalid: the issue is whether the inferential procedure accounts for how that statistic was used. The calculation here assumes independent standard normal stage statistics, equal stage information, two candidates, a maximum-statistic selection rule, and a one-sided test. Different correlations, numbers of candidates, selection thresholds, information fractions, or test directions change the null rejection probability.
Common mistake: “The pooled statistic uses data from both stages, so under the null it must be standard normal.” That reasoning would apply to a treatment fixed independently of the data, with the stated stage assumptions. After selecting the larger interim result, the interim component is no longer an unselected standard normal draw. The correction is to derive or simulate the null distribution of the complete selection-and-testing procedure, rather than applying the prespecified-treatment reference distribution without adjustment.
Worked Example: Testing With Independent Later-Stage Data
Suppose the interim statistics select candidate B, but the confirmatory statistic uses only B’s independent later-stage data. In a hypothetical trial, that later-stage statistic is \(W_B=2.10\). Under the complete null and the independence assumptions above, conditional on any interim selection, \(W_B\sim N(0,1)\). Its one-sided p-value is \(1-\Phi(2.10)\approx0.0179\), below the stated level 0.025, so this particular later-stage-only test rejects.
The relevant calibration follows from the conditional null distribution: for any selected candidate \(J\), independence gives \(P(W_J>1.96\mid Z_1,Z_2)=1-\Phi(1.96)\approx0.025\). Averaging over the interim results leaves the probability at approximately 0.025. This does not say every later-stage result is valid in every adaptive design; it shows why excluding selection-affected interim data avoids this particular inflation under the stated assumptions.
Check Your Understanding
Answer each question, then compare with the brief solutions.
- Under the two-candidate model, what is the density of the larger interim statistic \(M\), and why?
- Why does a prespecified candidate’s pooled statistic have a different null distributional argument from the statistic for the selected candidate?
- With equal information, \(Z_J=1.8\) and \(W_J=0.6\). Calculate the pooled statistic and state whether it exceeds 1.96.
- Why does using only independent later-stage data avoid the specific reuse problem described here?
- Name two assumptions of the numerical inflation calculation that might fail in a practical seamless trial.
Solutions. (1) \(f_M(x)=2\phi(x)\Phi(x)\), since \(P(M\le x)=\Phi(x)^2\) and differentiation gives the density. (2) A prespecified candidate is not chosen for having a favorable interim statistic; selection makes the interim component conditional on being the larger value. (3) \(T=(1.8+0.6)/\sqrt{2}=2.4/1.414\approx1.697\), which does not exceed 1.96. (4) Under the model, later-stage statistics are independent of the interim selection, so the selected candidate’s later-stage statistic retains its standard normal null distribution. (5) Examples include a shared control that correlates candidate statistics, unequal stage information, more than two candidates, or a selection rule other than choosing the largest interim statistic.