Selection on a Noisy Interim Statistic

Why Selection Changes the Error Probability

Suppose an interim review compares two candidate treatments and retains the one with the more favorable interim statistic. If that statistic is noisy, the selected treatment tends to have benefited from a positive random fluctuation. A later statistic based on additional, correlated data may also tend to be high, even when neither treatment truly works. As a result, testing the selected treatment at an unchanged fixed threshold need not have the nominal probability of a false rejection.

The calculation here concerns the global null: both candidate treatments have no effect relative to the control, and the null hypotheses for both candidates are true. It isolates the effect of choosing the larger interim statistic in a simple model. The resulting probability is not universal; it depends on how the interim and later statistics are related.

Definition: Let \(X_1\) and \(X_2\) be interim statistics oriented so that larger values favor their respective treatments. The selection rule is \(J=\arg\max_{j\in\{1,2\}}X_j\). A later test rejects for the selected treatment if its final statistic \(Z_J\) exceeds a prespecified upper-tail critical value \(c\). Under the global null, \(P(Z_J>c)\) is the probability of a false rejection for this procedure.

For a concrete calculation, assume the following model under the global null:

X_1,X_2,\varepsilon_1,\varepsilon_2 \text{ are mutually independent }N(0,1), \qquad Z_j=\rho X_j+\sqrt{1-\rho^2}\,\varepsilon_j,\quad j=1,2, \quad 0\leq\rho<1.

Each interim and final statistic is marginally standard normal under its null, and the correlation between \(X_j\) and its own later statistic \(Z_j\) is \(\rho\). The model also assumes independent interim statistics across candidates and independent later residuals; in particular, \(Z_j\) has no additional dependence on the other candidate's interim statistic. These are simplifying assumptions, not automatic properties of an adaptive trial. We use a one-sided test with nominal level \(\alpha=0.025\), so \(c\) is the standard normal upper-tail critical value, approximately 1.96.

Deriving the Selected Rejection Probability

The selected interim value is \(M=\max(X_1,X_2)\). Since \(X_1\) and \(X_2\) are independent standard normal variables, its cumulative distribution function is \(P(M\leq x)=\Phi(x)^2\). Differentiating gives the density \(2\phi(x)\Phi(x)\). Conditional on \(X_j=x\), the model gives \(Z_j\sim N(\rho x,1-\rho^2)\). Thus the conditional probability of crossing the fixed boundary is \(1-\Phi((c-\rho x)/\sqrt{1-\rho^2})\).

Combining the density of the selected value with this conditional probability yields the main calculation.

Result: Under the stated model, the probability of rejecting for the selected candidate under the global null is

P(Z_J>c) = \int_{-\infty}^{\infty} 2\phi(x)\Phi(x) \left[ 1-\Phi\!\left(\frac{c-\rho x}{\sqrt{1-\rho^2}}\right) \right]dx.

This is generally larger than the fixed-test probability \(P(Z_j>c)=\alpha\) when \(\rho>0\): the selected interim value is unusually favorable on average, and some of that favorable fluctuation carries into the later statistic.

The same probability has a useful second representation. The events \(Z_1>c,\ X_1>X_2\) and \(Z_2>c,\ X_2>X_1\) are disjoint and have equal probability by symmetry; ties have probability zero. Given \(Z_1=z\), the bivariate normal model implies \(X_1\) has conditional mean \(\rho z\) and variance \(1-\rho^2\), while \(X_2\) remains an independent standard normal variable. Therefore \(X_1-X_2\) conditionally has mean \(\rho z\) and variance \(2-\rho^2\). Integrating over \(z>c\) gives

P(Z_J>c) =2P(Z_1>c,\ X_1>X_2) =2\int_c^\infty \phi(z)\Phi\!\left(\frac{\rho z}{\sqrt{2-\rho^2}}\right)dz.

This derivation also makes the selection mechanism visible: a final statistic above the boundary is more likely to belong to the selected candidate when its interim statistic tends to be high.

Worked Example: Numerical Global-Null Error Probability

Consider a hypothetical two-candidate trial using the model above, with interim-to-final correlation \(\rho=0.69\). Select the candidate with the larger interim statistic and test its later statistic at \(c=1.96\).

The model assumptions are that both candidates are null, \(X_1\) and \(X_2\) are independent standard normal statistics, and each final statistic has the specified correlation with its own interim statistic and an independent normal residual. The fixed test for either prespecified candidate has upper-tail probability about 0.0250. But the selected test has probability

\begin{aligned} P(Z_J>1.96) &= 2\int_{1.96}^{\infty} \phi(z)\Phi\!\left( \frac{0.69z}{\sqrt{2-0.69^2}} \right)dz\\ &\approx 0.0450. \end{aligned}

For example, \(2-0.69^2=1.5239\), so the denominator inside \(\Phi\) is approximately \(\sqrt{1.5239}=1.2345\). Numerical integration of the displayed expression gives approximately 0.0450. The increase over the nominal level is \(0.0450-0.0250=0.0200\), or 2.00 percentage points. The selected procedure's global-null rejection probability is about \(0.0450/0.0250=1.80\) times the nominal level. Thus the fixed 1.96 boundary does not, in this model, deliver a 2.50% false-rejection probability after selection.

What a Particular Interim Result Implies

The overall error probability averages over all possible interim results. It is also useful to examine the conditional probability after a specific interim review. Under the model, once candidate \(j\)'s interim value \(X_j=x\) is known, its later statistic has mean \(\rho x\) and standard deviation \(\sqrt{1-\rho^2}\). The conditional crossing probability is therefore

P(Z_j>c\mid X_j=x) = 1-\Phi\!\left(\frac{c-\rho x}{\sqrt{1-\rho^2}}\right).

Worked Example: Conditional Risk After a Favorable Interim Result

Suppose, hypothetically, that the interim statistics are \(X_1=1.5\) and \(X_2=0.4\). Candidate 1 is selected because \(1.5>0.4\). With \(\rho=0.69\), \(c=1.96\), and the model assumptions above, the conditional mean and standard deviation of \(Z_1\) are

E(Z_1\mid X_1=1.5)=0.69(1.5)=1.035, \qquad \sqrt{1-0.69^2}=\sqrt{0.5239}\approx0.7238.

Consequently,

\begin{aligned} P(Z_1>1.96\mid X_1=1.5) &=1-\Phi\!\left(\frac{1.96-1.035}{0.7238}\right)\\ &=1-\Phi\!\left(\frac{0.925}{0.7238}\right)\\ &\approx 1-\Phi(1.278) \approx 0.101. \end{aligned}

Conditional on this interim value, the chance of crossing the unchanged boundary is about 10.1% under the null, rather than the 2.50% chance for a prespecified test with no conditioning on its interim statistic. This is a conditional calculation for the stated model; it is not a claim that every observed selected result has this probability.

Assumptions, Diagnostics, and Alternatives

The calculation depends on the selection rule and the joint distribution, not just on the fact that two candidates were considered. Before using it to describe a trial, check the following features of the design and statistical model:

Selection rule: The derivation selects the larger of two statistics oriented in the favorable direction. Selecting the smaller p-value, selecting by a different interim measure, or allowing additional selection steps changes the event being calculated.
Null configuration: The numerical result assumes both candidates are null. It is a global-null calculation and does not by itself establish error control when one candidate is effective and another is null.
Dependence: The independence of the interim statistics and residual terms is part of the model. Shared controls, correlated outcomes, or overlapping data can create other correlations. If these features are present, the displayed integral need not apply.
Interim-to-final relationship: The value \(\rho=0.69\) determines how much the interim fluctuation carries forward. It must represent the joint behavior of the actual statistics under the null; it is not a universal constant.
Direction and boundary: The calculation uses an upper-tail test and a fixed threshold. A two-sided test or a different prespecified threshold requires changing the rejection event.

A useful diagnostic is to write down the selection event and the final rejection event together, then check whether the final statistic's distribution remains the same after selection. Under the model here, it does not: \(Z_J\) is associated with the favorable interim statistic that defined \(J\). The marginal validity of each \(Z_j\) when tested separately does not imply that the statistic selected using the \(X_j\)'s has the same marginal distribution as a prespecified one.

There are important alternatives. If one candidate is specified in advance, testing its later statistic at the fixed one-sided 0.025 boundary has global-null rejection probability 0.025 in this model. Alternatively, suppose interim selection is followed by a new statistic \(W_j\) that is independent of all interim data and is standard normal under the null for either candidate. For any selection rule based only on the interim data,

P(W_J>c) = \sum_{j=1}^{2}P(J=j)P(W_j>c\mid J=j) = \sum_{j=1}^{2}P(J=j)\alpha =\alpha.

That independent-confirmation argument differs from using \(Z_J\), which is correlated with the interim data. It depends on genuine independence and valid null distributions; it is not achieved merely by labelling later observations “confirmatory.” Other adaptive testing strategies can account for interim information by a design-specific method, but the selected fixed-boundary calculation above should not be mistaken for such a correction.

Common Mistakes and Limitations

Using the nominal level for the selected test. The reasoning “each candidate's test has size 0.025, so the selected one also has size 0.025” ignores that selection uses a statistic associated with the later test. The calculation gives about 0.0450 under the stated model.
Counting two tests as if they were independent final claims. The procedure described makes one later rejection attempt, for the selected candidate. Its error probability is not obtained by simply adding two fixed-test probabilities; the interim selection and later dependence determine it.
Treating the numerical result as universal. The value 0.0450 follows from two independent interim statistics, \(\rho=0.69\), and the specified normal model. Changing the number of candidates, dependence, selection rule, or correlation changes the calculation.
Confusing conditional and unconditional error. The approximately 0.101 conditional probability in the worked example is for a particular interim value. The approximately 0.0450 unconditional probability averages over all possible interim results under the model.

The calculation is a focused illustration of selection on noisy interim evidence. Its main practical value is to expose the dependence that a design's error-control argument must address. A numerical model can diagnose a risk, but it does not establish error control for a more complex adaptive design without checking that the model and null configuration match that design.

Check Your Understanding

Use the model and notation from this tutorial. The solutions are included after the questions.

If the later statistic is independent of its interim statistic (\(\rho=0\)), what is the selected global-null rejection probability at \(c=1.96\)? Explain briefly.
For \(\rho=0.69\), calculate the conditional rejection probability at \(X_j=1.0\), using \(c=1.96\) and standard deviation approximately 0.7238.
Using the numerical result for \(\rho=0.69\), state the absolute increase over the nominal 0.025 level in probability and in percentage points.
In the limiting model \(\rho=1\), so that \(Z_j=X_j\), express the selected rejection probability at \(c=1.96\) using \(\Phi(1.96)\), and give its approximate value.

Solutions. (1) When \(\rho=0\), the final statistics are independent of the interim selection. Conditional on either selected candidate, its final statistic is still standard normal, so \(P(Z_J>1.96)=1-\Phi(1.96)\approx0.0250\). (2) The conditional mean is \(0.69(1.0)=0.69\), so the standardized boundary is \((1.96-0.69)/0.7238=1.754\). The tail probability \(1-\Phi(1.754)\) is approximately 0.0397. (3) The increase is \(0.0450-0.0250=0.0200\), equal to 2.00 percentage points. (4) With \(Z_j=X_j\), rejection occurs exactly when the larger interim statistic exceeds 1.96. Hence the probability is \(P(\max(X_1,X_2)>1.96)=1-\Phi(1.96)^2\approx1-(0.975)^2=0.0494\).

Selection on a Noisy Interim Statistic

What You'll Learn