Why an Added Look Needs Calibration
A fixed-design test specifies a final analysis and controls its type I error under the null hypothesis. Adding an interim analysis provides an opportunity to examine accumulating data earlier. If the interim result is used to make an additional rejection decision, however, the trial now has more than one opportunity to reject. Simply applying the original significance threshold at both analyses generally increases the probability of at least one false rejection.
A different strategy is to calculate how much type I error the original design had left, conditional on the interim data that have actually been observed. This quantity is the conditional error. A continuation test that uses no more than this conditional allowance preserves the original unconditional error bound. In the normal model below, the calculation also gives an explicit continuation threshold.
Fixed Design and Conditional Error
Suppose the original design has a single, one-sided final test. Under the null hypothesis \(H_0\), its final standardized statistic \(Z_1\) has a standard normal distribution, and it rejects when \(Z_1\ge c\). Write \(\Phi\) for the standard normal cumulative distribution function. The original type I error is then \(\alpha=1-\Phi(c)\).
Now add an interim look at information fraction \(t\), where \(0<t<1\). Use the canonical normal model: under \(H_0\), the interim statistic \(Z_t\) and the independent future increment \(W\) are each standard normal, and \(Z_1=\sqrt{t}Z_t+\sqrt{1-t}W\). This representation is appropriate when information accumulates as specified, the standardized score has the stated joint normal distribution, and the future increment is independent of the interim data under \(H_0\). The fraction \(t\) is the interim information divided by the original planned final information.
Given \(Z_t=z\), only the independent future increment \(W\) remains random. Substituting \(z\) into the fixed-design final statistic gives $$ A(z)=P_0\left(\sqrt{t}z+\sqrt{1-t}W\ge c\right) =1-\Phi\left(\frac{c-\sqrt{t}z}{\sqrt{1-t}}\right). $$ The expression is the original test’s remaining rejection probability after learning the interim result. It is not a new significance level that can be spent independently at every look.
Choosing the Continuation Threshold
Suppose the trial continues and the final decision is based on the same canonical normal final statistic. At interim value \(z\), choose a final-statistic threshold \(b(z)\) so that the conditional probability of crossing it equals the original conditional error. Since the conditional distribution of \(Z_1\) is normal with mean \(\sqrt{t}z\) and variance \(1-t\), the required threshold is $$ P_0(Z_1\ge b(z)\mid Z_t=z)=A(z), \qquad b(z)=c. $$ Thus, if the original final statistic and its boundary are retained, the conditional error is preserved automatically: the interim look reveals information about the final statistic but does not change the final rejection rule.
More generally, suppose the continuation test uses an incremental statistic \(W\), which is standard normal under \(H_0\) conditional on the interim data. To attain conditional rejection probability \(A(z)\), reject at the final analysis when \(W\ge a(z)\), where $$ a(z)=\Phi^{-1}(1-A(z)) =\frac{c-\sqrt{t}z}{\sqrt{1-t}}. $$ This is the same original final boundary expressed as a threshold on the future increment. A different continuation statistic can also be calibrated: if its conditional null distribution given the interim data is \(F_z\), a continuous upper-tail test uses threshold \(F_z^{-1}(1-A(z))\). The conditional distribution must be specified correctly for this calibration to be valid.
Worked Example: Conditional Error at a Specific Interim Result
Consider a hypothetical one-sided fixed design with final boundary \(c=2.035\), and suppose the new look occurs at \(t=0.25\). The observed interim statistic is \(z=1\). Under the stated canonical normal null model, the original test’s conditional error is $$ A(1)=1-\Phi\left(\frac{2.035-\sqrt{0.25}(1)}{\sqrt{0.75}}\right) =1-\Phi\left(\frac{1.535}{0.8660}\right) \approx 1-\Phi(1.772) \approx 0.0382. $$
For a continuation test based on the future increment, the threshold is \(a(1)=1.772\): reject if \(W\ge1.772\). Expressed as a threshold on the combined final statistic, this is $$ b(1)=\sqrt{0.25}(1)+\sqrt{0.75}(1.772) =0.5+0.8660(1.772) \approx 2.035. $$ The conditional rejection probability is about \(0.0382\), matching the original test’s conditional error. The original unconditional error is \(1-\Phi(2.035)\approx0.0209\); a conditional error larger than that at a particular interim result is possible because the interim result itself is random.
Why Conditional Calibration Preserves the Error Bound
The key justification is the law of total probability. Let \(R\) denote rejection at the final analysis after adding the look. If, for every interim result \(z\), the continuation rule has conditional rejection probability \(A(z)\), then $$ P_0(R)=E_0\{P_0(R\mid Z_t)\} =E_0\{A(Z_t)\} =E_0\{P_0(Z_1\ge c\mid Z_t)\} =P_0(Z_1\ge c)=\alpha. $$ The equalities use, respectively, conditioning on the interim statistic, the conditional-error calibration, and the law of total probability applied to the original fixed test. Thus exact matching preserves the original type I error. If the continuation probability is no greater than \(A(z)\) for every \(z\), the same argument gives \(P_0(R)\le\alpha\).
This argument depends on calculating the conditional error under the same null model and conditioning information used to calibrate the continuation test. For a composite null, preserving error at one null parameter value does not by itself establish control over the whole null; the conditional-error bound must be valid for the relevant null values, or the design must use a suitable worst-case bound.
Worked Example: How the Threshold Changes With the Interim Result
Suppose a hypothetical fixed design rejects for \(Z_1\ge1.96\), and an added look occurs halfway through the planned information, so \(t=0.5\). The continuation threshold on \(W\) is \(a(z)=(1.96-\sqrt{0.5}z)/\sqrt{0.5}\).
If \(z=0\), then $$ a(0)=\frac{1.96}{0.7071}\approx2.772, \qquad A(0)=1-\Phi(2.772)\approx0.0028. $$ The interim result is not positive, so the conditional probability that the original final test would reject is small. The continuation threshold on the future increment is correspondingly high.
If instead \(z=1.5\), then $$ a(1.5)=\frac{1.96-0.7071(1.5)}{0.7071} =\frac{0.8993}{0.7071}\approx1.272, \qquad A(1.5)=1-\Phi(1.272)\approx0.1017. $$ Here the interim statistic is more favorable, so the conditional error is larger and the required future-increment threshold is lower. In either case, converting back to the combined final statistic gives the same boundary, \(Z_1\ge1.96\), and therefore the same conditional rejection probability as the original design.
Adding a Look Without Adding an Uncalibrated Rejection Chance
The formulas also clarify an important distinction. Merely collecting interim data, while making no interim rejection decision and retaining the original final test, does not create a second rejection opportunity. The interim data may be used to express the final rule conditionally, but the final test still has its original rejection probability.
By contrast, applying the same nominal boundary at the interim and again at the final analysis is not generally valid. The probability of rejection at either look is the probability of a union of events, which is at least the probability of rejection at the final look and can be larger. The conditional-error approach calibrates the continuation probability to the original allowance; it does not authorize an additional unadjusted test.
If a design includes an interim decision that rejects immediately, stops for futility, or otherwise removes some paths from the original final test, that decision must be included in the conditional-error accounting. A deterministic early rejection on a set of interim outcomes has conditional rejection probability one on that set, and cannot simply be added while claiming the original conditional-error bound there. The continuation rule must account for the probability already assigned to rejection, and stopping without rejection can make the overall test conservative.
Record its null model, final statistic, rejection boundary, and type I error.
Specify the information fraction and the interim statistic, and verify the conditional null distribution of future data.
Evaluate the original test’s conditional rejection probability at the observed interim data.
Choose a continuation rejection probability no larger than the conditional error, using the correct conditional distribution.
Average the conditional rejection probabilities under the null and verify that the result is no greater than the original error bound.
Assumptions, Diagnostics, and Limitations
The normal formula requires more than a normally distributed interim statistic by itself. The final statistic must have the stated joint relationship with the interim statistic, the future increment must have the stated conditional distribution, and \(t\) must represent the information fraction for the planned final analysis. If the actual information fraction differs, or if the variance or dependence structure is misspecified, the calculated conditional error can be wrong.
Useful checks include verifying the interim and planned final information, confirming that the future increment is independent of interim data under the null in the assumed model, and checking that the continuation threshold reproduces the intended conditional probability. If the actual continuation test uses a different statistic, its conditional null distribution must be calculated or justified rather than assumed to be standard normal. Discrete tests may not attain every conditional error exactly without randomization; a nonrandomized rule can instead be conservative.
The main alternatives serve different purposes. Keeping the original final test and ignoring the interim result is simple and preserves its size, but gives up the opportunity to adapt decisions using that result. Repeatedly applying an unadjusted nominal threshold is simple but can inflate type I error. Conditional-error calibration allows the interim result to inform the continuation rule while preserving the original bound, provided its assumptions hold. It does not, by itself, guarantee good power, resolve misspecification, or justify unplanned changes to the test.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- Under the canonical normal model, what is the conditional distribution of the final statistic given \(Z_t=z\)?
- A one-sided fixed test has boundary \(c=1.645\) and an added look at \(t=0.36\). For observed \(z=0\), calculate the conditional error and the future-increment threshold.
- Why does matching the continuation probability to the conditional error preserve the unconditional type I error?
- Is a conditional error larger than the original \(\alpha\) necessarily a sign of error in the calculation? Explain.
- Name two model or implementation checks needed before using the normal continuation-threshold formula.
Solutions. (1) Given \(Z_t=z\), \(Z_1\sim N(\sqrt{t}z,1-t)\). (2) The threshold is \(a(0)=1.645/\sqrt{0.64}=2.056\), so \(A(0)=1-\Phi(2.056)\approx0.0199\). (3) By the law of total probability, the unconditional rejection probability is the expectation under \(H_0\) of the conditional rejection probability; if this equals the original conditional error at every interim result, that expectation equals the original test’s \(\alpha\). (4) No. Conditional error varies with the interim result, and its null expectation, not each pointwise value, equals \(\alpha\). (5) Examples include confirming that the information fraction is correct and verifying the assumed conditional distribution and independence of the future increment under the null.