Why Removing a Look Requires Recalibration
A planned interim analysis can become unavailable or impractical, for example because data collection or trial operations no longer support a decision at that time. Simply deleting the interim decision and keeping the original final boundary is not generally safe: the original design’s probability of rejection may depend on its planned sequence of looks. A principled replacement starts with the information available at the last analysis that remains and asks how much probability of rejection the original design allowed beyond each possible history.
That history-specific allowance is the conditional error. A replacement test that spends no more than this allowance at every retained history cannot increase the null rejection probability. Matching the allowance exactly preserves the original error, subject to the model and conditioning assumptions described below.
The conditioning information matters. It includes the retained data and any observed design information that affects the original continuation rule or the distribution of the remaining statistics. If a study’s original rule uses the interim statistic, the complete interim history—not merely a label for the adaptation decision—will generally be needed to determine \(A(H)\). The earlier material on conditional error functions develops this conditioning principle and its relation to the conditional error preservation theorem.
Preservation by Conditional Expectation
The central justification is short, but its conditions matter. Suppose the original design has null rejection probability \(\alpha_0\), and the replacement is constructed under the same null model and retained history. If the replacement matches conditional error, then
The first and third equalities use the law of total expectation. Thus, matching conditional rejection probabilities at the retained histories preserves the original unconditional type I error. If instead the replacement has conditional rejection probability no greater than \(A(H)\) at every history, the same argument gives \(P_0(\phi_1=1)\leq\alpha_0\). That is conservative preservation, not exact preservation.
For a one-sided final test, suppose the final statistic \(T\), conditional on \(H\) under the null, has a continuous cumulative distribution function \(G_H\). To match a conditional error \(A(H)\), reject when
If \(T\mid H\sim N(m_H,s_H^2)\), where \(s_H>0\), this becomes
where \(\Phi\) is the standard normal cumulative distribution function. The mean and standard deviation here are conditional null quantities, not estimates chosen to make the desired cutoff appear. When \(A(H)=0\), the replacement must not reject at that history; when \(A(H)=1\), it must reject with probability one under the conditional null. For discrete statistics, a cutoff may not attain the target probability exactly without randomization.
Worked Calibrations
Worked Example: Removing a Look in a Simple Two-Decision Design
Consider a hypothetical continuation after a retained history \(H\). Under the null, the original design would observe two independent future statistics \(U\) and \(V\), each distributed as \(N(0,1)\) conditional on \(H\). It would reject at the omitted interim if \(U>c\); if not, it would reject at the final analysis if \(V>c\). Each one-sided tail probability at \(c=\Phi^{-1}(0.98)\approx2.05375\) is 0.02000. Find the conditional error and calibrate a replacement that uses only \(V\).
The design rejects either at the omitted look or, if that does not happen, at the final look. Independence gives
The replacement uses \(V\sim N(0,1)\) and must have upper-tail probability 0.03960. Its exact cutoff is \(\Phi^{-1}(1-0.03960)=\Phi^{-1}(0.9604)\approx1.7554\). Thus it rejects for \(V>1.7554\), with the unrounded quantile understood. This matches the original conditional rejection probability, even though the omitted interim decision no longer occurs. The example is deliberately simple: it assumes the final statistic is independent of the omitted statistic under the conditional null.
Worked Example: Calibrating a Conditional Normal Final Statistic
In a hypothetical trial, suppose the original continuation rule has conditional error \(A(H)=0.04990\) at a particular retained history. Under the null and conditional on that history, the final statistic available to the replacement has distribution \(T\mid H\sim N(0.200,0.84)\), where 0.84 is the variance. Find the replacement boundary.
The standard deviation is \(\sqrt{0.84}\approx0.916515\), and the standard normal upper-tail quantile for probability 0.04990 is \(\Phi^{-1}(1-0.04990)\approx1.64582\). Therefore,
The replacement rejects for \(T>1.70842\). Standardizing this boundary gives \((1.70842-0.200)/0.916515\approx1.64582\), so the conditional upper-tail probability is approximately 0.04990. If the cutoff were instead rounded to 1.708, the corresponding conditional tail probability would be about 0.04995; using the unrounded boundary, or reporting the probability for the rounded boundary, avoids claiming false precision.
Worked Example: Different Retained Histories, Different Boundaries
Suppose a hypothetical original design has conditional errors 0.03970 when the retained statistic \(X\leq0\), and 0.05950 when \(X>0\). Assume the replacement’s final statistic is conditionally standard normal under the null in either group: \(T\mid H\sim N(0,1)\). Determine the two final cutoffs.
For \(X\leq0\), the required standard normal upper-tail probability is 0.03970. The cutoff is \(\Phi^{-1}(1-0.03970)\approx1.7542\), so reject for \(T>1.7542\). For \(X>0\), the required upper-tail probability is 0.05950. The cutoff is \(\Phi^{-1}(1-0.05950)\approx1.5590\), so reject for \(T>1.5590\). The more permissive boundary in the second group is appropriate because that history has the larger original conditional error. The exact cutoffs are the respective quantiles; the displayed decimals are approximations.
This example illustrates why a single overall final boundary may not preserve conditional error. The replacement boundary can depend on the retained history even when the conditional distribution of \(T\) is the same across histories.
Assumptions and Practical Checks
The preservation argument is an identity once the conditional probabilities are correct. The challenge is ensuring that they are calculated for the right design and null model. In particular:
- Use the original rejection rule. \(A(H)\) must include every route by which the original design could reject after the retained history, including rejection at an omitted look and subsequent rejection at the final look.
- Condition on the relevant history. The original continuation probability may depend on the observed statistic, the retained information, or an observed adaptation. Omitting information that affects the original rule can give the wrong conditional error.
- Use the conditional null distribution. The replacement cutoff must be calibrated under the null distribution of the final statistic given \(H\). A marginal distribution or an alternative distribution does not establish type I error preservation.
- Use the same null hypothesis being protected. For a simple null, calculate under that null. For a composite null, preservation must hold for every null value to guarantee the same size; controlling it only at one nuisance-parameter value may fail elsewhere.
- Check the attainable probability. A continuous statistic permits exact tail calibration in the usual case. With a discrete statistic, a nonrandomized cutoff may have conditional error below, but not exactly equal to, the target.
A useful diagnostic is to tabulate or plot \(A(H)\) over plausible retained histories and compare it with the replacement’s conditional rejection probability. Check the boundaries of the history range as well as typical values: a formula that is accurate near the center may give implausible or unattainable cutoffs in the tails. Then integrate or simulate the replacement’s rejection probability under the null and compare it with the original design’s null error. Simulation is a check on implementation and numerical integration, not a substitute for specifying the correct conditional model.
What Changes and What Does Not
Removing a look changes the operational path of the trial: no decision is made at that time, and the replacement may use a different final boundary. Conditional-error calibration does not assert that the original and replacement designs have the same power, expected sample size, or decision timing. Its guarantee concerns null rejection probability under the conditions used to calculate and preserve \(A(H)\).
| Approach | Type I error implication | Main trade-off |
|---|---|---|
| Keep the original final boundary and omit the interim | Not guaranteed to preserve the original error | Simple operationally, but the original sequence of decisions has changed |
| Calibrate the final test to \(A(H)\) | Preserves error exactly when conditional probabilities match under the null | Requires the relevant history and conditional null distribution |
| Use a final test with conditional error no greater than \(A(H)\) | Controls error at or below the original level | May be conservative and reduce power |
These approaches should not be conflated. Retaining an unadjusted final cutoff is not automatically conservative: its error depends on the original interim rule and on how the final statistic is related to the interim information. Conditional-error calibration is a targeted way to preserve the original error allowance, not a guarantee that every other design characteristic remains unchanged.
Common Mistake
Wrong reasoning: “The omitted analysis was only an interim look, so deleting it and keeping the original final boundary cannot increase type I error.” This ignores that the original design could reject at the interim and had a particular conditional chance of rejection afterward. Removing one route to rejection while leaving the remaining rule untouched does not, by itself, establish what happens to the overall error; the final statistic and the interim statistic may also be dependent.
Correction: At each retained history, calculate the original probability of eventual rejection, including all planned future looks. Then calibrate the replacement final test so its conditional null rejection probability is that same amount, or no greater than it. Verify the conditional distribution and the resulting unconditional error under the null model.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- At a retained history, an original design has conditional error 0.06. A replacement has conditional rejection probability 0.04 at every history. What can be concluded about its unconditional error relative to the original?
- A one-sided replacement statistic has conditional null distribution \(N(0.5,1.44)\), and the conditional error is 0.025. Find the final rejection boundary.
- Why is preserving an average conditional error over histories not necessarily enough to justify a proposed history-dependent replacement rule?
- Suppose a final statistic is discrete and no cutoff gives conditional rejection probability exactly 0.03. What are two ways to proceed?
Solutions. (1) At the stated history, the replacement’s conditional rejection probability 0.04 is below the original’s 0.06. This alone does not establish that its unconditional error is no greater: the original conditional error must be at least 0.04 at every history, or an unconditional comparison must be shown separately. Exact preservation is not established. (2) The standard deviation is \(\sqrt{1.44}=1.2\), and \(\Phi^{-1}(0.975)\approx1.9600\). The boundary is \(0.5+(1.2)(1.9600)=2.852\), so reject above approximately 2.852. (3) The conditional-error argument requires the replacement’s rejection probability to match or be bounded by the original conditional error at each retained history, not merely to have a matching average under a chosen history distribution. (4) Choose a nonrandomized cutoff whose rejection probability is below 0.03, giving conservative preservation, or randomize at a boundary value to attain 0.03 exactly.