Worked Example: Conditional-Error Recalibration

Why Recalibrate at the Interim?

After an interim analysis, a trial may increase its planned sample size. The final test then combines the interim evidence with a second-stage statistic whose weight may differ from the weight in the original design. Applying the original fixed-design critical value to this changed statistic need not preserve the original Type I error. Conditional-error recalibration gives a direct way to specify how much rejection probability remains available after each possible interim result.

The key quantity is not a new significance level chosen after seeing the data. It is the rejection probability that the original test would have had, conditional on the observed interim result. A revised test is calibrated so that, under the null, its conditional rejection probability does not exceed that amount.

Definition: The conditional error at interim history $X_1=x$ is the probability, under the null hypothesis, that the original design would reject at the end, conditional on that interim history: $$ A(x)=P_0(\text{original design rejects}\mid X_1=x). $$ A recalibrated continuation has conditional error control if, for every possible interim history $x$, its conditional probability of rejection under the null is no greater than $A(x)$.

Model and Original Conditional Error

Consider a one-sided test of the simple null $H_0:\theta=0$ against positive effects. Assume a normally distributed score or estimator with known variance, independent increments across the two stages, and a fixed interim information $I_1$. Under $H_0$, let the stagewise standardized statistics be independent, with $Z_1\sim N(0,1)$ and $Z_2\sim N(0,1)$. The original total information is $I=I_1+I_2$, and its final statistic is

T_{\mathrm{old}}=\sqrt{\frac{I_1}{I}}Z_1+\sqrt{\frac{I_2}{I}}Z_2.

The weights are the square roots of the information fractions, so $T_{\mathrm{old}}\sim N(0,1)$ under the null. For a one-sided test with significance level $\alpha$, the original rule rejects when $T_{\mathrm{old}}\ge c$, where $c=z_{1-\alpha}$ is the standard normal quantile with upper-tail probability $\alpha$.

Write $w_1=\sqrt{I_1/I}$ and $w_2=\sqrt{I_2/I}$. Once $Z_1=x$ is observed, rejection requires $Z_2\ge(c-w_1x)/w_2$. Since the second-stage statistic is independent standard normal under the null, the original conditional error is

A(x)=1-\Phi\left(\frac{c-w_1x}{w_2}\right),

where $\Phi$ is the standard normal cumulative distribution function. A favorable interim result generally leaves more conditional error available; an unfavorable result leaves less. This does not change the originally specified overall $\alpha$: averaging $A(Z_1)$ over the null distribution of $Z_1$ gives the original rejection probability $\alpha$.

Worked Example: Computing the Original Conditional Error

Suppose a hypothetical one-sided design has $I_1=40$, $I_2=60$, and $\alpha=0.025$. At the interim, the observed statistic is $Z_1=1.20$. Assume the stated independent standard normal increments and use $c=z_{0.975}=1.960$.

The original weights are $$ w_1=\sqrt{\frac{40}{100}}=0.6325,\qquad w_2=\sqrt{\frac{60}{100}}=0.7746. $$ Conditional on $Z_1=1.20$, the original test rejects if $$ Z_2\ge\frac{1.960-(0.6325)(1.20)}{0.7746} =\frac{1.2010}{0.7746} =1.550. $$

Therefore, $$ A(1.20)=1-\Phi(1.550)\approx 0.0606. $$ The conditional error is about 6.06%. It exceeds 2.5% because it is conditional on a favorable interim statistic, not the unconditional Type I error of the design.

Revising the Final Boundary

Suppose the first-stage information remains $I_1$, but the planned total information increases to $I_{\mathrm{new}}>I_1$. Let $Z_{2,\mathrm{new}}$ be the standardized statistic from the enlarged second stage. Under the stated null model, it remains standard normal and independent of $Z_1$. Define $$ a_{\mathrm{new}}=\sqrt{\frac{I_1}{I_{\mathrm{new}}}},\qquad b_{\mathrm{new}}=\sqrt{1-\frac{I_1}{I_{\mathrm{new}}}}, $$ so the revised combined statistic is $T_{\mathrm{new}}=a_{\mathrm{new}}Z_1+b_{\mathrm{new}}Z_{2,\mathrm{new}}$.

To make the revised conditional rejection probability equal to the original conditional error $A(x)$, use the revised boundary

c_{\mathrm{new}}(x) =a_{\mathrm{new}}x+b_{\mathrm{new}}z_{1-A(x)}.

Indeed, conditional on $Z_1=x$, the revised test rejects precisely when $Z_{2,\mathrm{new}}\ge z_{1-A(x)}$. Its null rejection probability given that interim result is therefore $A(x)$. If the new procedure chooses a more conservative continuation, it may instead use a stage-two cutoff at least as large as $z_{1-A(x)}$.

Justification: Suppose the recalibrated conditional rejection probability is no greater than $A(x)$ for every interim result $x$. Taking expectations under the null gives $$ P_0(\text{revised design rejects}) =E_0\{P_0(\text{reject}\mid Z_1)\} \le E_0\{A(Z_1)\} =P_0(\text{original design rejects}) \le\alpha. $$ Thus conditional control at every interim result implies overall Type I error control. Exact matching gives equality with the original design’s rejection probability.

Worked Example: Increasing Information and Recalibrating the Boundary

Continue the hypothetical design with $I_1=40$, observed $Z_1=1.20$, and original conditional error $A(1.20)\approx0.0606$. Increase the total planned information from 100 to 160, so $I_{2,\mathrm{new}}=120$. The revised weights are $$ a_{\mathrm{new}}=\sqrt{\frac{40}{160}}=0.5000,\qquad b_{\mathrm{new}}=\sqrt{\frac{120}{160}}=0.8660. $$

The stage-two critical value that preserves the original conditional error is $$ z_{1-A(1.20)}=z_{0.9394}\approx1.550. $$ Consequently, the revised boundary on the combined statistic is $$ c_{\mathrm{new}}(1.20) =(0.5000)(1.20)+(0.8660)(1.550) =0.6000+1.342 \approx1.942. $$ The revised test rejects if $T_{\mathrm{new}}\ge1.942$, equivalently if $Z_{2,\mathrm{new}}\ge1.550$. Under the null, that stage-two condition has probability about 0.0606 given $Z_1=1.20$, matching the original conditional error.

For illustration, if the observed enlarged-stage statistic were $Z_{2,\mathrm{new}}=1.60$, then $T_{\mathrm{new}}=(0.5000)(1.20)+(0.8660)(1.60)=1.986$. Since $1.986>1.942$, this recalibrated test rejects. This is a hypothetical calculation, not evidence from an actual trial.

Different Interim Results, Different Remaining Error

The recalibration must be defined for every possible interim result, not just the one observed. The original conditional-error function supplies that rule: each interim statistic determines the error allowance and therefore the final critical value. The next calculation shows how an unfavorable interim result leads to a stricter continuation boundary.

Worked Example: A Less Favorable Interim Result

Use the same original design, with $I_1=40$, $I_2=60$, and $c=1.960$, but suppose the observed interim statistic is $Z_1=0.50$. The original stage-two cutoff is $$ \frac{1.960-(0.6325)(0.50)}{0.7746} =\frac{1.644}{0.7746} \approx2.122. $$ Thus $A(0.50)=1-\Phi(2.122)\approx0.0169$, which is smaller than the unconditional $\alpha=0.025$.

If total information is again increased to 160, the revised weights are 0.5000 and 0.8660. The conditional-error-preserving stage-two cutoff remains $z_{1-0.0169}\approx2.122$, while the combined-statistic boundary becomes $$ c_{\mathrm{new}}(0.50) =(0.5000)(0.50)+(0.8660)(2.122) =0.2500+1.838 \approx2.088. $$ The revised test rejects only if $Z_{2,\mathrm{new}}\ge2.122$, giving conditional null rejection probability about 1.69% at this interim result.

The stage-two cutoff is determined by the conditional error, while the numerical cutoff on the combined statistic also reflects the revised weights and observed interim value. These are two expressions of the same calibrated rule.

Assumptions, Diagnostics, and Alternatives

The calculations above rely on a correctly specified null distribution, independent stagewise increments, and a valid standard normal approximation for the stage-two statistic. The interim information and statistic must be defined consistently with the original design. The adaptation rule must also be specified as a function of interim history so the conditional bound applies across all possible results. For a composite null, conditional error must be controlled over the null parameter space, or evaluated using an appropriate least-favorable null; calibration at a single null value is not automatically sufficient.

Before using the recalibrated boundary, check that the new second-stage statistic has the assumed conditional distribution under the null. If its variance, degrees of freedom, or distribution changes with the sample-size decision, the standard normal quantile in the formula may be inappropriate. Discrete tests, nuisance-parameter estimates, dependence between stages, or a changed test statistic can require a different conditional distribution or a conservative bound. These issues are reasons to revisit the model and the conditional calculation, not to apply the normal formula mechanically.

A useful diagnostic is to compare, across a range of interim statistics, the conditional rejection probability of the proposed continuation with the original function $A(x)$. It should never exceed $A(x)$ under the relevant null distribution. Also check that the revised combined-statistic boundary and the implied stage-two cutoff agree algebraically. A mismatch often indicates that the wrong weight, tail, or conditional distribution was used.

Conditional-error recalibration is one way to preserve the original test’s remaining error allowance. The earlier material on inverse-normal re-estimation expresses a related calibration through a combination statistic; the convenient representation depends on the planned adaptation and the available stagewise statistics. Retaining the original fixed-design cutoff is simpler, but generally does not preserve the original conditional error when weights or the statistic change. In all cases, recalibration controls error, not power: a larger sample may improve power, but the realized interim result and the revised rule determine whether the trial ultimately rejects.

Common Pitfalls

Using $\alpha$ as the conditional error. The remaining error is $A(x)$, which depends on the interim result. In the first example it is about 0.0606, not 0.025.
Applying the old combined-statistic cutoff with new weights. The original critical value is calibrated to the original statistic’s null distribution and does not in general preserve conditional rejection probabilities after reweighting.
Calibrating only the observed branch. A valid adaptive design needs a rule for every possible interim history; controlling one realized branch alone does not establish overall Type I error control.
Using the wrong tail or conditional distribution. For an upper-tailed test the cutoff is $z_{1-A(x)}$. A different test direction or a non-normal stage-two distribution requires the corresponding correct quantile.

Key takeaway: Compute the original conditional error at each interim result, then choose the revised final boundary so the new conditional null rejection probability is no larger. Averaging these conditional bounds establishes overall Type I error control.

Check Your Understanding

Answer each question, then compare with the solutions below.

In the first worked example, why is a conditional error of about 0.0606 compatible with an original overall significance level of 0.025?
For a revised test, what stage-two upper-tail probability should the cutoff $z_{1-A(x)}$ have under the null?
Suppose a proposed adaptation has conditional rejection probability 0.03 when the original conditional error at that interim result is 0.02. Is conditional-error control satisfied at that result?
Why is it not enough to calculate a revised boundary only for the interim result that happened to occur?

Solutions. (1) The 0.0606 is conditional on the favorable observed $Z_1=1.20$; averaging the original conditional error over all null interim results gives the overall rejection probability 0.025. (2) Its upper-tail probability must equal $A(x)$, since $P(Z_{2,\mathrm{new}}\ge z_{1-A(x)})=A(x)$ for a standard normal stage-two statistic. (3) No. The proposed conditional rejection probability, 0.03, exceeds the allowance 0.02. (4) Overall control follows by averaging conditional bounds across interim results, so the rule must satisfy the bound over all possible histories, not just the realized one.