Worked Calibration With A Sample-Size Change

Why Recalibrate After Increasing the Sample Size?

In a two-stage trial, the interim data already contribute evidence to the final test. Increasing the number of observations collected after interim changes how much new information enters the final score. Consequently, simply keeping the original final-score cutoff need not preserve the probability of rejection conditional on the observed interim result. Conditional-error calibration gives a direct solution: retain the original design’s conditional probability of rejection, and choose the new cutoff to match it.

The calculation below is for a one-sided test in a normal model with known variance. It illustrates a fixed increase in the second-stage sample size after observing the interim score. If the increase is selected by a rule using the interim data, the cutoff is calculated for the resulting increase and the observed interim score; it is not enough to condition only on which sample-size option was selected.

Model and Original Conditional Error

Suppose observations in each stage are independent and distributed as $N(\theta,\sigma^2)$, with known $\sigma$. The one-sided hypotheses are $H_0:\theta\leq 0$ and $H_a:\theta>0$. Let $n_1$ be the first-stage sample size, $n_2$ the original second-stage sample size, and $\bar{x}_1$ and $\bar{x}_2$ the stage-specific sample means. Under the boundary null $\theta=0$, define the standardized stage scores

Z_1=\frac{\sqrt{n_1}\bar{x}_1}{\sigma}, \qquad Z_2=\frac{\sqrt{n_2}\bar{x}_2}{\sigma}.

Under the boundary null, $Z_1$ and $Z_2$ are independent standard normal random variables. The original final score is the standardized pooled score

Z_{\mathrm{old}} = \frac{\sqrt{n_1}Z_1+\sqrt{n_2}Z_2}{\sqrt{n_1+n_2}}.

The original design rejects at its final analysis when $Z_{\mathrm{old}}\geq c$, where $c=\Phi^{-1}(1-\alpha)$, $\alpha$ is the one-sided significance level, and $\Phi$ is the standard normal cumulative distribution function. Assume there is no earlier efficacy rejection in this example. At a realized interim score $Z_1=z_1$, the original conditional error is the probability, under the boundary null, that the original design would reject given that interim result.

Definition: The original conditional error at $Z_1=z_1$ is $$ A(z_1) = P_0(Z_{\mathrm{old}}\geq c\mid Z_1=z_1) = 1-\Phi\!\left( \frac{c\sqrt{n_1+n_2}-\sqrt{n_1}z_1}{\sqrt{n_2}} \right). $$ It is the original design’s conditional rejection probability, not the probability that the null hypothesis is true.

The subscript 0 indicates calculation at the boundary $\theta=0$. For the one-sided composite null, this boundary is the least-favourable value for a high-score rejection: at any $\theta<0$, the independent second-stage score has a lower mean, so its conditional chance of crossing the same upper cutoff is no larger.

Deriving the New Threshold

Now suppose the second-stage sample size is increased to $n_2^\ast>n_2$. Keep the first-stage data and their contribution, but calculate the new pooled score using all $n_2^\ast$ second-stage observations. Under the boundary null, conditional on $Z_1=z_1$, the new final score has a normal distribution:

Z_{\mathrm{new}}\mid Z_1=z_1 \sim N\!\left( \frac{\sqrt{n_1}z_1}{\sqrt{n_1+n_2^\ast}}, \frac{n_2^\ast}{n_1+n_2^\ast} \right).

Here the second argument is the variance. To preserve the original conditional error, choose a new final-score cutoff $b(z_1)$ satisfying $P_0(Z_{\mathrm{new}}\geq b(z_1)\mid Z_1=z_1)=A(z_1)$. The standardized distance from the new conditional mean to this cutoff must therefore be the standard normal quantile with upper-tail probability $A(z_1)$.

\frac{ b(z_1)-\sqrt{n_1}z_1/\sqrt{n_1+n_2^\ast} }{ \sqrt{n_2^\ast/(n_1+n_2^\ast)} } = \Phi^{-1}\!\left(1-A(z_1)\right).

Solving for $b(z_1)$ gives the calibrated threshold. The argument also shows why the original conditional error must be calculated first: it determines the quantile used with the new conditional distribution.

Calibration formula: For a finite interim score, $0<A(z_1)<1$, and the new final-score threshold is $$ b(z_1) = \frac{\sqrt{n_1}z_1}{\sqrt{n_1+n_2^\ast}} + \sqrt{\frac{n_2^\ast}{n_1+n_2^\ast}}\, \Phi^{-1}\!\left(1-A(z_1)\right). $$ Reject under the recalibrated design when $Z_{\mathrm{new}}\geq b(z_1)$.

This construction matches the conditional rejection probability at the observed interim result. It does not claim that the new and old score statistics have the same distribution, or that their cutoffs should be numerically identical.

Worked Example: Increase the Second-Stage Sample Size

Assume $n_1=50$, $n_2=50$, and an increase to $n_2^\ast=100$. Use a one-sided $\alpha=0.025$ test, so $c=\Phi^{-1}(0.975)=1.960$. Suppose the observed interim score is $z_1=1.200$. Calculate the original conditional error and the new final-score threshold.

First calculate the standardized distance from the interim result to the original final boundary, measured in second-stage score units:

\frac{c\sqrt{n_1+n_2}-\sqrt{n_1}z_1}{\sqrt{n_2}} = \frac{1.960\sqrt{100}-\sqrt{50}(1.200)}{\sqrt{50}} = 1.960\sqrt{2}-1.200 = 1.572.

Thus the original conditional error is

A(1.200)=1-\Phi(1.572)\approx 0.0580.

Under the increased sample size, the new score conditional on the interim result has mean and standard deviation

\mu_{\mathrm{new}\mid 1} = \frac{\sqrt{50}(1.200)}{\sqrt{150}} = 0.6928, \qquad SD_{\mathrm{new}\mid 1} = \sqrt{\frac{100}{150}} = 0.8165.

To give the upper tail probability $0.0580$, use the same standard normal quantile, $1.572$. The new final-score threshold is therefore

$$ b(1.200)=0.6928+(0.8165)(1.572)=1.976. $$

The recalibrated rule rejects when the new pooled score is at least $1.976$. By construction, its rejection probability conditional on this interim score is $1-\Phi(1.572)\approx0.0580$, matching the original design’s conditional error.

Worked Example: A Less Promising Interim Result

Keep $n_1=n_2=50$, $n_2^\ast=100$, and $c=1.960$, but suppose the interim score is $z_1=0$. Find the conditional error and recalibrated cutoff.

The original second-stage score would need to exceed

\frac{1.960\sqrt{100}-\sqrt{50}(0)}{\sqrt{50}} = 1.960\sqrt{2} = 2.772.

Therefore $A(0)=1-\Phi(2.772)\approx0.0028$. The new conditional mean is zero, and its standard deviation is $\sqrt{100/150}=0.8165$. The new threshold is

$$ b(0)=0+(0.8165)(2.772)=2.263. $$

This higher cutoff, relative to the new conditional mean, is needed because the original design allowed only about a $0.28\%$ conditional rejection probability at this interim result. The sample-size increase does not erase that conditional constraint.

Check the Calibration and Its Assumptions

A direct check is to standardize the calibrated cutoff under the new conditional distribution. Substitution into the formula gives

\frac{b(z_1)-\sqrt{n_1}z_1/\sqrt{n_1+n_2^\ast}} {\sqrt{n_2^\ast/(n_1+n_2^\ast)}} = \Phi^{-1}(1-A(z_1)).

Applying the standard normal upper tail to both sides recovers $A(z_1)$. This is the core justification: the cutoff is selected so the conditional rejection probability is exactly the original conditional error, subject to the model and calculation being appropriate.

The calculation relies on independent stage data, a normal outcome model with known variance, a prespecified one-sided test and original boundary, and a final score formed by the stated pooled-score formula. The sample-size change must be known when applying the new score distribution. If the trial also has an interim efficacy or futility rule, the original conditional error must reflect that rule and any continuation requirement; the final-only expression above cannot simply be used unchanged.

In practice, check that the interim score, the sample sizes, and the boundary are on the same standardized scale; confirm that the increased second-stage observations are independent of stage one; and evaluate the tail probability again after calculating the cutoff. A discrepancy usually signals a scale, sign, or variance error. In particular, the variance of the new conditional score is $n_2^\ast/(n_1+n_2^\ast)$, not its square root.

Worked Example: Why Keeping the Old Cutoff Can Fail

Return to the first example, where $z_1=1.200$, the increased-design conditional mean is $0.6928$, and its conditional standard deviation is $0.8165$. Suppose the original numerical cutoff $1.960$ is retained without calibration. Its conditional rejection probability after the increase is

P_0(Z_{\mathrm{new}}\geq1.960\mid Z_1=1.200) = 1-\Phi\!\left(\frac{1.960-0.6928}{0.8165}\right) = 1-\Phi(1.552) \approx0.0604.

This is larger than the original conditional error, approximately $0.0580$. The difference is not enormous in this particular numerical case, but it is real: reusing a numerical cutoff does not in general preserve a conditional tail probability when the conditional distribution changes.

How This Calibration Differs From Other Choices

Conditional-error calibration answers a specific question: after observing the interim data, what final threshold under the changed design retains the original design’s conditional rejection probability? It differs from choosing a threshold to obtain a desired conditional power. Conditional power concerns rejection probability under a specified alternative; conditional error concerns the null rejection probability that constrains type I error.

It also differs from applying the original cutoff to the new score, as the preceding example shows. A fresh unconditional calibration of a new design might set its overall null rejection probability to $\alpha$, but that alone does not ensure the original conditional error is respected at each interim result. Matching conditional error is the stronger, data-conditional requirement used in this construction.

The method has limits. It preserves the original conditional error under the stated model; it does not validate a misspecified variance, repair dependence between stages, or automatically account for adaptations or stopping rules omitted from the calculation. If the adaptation uses additional information, or if the interim statistic or analysis changes, the conditional error must be defined for the relevant observed information and original decision rule before recalibrating.

Common Mistake

Wrong reasoning: “The trial still uses the same one-sided significance level, so the old final cutoff remains valid after increasing the sample size.”
Correction: A significance level specifies a null rejection probability for a particular test statistic and design. The increased sample size changes the conditional distribution of the new final score. Calculate the original conditional error at the observed interim score, then solve for the new cutoff under that changed conditional distribution.

Key takeaway: Calculate $A(z_1)$ from the original design first. Then set the new threshold to the conditional mean plus the conditional standard deviation times $\Phi^{-1}(1-A(z_1))$. This matches the original null rejection probability conditional on the interim score.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

With $n_1=n_2=40$, $n_2^\ast=80$, $c=1.960$, and $z_1=0.800$, calculate the original conditional error and the new final-score threshold.
In the calibration formula, what is the variance of the new final score conditional on $Z_1=z_1$?
For the first worked example, does retaining the old cutoff of $1.960$ give a conditional rejection probability equal to the original conditional error? Explain using the calculated probabilities.
Name two assumptions that are needed for the stated calibration formula to apply as written.

Solutions. (1) The original second-stage standardized boundary is $1.960\sqrt{2}-0.800=1.972$, so $A(0.800)=1-\Phi(1.972)\approx0.0243$. The new conditional mean is $\sqrt{40/120}(0.800)=0.4619$, and the standard deviation is $\sqrt{80/120}=0.8165$. Thus $b=0.4619+(0.8165)(1.972)\approx2.072$. (2) The variance is $n_2^\ast/(n_1+n_2^\ast)$; the standard deviation is its square root. (3) No. The old cutoff gives a conditional rejection probability of about $0.0604$, rather than the original conditional error of about $0.0580$. (4) Examples include independent stage data, a normal outcome model with known variance, the stated pooled-score construction, and use of the stated one-sided original boundary. At least two are required for the calculation as written.