Why Fixed-Design Critical Values Fail After Re-Estimation

Why the Cutoff Can Stop Controlling the Error Rate

A one-sided fixed-design test is calibrated so that, under \(H_0\), its test statistic exceeds a critical value with probability \(\alpha\). Re-estimating sample size after observing interim outcomes changes the final statistic's relationship to those outcomes. Even if each possible fixed sample size would use the same nominal cutoff, the probability of rejection after the sample size has been selected can differ from \(\alpha\).

The key issue is not simply that the eventual sample size is random. It is that the sample size is chosen using data that also contribute to the final test statistic. The selection rule changes which interim outcomes are paired with which final-statistic weights. To assess calibration, calculate the null rejection probability under the entire rule: selection, data collection, and final testing.

Definition: The Type I error probability under an adaptive rule is \(P(\text{reject }H_0\mid H_0)\), where the probability includes the interim data, the sample size decision they induce, and the subsequent data. A fixed-design cutoff is calibrated for the adaptive rule only if this probability equals the target \(\alpha\) (or is no greater than \(\alpha\), for a conservative test).

Model and Rejection-Probability Calculation

Consider a one-sided test of \(H_0:\theta=0\) against \(H_a:\theta>0\). Assume independent normally distributed observations with known variance, a common mean parameter \(\theta\), and a fixed observation rate. Equivalently, use independent standardized stage statistics \(Z_1\) and \(Z_2\), each with variance 1 under \(H_0\). Under the null, \(Z_1\sim N(0,1)\), and the stage-two statistic is independent of the interim data and satisfies \(Z_2\sim N(0,1)\) for each sample size selected. These assumptions let us isolate the effect of outcome-dependent sample size; unknown variance, non-normal outcomes, or dependence between stages require their own calibration arguments.

If stage one has \(n_1\) observations and the final total is \(N\), the usual pooled normal statistic can be written as a weighted sum of the stage statistics. With equal information per observation, the weights are the square roots of the information fractions. Under a fixed design, their squares sum to 1, so the final statistic has a standard normal distribution under \(H_0\).

Z_{\mathrm{final}}= \sqrt{\frac{n_1}{N}}Z_1+ \sqrt{\frac{N-n_1}{N}}Z_2.

Suppose the rule selects a final total \(N(z)\) after observing \(Z_1=z\). Conditional on that interim value, the final statistic has mean \(\sqrt{n_1/N(z)}\,z\) and variance \((N(z)-n_1)/N(z)\) under \(H_0\). Thus the conditional probability of crossing a fixed cutoff \(c\) is

1-\Phi\left( \frac{c-\sqrt{n_1/N(z)}\,z} {\sqrt{(N(z)-n_1)/N(z)}} \right).

Integrating this conditional probability against the null density of \(Z_1\), separately over each sample-size branch, gives the overall rejection probability. Each branch is weighted by how often the interim data enter it; it is not enough to report a conditional rejection probability for one branch.

P(\text{reject }H_0\mid H_0) = \sum_j\int_{A_j}\phi(z) \left[ 1-\Phi\left( \frac{c-\sqrt{n_1/N_j}\,z} {\sqrt{(N_j-n_1)/N_j}} \right) \right]\,dz,

where \(A_j\) is the set of interim values selecting final total \(N_j\), and \(\phi\) and \(\Phi\) are the standard normal density and distribution functions. The formula assumes the stage-two statistic remains independent and standard normal under the null conditional on the chosen sample size. If the adaptation changes the test statistic, endpoint, or data-generating assumptions, this calculation must be modified accordingly.

Worked Examples

Worked Example: Confirming the Fixed-Design Calibration

Suppose \(n_1=1\), the planned total is \(N=2\), and the final test rejects when \(Z_{\mathrm{final}}>1.6449\). Under the stated null model, the two stage statistics are independent standard normal variables. The fixed-design statistic is

Z_{\mathrm{final}} =\sqrt{\frac{1}{2}}Z_1+\sqrt{\frac{1}{2}}Z_2 =0.7071Z_1+0.7071Z_2.

Its mean under \(H_0\) is 0, and its variance is \(0.7071^2+0.7071^2\approx1\). Therefore it is standard normal under the null, and

P(Z_{\mathrm{final}}>1.6449\mid H_0) =1-\Phi(1.6449)\approx0.0500.

The cutoff is calibrated for this fixed plan at a one-sided Type I error rate of about 5.00%. That fact alone does not establish calibration after a rule changes the total based on \(Z_1\).

Worked Example: Increasing the Sample Size After a Negative Interim Result

Use the same hypothetical normal model and cutoff \(c=1.6449\). Start with \(n_1=1\) and a planned total of 2. If \(Z_1\geq0\), retain that total. If \(Z_1<0\), increase the total to 11, giving 10 independent stage-two observations. The assumptions above ensure that, conditional on the interim data and selected sample size, the standardized stage-two statistic is independent and standard normal under \(H_0\).

For \(Z_1\geq0\), the statistic retains the fixed-plan weights \(0.7071\) and \(0.7071\). For \(Z_1<0\), the weights become

\sqrt{\frac{1}{11}}=0.3015,\qquad \sqrt{\frac{10}{11}}=0.9535.

The adaptive-branch contribution to the rejection probability is therefore

\int_{-\infty}^{0}\phi(z) \left[ 1-\Phi\left(\frac{1.6449-0.3015z}{0.9535}\right) \right]\,dz \approx 0.01293.

For example, at \(z=0\), the integrand is approximately \(\phi(0)[1-\Phi(1.6449/0.9535)]\approx0.01686\). At \(z=-0.5\), it is approximately \(\phi(-0.5)[1-\Phi((1.6449+0.15075)/0.9535)]\approx0.01050\). These are integrand values, not the total probability for the branch; numerical integration over all \(z<0\) gives approximately 0.01293.

The contribution from \(Z_1\geq0\), where the fixed plan is retained, is

\int_{0}^{\infty}\phi(z) \left[ 1-\Phi\left(\frac{1.6449-0.7071z}{0.7071}\right) \right]\,dz \approx 0.04875.

Adding the disjoint branch contributions gives

P(\text{reject }H_0\mid H_0) \approx0.01293+0.04875=0.06168.

The null rejection probability is about 6.17%, not 5.00%. The increase is about 1.17 percentage points, or 23.4% relative to the nominal 5% rate. The fixed-design cutoff is too permissive for this adaptive rule, even though the statistic and cutoff are familiar on each branch.

Worked Example: Adaptation Independent of the Interim Statistic

Now suppose an external, independent randomizer chooses whether the total is 2 or 11, each with probability \(1/2\). The randomizer is independent of \(Z_1\), \(Z_2\), and all trial outcomes; the same normal model and fixed cutoff apply. Conditional on either randomized plan, the correctly weighted final statistic is standard normal under \(H_0\). Consequently,

P(\text{reject }H_0\mid H_0) =\frac{1}{2}(0.0500)+\frac{1}{2}(0.0500) =0.0500.

This example shows why random sample size alone is not the explanation for the preceding inflation. The important distinction is whether the selection is associated with interim outcomes that also enter the final statistic. Independence of the decision is sufficient in this example, not a universal guarantee for every adaptive design: the final statistic must still be valid conditional on the selected plan.

What the Calculation Reveals

Under a fixed plan, the weighted sum has the standard normal null distribution because its weights are fixed and its independent components have known variances. After the interim-dependent rule, the weights are functions of \(Z_1\). The overall final statistic is then a mixture over different conditional distributions, with the mixture components selected by regions of \(Z_1\). A standard normal cutoff need not give the same tail probability for that mixture.

In the numerical example, a negative \(Z_1\) selects a larger sample and changes the first-stage weight from \(0.7071\) to \(0.3015\). A negative interim contribution is thus downweighted at the final analysis. This can make rejection more likely than it would be if the original fixed-plan weights had remained in place. Meanwhile, the \(Z_1\geq0\) branch contributes its own probability under the rule. The overall Type I error is the sum of both contributions, not the nominal error rate attached to either fixed plan considered in isolation.

Key distinction: A cutoff calibrated for a fixed design controls the error rate for that design. It does not automatically control the error rate when the design is selected using interim outcomes. Calibration must account for the selection rule and its effect on the distribution of the final test statistic.

Diagnostics, Alternatives, and Limitations

A direct diagnostic is to compute or simulate the rejection probability under the null while applying the complete adaptation algorithm. In an analytic calculation, check that the branch regions partition the possible interim outcomes, the conditional distributions match the assumed model, and the branch probabilities are integrated with the correct null density. In simulation, record the selected sample size as well as rejection, and compare the overall rejection proportion with \(\alpha\); a result close to the target does not by itself establish precision, which depends on the number of simulations.

A fixed-design cutoff may remain appropriate when the sample-size choice is independent of the test outcomes and the conditional final statistic is correctly standardized. It may also be valid under a specifically justified adaptive procedure. Neither conclusion follows merely from using a normal statistic or from preserving the same endpoint.

When the fixed cutoff fails, alternatives include recalibrating a cutoff for the complete rule, or using an adaptive testing method designed to preserve the Type I error probability under its stated assumptions. Combination tests and conditional-error methods are examples of approaches that explicitly account for adaptation. Their validity depends on their own specifications and conditions; they are not obtained by simply reusing a fixed-design critical value.

Outcome-dependent weights: if the same interim statistic affects both the sample size and the final statistic, derive the null rejection probability rather than assuming the fixed-design distribution still applies.
Incorrect conditioning: a branch-specific conditional probability must be weighted by the probability of entering that branch.
Model mismatch: normality, independence, known variance, and equal information per observation were assumed here. Missingness, estimated variance, unequal information, or changes to the endpoint can alter the calculation.
Unexamined adaptations: recalculating sample size is not the only possible change. Any outcome-dependent change to the final analysis can affect error control.

Specify the null model.
State the distribution of interim and future data, including independence and variance assumptions.

Write the rule and statistic.
For each interim-data region, identify the selected sample size, the final statistic, and the fixed cutoff.

Calculate each branch contribution.
Condition on the interim statistic, integrate the conditional rejection probability over that branch, and sum the results.

Compare with the target.
Interpret the summed probability as the Type I error rate of the complete procedure under the specified null model.

Common Mistake

Wrong reasoning: “Each planned sample size uses a standard normal statistic and the 1.6449 cutoff, so the adaptive procedure still has a 5% Type I error rate.” This ignores that the chosen sample size is associated with the interim statistic in the outcome-dependent rule.

Correction: Calculate the null rejection probability after applying the rule. The conditional statistic can be standard normal for a fixed plan, while the unconditional distribution after outcome-based selection is not standard normal. In the worked rule, the full probability is approximately 0.06168, which exceeds the nominal 0.05.

Check Your Understanding

Use the stated model and distinguish fixed-plan calibration from calibration of the full adaptive procedure.

For a fixed total of \(N=2\) with \(n_1=1\), what are the weights in the final statistic, and why does the 1.6449 cutoff have a 5% one-sided null rejection probability?
In the sample-size rule of the second worked example, what is the overall null rejection probability, and how does it compare with the target?
Why is the integrand value at \(z=-0.5\) not itself the rejection probability of the adaptive branch?
If an independent randomizer selects between two plans, each of which has a valid 5% test conditional on that plan, what is the overall rejection probability?

Solutions. (1) The weights are \(\sqrt{1/2}=0.7071\) and \(\sqrt{1/2}=0.7071\). Their squared sum is 1, and the independent standard normal stage statistics therefore give a standard normal final statistic under \(H_0\); \(1-\Phi(1.6449)\approx0.0500\). (2) The branch probabilities are approximately 0.01293 and 0.04875, so the total is approximately 0.06168, or 6.17%, above the 5.00% target. (3) It is a density-weighted conditional rejection probability at one value of the interim statistic. The branch probability requires integrating that quantity over all \(z<0\). (4) It is \(0.5(0.05)+0.5(0.05)=0.05\), provided each test is conditionally valid as specified.