Why the Sample-Size Rule Matters
A fixed-sample test is calibrated for a particular sampling plan: the sample size is set without using the outcome data, and the rejection threshold is chosen to control the probability of a false rejection under the null. In an adaptive trial, the number of participants may instead depend on interim results. The final statistic then has a different relationship to the data-dependent decision rule than it has under the fixed plan. Applying the old cutoff at each possible stopping point does not, in general, preserve the old Type I error.
The key issue is not simply that the realized sample size varies. A size chosen by a random mechanism independent of the trial outcomes can preserve a test’s size if the test is calibrated at each possible size. The concern here is that the observed interim statistic helps choose the sample size or stopping time. The rule can make rejection more likely under the null than the fixed-sample calculation allows.
A Normal Model and an Interim Rule
We will use a one-sided test to isolate the issue. Suppose individual outcomes \(X_i\) are independent and normally distributed with mean \(\theta\) and known standard deviation \(\sigma\). The hypotheses are \(H_0:\theta\leq 0\) and \(H_a:\theta>0\). At the boundary \(\theta=0\), a fixed-sample upper-tail \(Z\)-test rejects when its standardized statistic exceeds \(c=1.645\), approximately the 95th percentile of the standard normal distribution. Its Type I error at the boundary is about 5%.
Consider a hypothetical trial with a planned maximum of 300 participants. An interim analysis is performed after 200 participants. If the interim statistic exceeds \(c\), the trial stops and rejects \(H_0\). Otherwise, 100 more participants are enrolled, and the trial rejects if the final statistic exceeds the same fixed-sample cutoff \(c\). Thus the sample size is 200 or 300, depending on the interim data.
Under the boundary null, let \(Z_1\) be the interim statistic and let \(W\) be the standardized contribution from the 100 additional participants. Independence of the observations implies that \(Z_1\) and \(W\) are independent standard normal random variables. The final statistic based on all 300 participants is
The weights follow from the sample sizes: the first 200 participants contribute two-thirds of the total information and the next 100 contribute one-third. Under \(\theta=0\), \(Z_1\) and \(Z_3\) each have a standard normal marginal distribution. But the rejection rule uses both statistics, and they are dependent.
Worked Example: Computing the Inflated Type I Error
Use the hypothetical 200-then-300 participant rule above, with \(c=1.645\). Under \(H_0\) at \(\theta=0\), calculate the probability of rejection when the same fixed-sample cutoff is used for both the interim and final analyses.
Rejection occurs either immediately, if \(Z_1>c\), or after continuation, if \(Z_1\leq c\) and \(Z_3>c\). These are disjoint paths, so
The immediate-rejection probability is \(1-\Phi(c)\), approximately \(0.0500\). Given \(Z_1=z\), the final statistic exceeds \(c\) exactly when
Since \(W\) is standard normal and independent of \(Z_1\), the continuation-path probability is
Therefore the overall Type I error is approximately \(0.0500+0.0241=0.0741\), or 7.41%. The fixed-sample cutoff has not controlled the probability of rejection for this adaptive rule at 5%.
Why the Calculation Differs from a Fixed-Sample Test
The distinction is clearest by comparing the rejection events. If the trial always enrolled 300 participants and tested only \(Z_3>c\), then \(Z_3\) would be standard normal under the boundary null, and the rejection probability would be about 0.05. In the adaptive rule, an interim value above \(c\) can already cause rejection. Among trials with \(Z_1\leq c\), the final statistic can still exceed \(c\); those additional rejections contribute about 0.0241. The rule therefore includes rejection opportunities that are not represented by the single fixed-sample calculation \(P(Z_3>c)\).
The conditional probability of rejection after observing an interim value \(Z_1=z\leq c\) is
It depends on \(z\), rather than being a constant 0.05. A high interim value below the stopping threshold leaves a greater chance of crossing the final boundary than a low interim value does. This is why the unconditional null probability must account for the distribution of interim data and the decision rule together.
Worked Example: Conditional Error at a Specific Interim Result
Suppose the hypothetical trial has known \(\sigma=10\) outcome units, and after 200 participants the observed sample mean is \(0.7071\) outcome units. The interim statistic is
The interim value is below 1.645, so the rule continues. Given \(Z_1=1.000\), the final test rejects if the additional-stage statistic satisfies
Thus the conditional probability of final rejection under the boundary null, for this interim result, is \(1-\Phi(1.435)\approx 0.0756\), or 7.56%. This is a conditional probability under the null model, not the probability that the null is true. It also does not by itself give the overall Type I error; the overall probability averages conditional rejection chances over all possible interim values and includes the early-rejection path.
Justification and Conditions
The integral calculation relies on a specific model and a specified rule. The observations are independent normal variables with a common mean and known standard deviation; the interim and additional-stage observations are disjoint; the interim occurs after 200 of the 300 maximum observations; and the rule uses the same upper-tail cutoff at both analyses. Under these assumptions, \(Z_1\) and \(W\) are independent standard normal variables at \(\theta=0\). Those facts justify conditioning on \(Z_1=z\) and using the standard normal tail probability for \(W\).
For the composite null \(\theta\leq0\), the boundary \(\theta=0\) is the least favorable case for this upper-tail rule. For any \(\theta<0\), both stage statistics have smaller means than at \(\theta=0\), while their variances and the decision boundaries are unchanged. The rejection events are increasing in the observed outcomes, so shifting the outcome distribution downward cannot increase their probability. Hence the maximum Type I error over \(H_0\) occurs at the boundary in this model.
More generally, a data-dependent sample size is not automatically invalid. A procedure must be assessed as a whole: its adaptation rule and its final rejection rule jointly determine the null rejection probability. The earlier material on conditional Type I error provides a way to frame this requirement: after each interim outcome, the remaining rejection chance must fit within the error probability available for that outcome. Simply reusing a fixed-sample cutoff does not establish that condition.
Write down when interim data are examined, how those data determine the sample size, and what outcomes lead to rejection at each stage.
Identify the distribution of the interim data and any future data under the null, including their dependence or independence.
Calculate the probability of rejection along each mutually exclusive path and add those probabilities, or condition on interim data and average over its null distribution.
Check that the supremum of the rejection probability over the null parameter space is no greater than \(\alpha\); do not infer control just because each cutoff came from a fixed-sample test.
When Fixed-Sample Calibration Does and Does Not Carry Over
| Sampling plan | What determines the sample size? | Implication for a fixed-sample cutoff |
|---|---|---|
| One fixed sample size | Set before observing outcomes | Valid if the test model and assumptions hold and the cutoff was calibrated for that size. |
| Random sample size independent of outcomes | An external random mechanism | Can preserve Type I error when each possible size uses a valid test at level \(\alpha\); averaging those rejection probabilities remains at most \(\alpha\). |
| Data-dependent stopping or expansion | Interim outcomes help select the size | Unadjusted fixed-sample cutoffs need not control Type I error; evaluate the joint adaptive rule. |
For independent randomization, the justification is direct. If the sample size \(N\) is independent of the outcomes and, for every allowed \(n\), the test has null rejection probability at most \(\alpha\), then
Worked Example: Independent Randomization of Sample Size
Before any outcomes are observed, suppose a hypothetical trial randomly assigns a 100-participant or 200-participant sample size, each with probability 0.5. At whichever size is selected, it conducts a valid one-sided level-0.05 \(Z\)-test using known variance. The randomization is independent of all participant outcomes.
Under the boundary null, the rejection probability at each sample size is 0.05. Therefore
Here the realized sample size varies, but it is not selected using interim evidence. The weighted-average calculation explains why the same conclusion does not follow for a sample size chosen in response to observed outcomes.
Diagnostics and Common Mistakes
A useful diagnostic is to draw the decision rule as a set of rejection paths and calculate the null probability of their union. If a trial can reject at an interim analysis and again after continuation, both routes must be included. If it changes the final sample size based on an interim statistic, the conditional distribution of the final statistic given that interim result must be considered. A calculation based only on the final statistic’s unconditional marginal distribution can miss the way the rule selected which trials continued.
- Mistake: “The final statistic is standard normal, so the test is level 0.05.” The final statistic may be standard normal under a fixed maximum sample size, but the adaptive rule may also reject early. Calculate the probability of the full rule’s rejection event.
- Mistake: “Each analysis uses a 5% cutoff, so the overall error is 5%.” The chances of crossing at different analyses are dependent, and the rule can use interim evidence to determine which analyses occur. Their rejection probabilities cannot simply be treated as one fixed-sample test.
- Mistake: “Any variable sample size invalidates inference.” Variation from outcome-independent randomization can preserve error control under the stated conditions. It is the combination of data-dependent selection and an unverified rejection rule that creates the concern.
- Mistake: “A non-significant interim result makes later testing safe.” A result below the cutoff can still be high enough to make a later rejection relatively likely. In the worked calculation, \(Z_1=1.000\) gives a conditional final rejection probability of about 7.56%.
The normal calculation is illustrative, not a universal inflation factor. The amount of inflation depends on the interim timing, the adaptation rule, the number of analyses, the test statistic, and the null model. For a different design, error control must be justified for that design rather than inferred from this example. Possible remedies include a rejection rule calibrated to the adaptive design or a conditional-error-based rule that limits the remaining rejection probability appropriately.
Check Your Understanding
Answer each question, then check the brief solutions below.
- In the 200-then-300 participant example, why is the event \(Z_1>c\) separate from the final rejection event included in the integral?
- For the rule in the worked example, what is the conditional final rejection probability when \(Z_1=0\), expressed as a normal tail probability?
- A trial chooses its sample size with an independent coin toss before collecting outcomes. Each possible sample size has a valid level-0.05 test. What upper bound does this give for the overall Type I error?
- Explain why observing \(Z_1=1.000\) does not mean there is a 7.56% probability that \(H_0\) is true.
Solutions. (1) The rule stops and rejects immediately when \(Z_1>c\); the integral covers only continuation outcomes \(Z_1\leq c\), so the paths are disjoint. (2) It is \(1-\Phi(\sqrt{3}c)\), because \(Z_1=0\) makes the required value of \(W\) equal to \(\sqrt{3}c\). (3) At most 0.05, by averaging the conditional error probabilities over the independent sample-size randomization. (4) The 7.56% is the probability of eventual rejection under the null conditional on that interim statistic and the stated model; it is not a probability assigned to the truth of the null hypothesis.