Type I Error Under Data-Dependent Sample Size Changes

Why Changing the Sample Size Can Change Type I Error

A fixed-sample test is calibrated for a particular sampling plan: its test statistic and rejection threshold together give a specified probability of rejection when \(H_0\) is true. If the final sample size is chosen using interim evidence about the treatment effect, the data influence both the final statistic and the amount of information in it. The fixed-sample calculation may then no longer describe the probability of a false rejection.

The key distinction is what drives the adaptation. An interim estimate of the treatment effect is directly related to the test statistic. A blinded estimate of a nuisance parameter, such as a pooled standard deviation, does not reveal the treatment contrast. Under suitable assumptions, the latter can allow sample-size adjustment without the same effect-driven selection, but it is not automatically valid in every design.

Definition: The Type I error rate of a testing procedure is its probability of rejecting \(H_0\) when \(H_0\) is true. For an adaptive design, this probability is calculated over all possible interim data, sample-size decisions, and subsequent observations under the null model. It is not enough to check the rejection probability for each possible final sample size as if that size had been selected in advance.

A Two-Stage Model Under the Null

Consider a one-sided test of a treatment effect, with higher responses favoring the treatment. Assume independent, normally distributed observations, a known common variance, balanced treatment allocation, and no effect under \(H_0\). Express the information in a stage as the inverse variance of the estimated treatment contrast. Let \(I_1\) be the information at the interim analysis and \(I_2\) the information added after the interim.

Under these assumptions, the standardized interim contrast \(Z_1\) and the standardized stage-two contrast \(Z_2\) are independent standard normal random variables under \(H_0\). If the final information is fixed in advance, the pooled standardized statistic is

Z_{\mathrm{final}}= \frac{\sqrt{I_1}Z_1+\sqrt{I_2}Z_2}{\sqrt{I_1+I_2}}.

For fixed \(I_1\) and \(I_2\), the numerator is normal with mean zero and variance \(I_1+I_2\), so \(Z_{\mathrm{final}}\sim N(0,1)\). This is the basis for using the usual one-sided critical value, such as 1.96 for \(\alpha=0.025\). But if \(I_2\) is chosen from \(Z_1\), the expression has a data-dependent weight on \(Z_1\). The fixed-information argument no longer applies directly.

Model condition: The independent-increments representation above relies on independent stage data and a correctly specified normal model with known variance. It is a useful reference model for seeing the selection issue. With estimated variance, dependence between stages, or a different test statistic, the null distribution and calibration need to be assessed for that specific design.

Worked Example: What Conditioning on the Interim Result Shows

Suppose the stage-two information is fixed to equal the interim information, \(I_2=I_1\). Under \(H_0\), the pooled statistic is \(Z_{\mathrm{final}}=(Z_1+Z_2)/\sqrt{2}\). If the observed interim statistic is \(Z_1=1.50\), then, conditional on this interim result, \(Z_{\mathrm{final}}\) has mean \(1.50/\sqrt{2}=1.061\) and variance \(1/2\).

Z_{\mathrm{final}}\mid Z_1=1.50 \sim N\left(1.061,\frac{1}{2}\right).

At a one-sided critical value of 1.96, the conditional probability of rejection is

P(Z_{\mathrm{final}}>1.96\mid Z_1=1.50) =1-\Phi\left(\frac{1.96-1.061}{\sqrt{1/2}}\right) \approx 1-\Phi(1.272) \approx 0.102.

This conditional probability is not the test’s overall Type I error rate. With a fixed sample size, averaging over all possible values of \(Z_1\) and \(Z_2\) still gives 0.025. The calculation illustrates why selecting a sample size based on \(Z_1\) matters: it changes which interim results are associated with which final-statistic distributions.

A Sample-Size Rule That Raises the Rejection Probability

Here is a deliberately simple adaptive rule that makes the selection effect explicit. Use a one-sided nominal level of 0.025, with critical value 1.96. If \(Z_1>0\), stop at the interim and test using \(Z_1\). If \(Z_1\leq 0\), add a second stage with \(I_2=I_1\), then test using \(Z_{\mathrm{final}}=(Z_1+Z_2)/\sqrt{2}\) and the same critical value. Assume the stage-two data are independent of the interim data, as in the model above.

Worked Example: Computing the Adaptive Type I Error

Under \(H_0\), rejection can occur in two disjoint ways. When \(Z_1>0\), rejection requires \(Z_1>1.96\), which has probability 0.025. When \(Z_1\leq 0\), rejection requires the two-stage statistic to exceed 1.96. Therefore,

\begin{aligned} P(\text{reject}\mid H_0) &=P(Z_1>1.96)\\ &\quad+P\left(Z_1\leq 0,\, \frac{Z_1+Z_2}{\sqrt{2}}>1.96\right). \end{aligned}

For the second probability, set \(y=-Z_1\), so \(y\geq 0\). Given \(Z_1=-y\), rejection requires \(Z_2>1.96\sqrt{2}+y\). Independence and the standard normal model give

P(\text{reject}\mid H_0) =0.025+\int_0^\infty \phi(y)\left[1-\Phi(1.96\sqrt{2}+y)\right]\,dy \approx 0.02531.

The integral is positive: some interim results at or below zero, which would not reject at the first stage, can be followed by sufficiently large stage-two results to cross the final threshold. Numerical integration gives an added probability of about 0.00031, so the overall Type I error is about 0.02531, rather than 0.025. The increase is small in this example, but the point is structural: the ordinary fixed-sample critical value does not, by itself, guarantee the target error rate after an effect-driven sample-size rule.

This rule is intentionally stark, not a recommendation for design. In practice, re-estimation may use a promising zone or a conditional-power calculation, and the sample size may increase only for particular interim results. Whether the resulting Type I error is above or below the target depends on the exact rule and the final test. It must be evaluated for the full procedure; adaptation does not imply that inflation is inevitable, nor that it is negligible.

Why Blinded Nuisance-Driven Re-Estimation Is Different

A blinded re-estimation rule uses information such as a pooled variance estimate without disclosing or using the interim treatment contrast. This is different from increasing sample size because the observed treatment effect looks favorable or promising. Under the null normal model, the treatment contrast and a pooled within-group variance estimate have useful independence properties. Those properties help explain why variance-based adjustment can preserve Type I error under appropriate procedures.

However, “blinded” is not a universal proof of validity. The exact result depends on how the nuisance parameter is estimated, how the sample size is selected, whether interim observations are reused, and how the final statistic and critical value are defined. For example, departures from the assumed model or a variance estimator that indirectly contains treatment information can undermine a simple argument. Blinding is a design feature that reduces a particular source of selection; the operating characteristics still need to be checked.

Worked Example: A Blinded Variance-Based Adjustment

Suppose a hypothetical balanced trial plans 80% power for a two-sided test at \(\alpha=0.05\), targeting a difference of 0.40 response units. Under a normal equal-variance planning approximation, the required number per group is

n \approx \frac{2(z_{0.975}+z_{0.80})^2\sigma^2}{\delta^2}.

Using \(z_{0.975}=1.96\), \(z_{0.80}=0.8416\), planned standard deviation \(\sigma=1.00\) response unit, and target difference \(\delta=0.40\) response units gives

n \approx \frac{2(1.96+0.8416)^2(1.00)^2}{(0.40)^2} =\frac{15.70}{0.16} \approx 98.1.

The design therefore plans about 99 participants per group. At an interim review, suppose a blinded pooled estimate of the standard deviation is 1.10 response units. Keeping the target difference and power unchanged, the sample-size calculation scales with the variance:

n_{\mathrm{new}} \approx 98.1\left(\frac{1.10}{1.00}\right)^2 =98.1(1.21) \approx 118.7.

A rule could therefore increase the target to about 119 participants per group, subject to any prespecified maximum. This calculation uses the estimated variability, not the direction or magnitude of the treatment contrast. It illustrates the nuisance-driven distinction; it does not, on its own, establish exact Type I error control for every way of implementing the adjustment.

Checking the Calibration and Choosing an Approach

The essential diagnostic is the null rejection probability of the complete design, including its adaptation rule and final analysis. In the normal model, it can sometimes be calculated by integrating over interim data, as in the example above. More complicated rules can be evaluated by simulation under \(H_0\): generate interim data, apply the actual sample-size rule, generate the added data, and apply the planned final test. The simulated rejection proportion estimates the Type I error; its Monte Carlo uncertainty should be reported when judging whether it differs meaningfully from the target.

Warning signs include applying a fixed-sample critical value without checking the adaptive procedure, using a rule based on an unblinded interim effect while claiming that the original test remains automatically valid, and evaluating only a few possible sample sizes without reproducing how each size was selected. A rule being written down in advance is important for reproducibility, but pre-specification alone does not make a nominal fixed-sample test valid.

The main alternatives address different goals. A fixed-sample design avoids adaptation and uses its planned information throughout. A prespecified group sequential design permits interim decisions but calibrates its rejection boundaries for the planned looks and decision rules. Blinded nuisance-driven re-estimation can respond to uncertain variance or other nuisance parameters without selecting directly on the observed treatment effect, subject to design-specific validity checks. For effect-driven re-estimation, use an inferential procedure and calibration designed for that adaptation rather than assuming the ordinary final test retains its nominal level.

Key takeaway: Under a fixed sample size, the null distribution of the final statistic supports the chosen critical value. When interim treatment-effect data determine the final information, that distribution can change. Calculate or simulate the Type I error of the entire rule-and-test procedure, and distinguish that case from blinded nuisance-driven adjustment.

Common Mistake

Wrong reasoning: “For every possible final sample size, a level-0.025 test has a 0.025 Type I error rate, so choosing among those sample sizes at interim still gives 0.025.”

Correction: That statement treats each sample size as if it had been fixed before observing \(Z_1\). Under an adaptive rule, the probability of using a particular sample size depends on the interim result, which also contributes to the final statistic. The overall rejection probability must include that selection mechanism. Likewise, a blinded nuisance estimate may avoid direct selection on the treatment contrast, but its validity rests on the model and implementation, not on the word “blinded” alone.

Check Your Understanding

Answer each question using the two-stage model and distinctions developed above. Solutions follow each prompt.

Under \(H_0\), with fixed \(I_1\) and \(I_2\), why does the pooled statistic have a standard normal distribution?
Solution: The independent stage statistics have mean zero and variance one. The numerator \(\sqrt{I_1}Z_1+\sqrt{I_2}Z_2\) is therefore normal with mean zero and variance \(I_1+I_2\). Dividing by \(\sqrt{I_1+I_2}\) gives a standard normal statistic.
Why does the calculation in the first worked example not show that the fixed-sample test has Type I error 0.102?
Solution: The value 0.102 is conditional on the particular interim result \(Z_1=1.50\). The fixed-sample Type I error averages the rejection probability over all interim and stage-two results under \(H_0\); that unconditional probability is 0.025.
In the sample-size rule example, what creates the additional rejection probability beyond 0.025?
Solution: When \(Z_1\leq 0\), the design adds a stage and can reject if \(Z_2\) is large enough. The probability of this event is the positive integral shown in the example, about 0.00031.
A blinded interim review estimates the pooled standard deviation but does not examine the treatment contrast. Is Type I error control guaranteed solely by this blinding?
Solution: No. Blinding avoids direct selection on the observed treatment effect, but validity still depends on the model, the variance estimator, how interim data are used, the sample-size rule, and the final test. The complete design needs appropriate justification or operating-characteristic evaluation.
Why is a prespecified effect-driven sample-size rule not necessarily enough to justify the ordinary fixed-sample critical value?
Solution: Pre-specification makes the rule explicit but does not remove the statistical dependence between the interim effect estimate, the selected information, and the final statistic. The null rejection probability must still be calculated for the combined rule and test.