Why Blinding Alone Does Not Guarantee Type I Error Control
Blinded sample-size re-estimation is often intended to respond to uncertainty in nuisance parameters, such as the outcome variance or event rate, without using the estimated treatment effect. That goal is important, but the word blinded does not itself establish that the final test retains its nominal type I error. A pooled statistic can contain information about the treatment contrast even when treatment labels are hidden. If the sample-size rule responds to that information, the final sample size can be associated with the same random variation used in the test.
The issue is not that every blinded re-estimation inflates type I error. A rule based on a nuisance estimate may be valid under suitable assumptions and a correctly specified final test. The issue is that the rule and test must be assessed together: the null distribution of the final test statistic must account for how the sample size and analysis were chosen.
For a fixed-sample test, its reference distribution is derived for a specified sample size and analysis. Once the sample size or final test depends on interim data, applying the same critical value does not automatically preserve that reference distribution. The relevant question is whether the complete adaptive procedure—not just the final calculation considered in isolation—has type I error at most \(\alpha\).
How a Blinded Summary Can Retain Contrast Information
Consider two independent treatment groups with \(n_1\) and \(n_2\) observations, group means \(\bar{x}_1\) and \(\bar{x}_2\), and within-group sums of squares \(SS_1\) and \(SS_2\). Let \(N=n_1+n_2\), and let \(s^2_{\mathrm{all}}\) be the usual sample variance calculated after pooling the observations without treatment labels. The total sum of squares around the pooled mean decomposes as
The first two terms describe within-group variation. The last term is a nonnegative contribution from the difference in group means. Thus, the pooled variance is not generally a pure measure of within-group nuisance variation: it can increase when the observed treatment contrast is large, regardless of which group has the larger mean. Hiding the labels removes the direction of the contrast, but need not remove information about its magnitude.
Worked Example: A Pooled Variance Contains a Contrast Component
Suppose a hypothetical interim data set has 10 observations in each group. Its within-group sums of squares add to 38, and the group means differ by 2 outcome units. Compute the pooled sample variance that would be obtained by ignoring group labels.
Here \(n_1=n_2=10\), \(N=20\), and \((\bar{x}_1-\bar{x}_2)^2=2^2=4\). The between-group contribution to the total sum of squares is
The total sum of squares is therefore \(38+20=58\), giving
For comparison, the pooled within-group variance is \(38/(20-2)=2.111\) outcome units squared. The difference arises from the contrast contribution. This calculation does not show that every rule using \(s^2_{\mathrm{all}}\) will inflate type I error. It shows why blinding the labels alone is not enough to establish that the re-estimation statistic is unrelated to treatment-effect information.
Under a null model with equal group means, the contrast still varies randomly from trial to trial. A blinded rule that responds to a pooled statistic containing a contrast component can therefore select sample sizes partly in response to random treatment differences. Whether that association is large enough to distort type I error, and in which direction, depends on the rule and final analysis.
A Direct Demonstration of Data-Dependent Test Size
The following simplified two-stage example isolates the dependence mechanism. It is a hypothetical, unblinded rule, not a recommended blinded design. Its purpose is to show explicitly why using a fixed critical value after a data-dependent decision need not give the nominal rejection probability. Suppose outcomes are normally distributed with known variance, allocation is balanced, and stage 1 and stage 2 observations are independent. Under \(H_0\), let \(Z_1\) and \(Z_2\) denote the standardized treatment contrasts from the two stages; then \(Z_1,Z_2\) are independent standard normal random variables.
Worked Example: A Data-Dependent Sample Size Raises the Null Rejection Probability
At the nominal two-sided level \(\alpha=0.05\), use the critical value 1.96. If \(|Z_1|\geq1.96\), stop at stage 1 and reject when \(|Z_1|\geq1.96\). Otherwise, add a second stage of the same size and test the combined data using the same critical value. Under the stated normal model, the combined standardized contrast is
The type I error probability of the entire rule is
The first term is approximately 0.05. The second term is strictly positive. For a numerical lower bound, consider \(1.5\leq Z_1<1.96\). In this interval, \(Z_2>1.272\) is sufficient for a positive final rejection, because
Under the null, the two stages are independent. Using \(\Phi(1.96)\approx0.9750\), \(\Phi(1.5)\approx0.9332\), and \(P(Z_2>1.272)\approx0.1017\), the probability of this subset of continued-trial rejections is approximately
By symmetry, an equal-probability subset occurs for \(-1.96<Z_1\leq-1.5\). These disjoint continuation events add at least \(2(0.00425)=0.00850\) to the initial 0.05 rejection probability. Thus this rule’s type I error is at least 0.0585, already above 0.05. The calculation illustrates the danger of adapting the sample size and reusing a fixed critical value without validating the combined procedure. A blinded rule does not have this exact decision structure, but a blinded statistic that tracks the magnitude of \(Z_1\) can create a related dependence that must be evaluated for that specific design.
What Must Be Checked
A defensible assessment starts with the null distribution of the full procedure. State the outcome model, allocation scheme, interim statistic, re-estimation rule, sample-size limits, and final rejection rule. Then evaluate the rejection probability under \(H_0\) while applying every step of the design. In particular, do not calculate type I error as if the selected sample size had been fixed in advance unless that conditional calibration has been justified.
A useful distinction is whether the adaptation statistic responds only to nuisance variation or also to treatment-effect information. Even when it is intended to be nuisance-only, inspect its mathematical construction. For example, a pooled summary may combine within-group variability with a term involving the squared group-mean difference. A simulation under \(H_0\) can then estimate the rejection probability of the complete rule, but the simulation must reproduce the actual interim statistic, sample-size decision, subsequent data generation, and final test. Simulation supports a claim only over the null models and parameter values it represents; it is not a proof for every possible nuisance parameter.
Record the outcome distribution, variance or other nuisance parameters, allocation, and independence assumptions used to define the test.
Determine whether it is affected by a treatment contrast, including its magnitude, rather than only by the intended nuisance parameter.
For each simulated or derived data set, make the sample-size decision and then apply the planned final analysis and rejection threshold.
Assess uncertainty in a simulation estimate and examine the null parameter settings for which control is required.
The main alternative is a re-estimation rule based on a statistic whose relationship to the treatment contrast is understood and whose resulting final test has been calibrated. Another option is to keep the original sample size and final analysis fixed, avoiding this adaptation pathway. A blinded procedure can also be compared with an explicitly unblinded adaptation, but the absence of access to treatment labels is an operational property, not a statistical proof of error control. Any alternative must be judged by its own assumptions and full-procedure operating characteristics.
Common Failure Modes and Limitations
- Equating masked labels with absence of effect information. A pooled statistic can retain the squared contrast. Check its formula, not only who can see treatment assignments.
- Checking only the sample-size estimator. A sensible nuisance estimate does not guarantee that the final test remains calibrated after the decision rule uses it.
- Using the fixed-design critical value without justification. The final statistic’s null distribution can change when the sample size or analysis is selected from interim data.
- Treating one simulation as universal evidence. A null simulation evaluates the scenarios encoded in it. It may miss other nuisance values, implementation details, or data-generating conditions.
The dependence itself does not determine the direction of the error. A particular rule may be conservative, maintain its nominal level, or inflate it; there is no general claim that blinded re-estimation must inflate type I error. The lesson is narrower and more useful: when the re-estimation statistic can carry treatment-effect information, nominal error control needs a design-specific derivation or a sufficiently careful evaluation of the complete procedure.
Check Your Understanding
Answer each question using the definitions and reasoning above. Brief solutions follow the questions.
- Why can a pooled variance calculated without treatment labels still contain information about the treatment contrast?
- In the two-stage example, why is the rejection probability greater than 0.05 even before calculating the entire continuation probability?
- Does any dependence between a blinded re-estimation statistic and the treatment contrast prove that type I error is inflated? Explain.
- What data-generating and analysis steps must a null simulation reproduce to estimate the type I error of an adaptive design?
Solutions. (1) Its total sum of squares includes a nonnegative between-group term proportional to \((\bar{x}_1-\bar{x}_2)^2\), even if group labels are hidden. (2) The rule already rejects with probability about 0.05 at stage 1, and there is additional positive probability of rejection among trials that continue. The worked calculation gives a lower bound of about 0.0585 for the total. (3) No. Dependence identifies a potential problem, but the direction and size of the error depend on the rule and final test; type I error must be established for the complete design. (4) It must generate interim data under the null, compute the actual blinded statistic, apply the sample-size rule, generate the remaining data under the stated model, and apply the planned final analysis and rejection threshold.