Intervals After Sample-Size Re-Estimation

Why the Re-Estimation Rule Matters

A trial may use interim information to revise its planned sample size. The revision might depend only on a nuisance quantity, such as a blinded estimate of outcome variability, or it might depend on an interim treatment-effect estimate. Those rules have different implications for inference. If the decision uses information related to the effect estimate, the final estimator and its standard error can be statistically linked to the decision that determined how much additional information was collected.

A confidence interval’s coverage is the probability, over repeated trials run under the specified design, that the interval contains the true parameter. The relevant design includes the rule that maps interim data to the final sample size. A formula that would have nominal coverage at a fixed sample size does not automatically retain that coverage after an effect-dependent sample-size decision.

A Normal-Outcome Model and Its Implications

For clarity, consider a two-arm trial with a continuous outcome. In each stage, participants are randomized equally, and observations are independent normal variables with common, known variance \(\sigma^2\). Let \(\theta=\mu_T-\mu_C\) be the treatment-minus-control mean difference. If stage \(j\) enrolls \(n_j\) participants per arm, the stage treatment-effect estimate \(D_j\) has distribution \(D_j\sim N(\theta,2\sigma^2/n_j)\). Assume the second-stage sample size \(n_2\) is determined by stage 1, and that stage 2 consists of new participants whose outcomes are independent of the interim data, conditional on the selected sample size.

With equal allocation and common variance, the pooled treatment-effect estimate is \(\hat{\theta}=(n_1D_1+n_2D_2)/(n_1+n_2)\). At a fixed total sample size per arm \(n_1+n_2\), its standard error is \(\sigma\sqrt{2/(n_1+n_2)}\), and the usual normal interval uses the critical value 1.96 for a nominal 95% interval. But when \(n_2\) depends on \(D_1\), the weights in \(\hat{\theta}\) depend on the interim estimate. The standardized pooled estimate need not have a standard normal distribution across repeated adaptive trials.

At a fixed \(n_2\), this is standard normal under the stated model. If \(n_2\) is a function of \(D_1\), however, that fixed-sample argument does not establish that \(Z_{\mathrm{pool}}\) is standard normal. The distribution must account for how often each sample size is selected and for the interim estimates that lead to each choice.

When the Ordinary Interval Can Be Valid

Sample-size adaptation does not invariably invalidate a conventional interval. Under the normal model above, suppose a blinded variance estimate is independent of the treatment-effect contrast, and the re-estimation rule uses only that estimate. If the resulting sample size is therefore independent of the treatment-effect score, then, conditional on the selected sample size, the pooled estimator has its usual normal distribution and the stated standard error. Its conditional coverage is 95%; averaging that conditional coverage over the possible sample sizes also gives 95%.

This argument depends on the model and on independence, not merely on calling a rule “blinded.” The exact result above assumes known variance and a sample-size decision independent of the effect contrast. With estimated variance, a nonnormal outcome, a rule that uses unblinded effect information, or other dependencies introduced by the design, the fixed-sample formula needs a separate justification. In particular, estimated nuisance parameters can change the distribution of the standardized statistic even when the treatment effect is not examined.

Evaluating Coverage Under an Effect-Dependent Rule

For an effect-dependent rule, a direct way to assess an interval procedure is to reproduce the complete trial repeatedly under a specified true value of \(\theta\). Each simulated trial must generate stage-1 outcomes, apply the sample-size rule exactly, generate the corresponding stage-2 outcomes, calculate the interval, and record whether that interval contains the true value. If \(M\) trials are simulated and \(K\) intervals cover \(\theta\), the estimated coverage is \(\hat{C}=K/M\). Its approximate Monte Carlo standard error is \(\sqrt{\hat{C}(1-\hat{C})/M}\). Simulation estimates coverage; it does not turn an uncalibrated interval into a calibrated one.

For example, suppose a hypothetical rule enrolls 20 participants per arm in stage 1, then chooses 60 per arm in stage 2 if the interim standardized effect is between 0.5 and 2.5, and otherwise chooses 20 per arm. Under the known-variance normal model, simulate both stage contrasts from their distributions at each chosen sample size, apply this rule, and compute the pooled interval for every replicate. A hypothetical run with \(M=100{,}000\) trials and \(K=93{,}700\) covering intervals gives \(\hat{C}=93{,}700/100{,}000=0.9370\). The estimated coverage is 93.70%, not 95%. Its Monte Carlo standard error is approximately \(\sqrt{0.9370(0.0630)/100{,}000}=0.000768\), or 0.0768 percentage points. The shortfall is much larger than this simulation error. These hypothetical simulation results would be evidence that the ordinary pooled interval is poorly calibrated for that rule; a single run should still be checked for correct implementation and adequate precision.

Simulation should be repeated over parameter values of interest, especially values near the null and values at which the sample-size rule changes. Coverage can vary with \(\theta\), so a result at one value is not a general coverage guarantee. Useful diagnostics include plotting estimated coverage against \(\theta\), checking that the simulated frequencies of each final sample size match the rule, and comparing the simulated distribution of the interval’s standardized statistic with its assumed reference distribution.

Intervals That Account for the Rule

One general method is test inversion. For each candidate value \(\theta_0\), specify a test whose null distribution is calculated under \(\theta=\theta_0\) while applying the actual sample-size rule. Choose the test’s rejection threshold so its rejection probability is at most \(\alpha\) under that adaptive design. The confidence set contains the candidate values not rejected. If the tests have size \(\alpha\) for every candidate value, this inversion gives coverage at least \(1-\alpha\), up to simulation or numerical error. The resulting set may not be symmetric around the pooled estimate, and in some designs it need not be a single interval; taking its interval hull preserves coverage but can make it more conservative.

A useful computational outline is:

This approach can use both stages, unlike the stage-2-only interval, but it requires a correctly specified rule and model and can be computationally demanding. If the rule was not prespecified or its implementation is uncertain, coverage calculations are correspondingly less reliable. A procedure calibrated for one sample-size rule should not be assumed valid for a modified rule.

Common Pitfalls and Practical Limits

• Assuming the final sample size is fixed after observing it. Computing a familiar interval with the realized sample size ignores how that size was selected. Condition on the decision only when the conditional distribution justifies it; otherwise account for the selection rule.
• Treating “blinded” as a proof of independence. State which interim statistic drives the decision and why it is independent of the inference statistic under the model. Blinding by itself is not a mathematical coverage argument.
• Equating an unbiased estimate with a valid interval. Even if an estimator has expectation \(\theta\), its standardized distribution may not be the assumed normal distribution. Coverage depends on the interval’s full sampling distribution.
• Reporting simulation coverage without its uncertainty. A simulated proportion is an estimate. Report the number of replicates and its Monte Carlo precision, and check the range of effects relevant to the trial.

No interval method is uniformly best. A conventional pooled interval is simple and can be appropriate when its distributional conditions hold. An interval based on independent post-interim data can offer transparent conditional validity but may sacrifice precision. Test inversion can incorporate all stages and the exact adaptation rule, at the cost of modeling and computation. The choice should match the rule actually used, the estimand, and the desired coverage guarantee.