Why Naive Pooled Tests Fail After Re-Estimation

Why pooling is not automatically safe

Pooling observations is usually an efficient way to estimate a common mean. In a fixed-design test with a known variance, the pooled mean has a familiar standard error, and the corresponding z-statistic has a standard normal distribution under the null hypothesis. But after an interim analysis, the total sample size may depend on the first-stage data. The final sample size is then part of the selection rule, not a fixed feature of the design. Applying a fixed-design calibration to the pooled observations can therefore fail to control the Type I error rate.

The issue is not that the arithmetic of pooling becomes wrong. The pooled statistic can still be calculated correctly. The issue is that its null distribution may no longer be the distribution used to choose the critical value. This distinction matters in a re-estimation design: correct arithmetic does not by itself provide a valid test.

The pooled statistic and its calibration

If the total sample size \(N=n_1+n_2\) is fixed in advance, the pooled mean is \((n_1\bar{x}_1+n_2\bar{x}_2)/N\). Its standardized statistic can be written either in terms of the stage means or in terms of stagewise z-statistics:

When the sample sizes are fixed, \(Z_1\) and \(Z_2\) are independent standard normal variables under \(H_0\). The squared weights sum to one, so \(Z_{\mathrm{pool}}\sim N(0,1)\). A one-sided test that rejects for \(Z_{\mathrm{pool}}>c\), with \(c\) chosen so that \(P(Z>c)=\alpha\) for \(Z\sim N(0,1)\), consequently has Type I error rate \(\alpha\).

The key change under re-estimation is visible by conditioning on the interim data. Write \(N=n_1+n_2(Z_1)\) when the second-stage size is a function of the first-stage result. Conditional on \(Z_1\), the second-stage statistic is still standard normal under \(H_0\), provided the model assumptions hold. The pooled statistic then has conditional mean and variance

Both quantities depend on the interim result through \(n_2\), and the conditional mean also depends directly on \(Z_1\). Thus the final statistic is not generally a standard normal variable conditional on the information that determined the sample size. Its unconditional null distribution is a mixture over the interim outcomes and the corresponding sample sizes. That mixture is not guaranteed to be standard normal.

A two-stage example with inflated Type I error

The following deliberately simple hypothetical design gives a direct demonstration. It uses a one-sided test at nominal level \(\alpha=0.025\), with known \(\sigma=1\), and sets \(\mu_0=0\). There is one observation in stage one, \(X_1\sim N(\mu,1)\), so \(Z_1=X_1\) under the null. Let \(c=1.96\), the usual approximate upper-tail standard normal critical value for a one-sided 0.025 test.

Use this interim rule: if \(Z_1>c\), stop and reject \(H_0\) using the single-stage test. Otherwise, collect one independent stage-two observation \(X_2\), and reject if the ordinary pooled statistic for two observations exceeds the same fixed-design critical value \(c\). This is a sample-size decision based on interim data: the total sample size is one or two. On the two-observation branch, “pooling” means using \((X_1+X_2)/\sqrt{2}\) as the z-statistic under \(H_0\).

This example uses early stopping to make the calculation transparent. The same structural concern arises whenever \(n_2\) is selected from interim results: the weights and the distribution of the final pooled statistic are then linked to those results. The example does not establish that every adaptation inflates Type I error by the same amount, or that every sample-size rule must inflate it. It establishes that nominal fixed-design calibration alone does not guarantee the desired error rate after adaptation.

What the interim conditioning reveals

How to analyze the design responsibly

The pooled estimate can remain useful as an estimate of the common mean. What cannot be assumed is that the usual fixed-design p-value or cutoff remains valid just because the observations have been combined correctly. A valid inferential procedure must preserve the Type I error rate under the actual adaptation rule, including its thresholds, possible stopping decisions, and second-stage sample sizes.

The earlier material on conditional error and combination tests provides approaches designed for adaptive testing. Their central practical distinction from naive pooling is that the rejection rule is calibrated to the adaptive design, rather than borrowed unchanged from a fixed-total-size experiment. For a re-estimation plan, the analysis should be specified alongside the adaptation rule, not chosen afterward based on which calculation is most favorable.

Useful checks include verifying that the stage-two size is determined only by the allowed interim information, confirming independence of future stage-two data from that information under the model, and checking that the final rejection rule is calibrated over the full range of interim outcomes. If the variance is estimated rather than known, or if adaptation changes the statistic or its distribution, the known-variance normal argument above is not sufficient. Those departures require a procedure justified for the relevant model and design.

Common mistake and limitations

• Mistake: “The pooled z-statistic has a standard normal distribution for any sample size, so the usual fixed-design critical value remains valid.” Correction: The standard normal result follows for fixed stage sizes under the stated model. When the second-stage size depends on stage-one data, the statistic’s weights and conditional distribution depend on the interim result; its null calibration must be checked under that adaptive rule.
• Mistake: “If the pooled estimate is calculated correctly, its p-value must be correct.” Correction: Estimation arithmetic and test calibration are separate. The estimate can be correct while the reference distribution used for the p-value is not.
• Mistake: “A large conditional rejection chance at one interim value is the overall Type I error.” Correction: Overall Type I error is the probability of rejection averaged over all possible interim outcomes under \(H_0\), including how each outcome determines the sample size.

The example assumes independent normal observations and known common variance. Actual applications may involve estimated variances, discrete outcomes, covariate adjustment, unequal variances, missing data, or adaptation based on nuisance estimates. Such settings can change the distribution of stagewise statistics, so the simple normal calculation should not be transferred without justification. Nor should the example be read as a universal numerical inflation factor: the amount, and sometimes the direction, of a calibration error depends on the adaptation and rejection rules.