Nuisance-Driven Re-Estimation and Error Control

Why Nuisance-Driven Re-Estimation Needs Its Own Validity Check

A second-stage sample size may be changed because the observed variability is larger than expected, or because the event rate suggests that the original plan will yield less information than anticipated. These are nuisance-driven changes: they use quantities that affect precision or information, rather than the estimated treatment effect itself. That distinction is useful, but it does not by itself guarantee type I error control. The key question is whether, after the nuisance information is observed and the sample size is chosen, the test still has a valid null distribution.

The relevant standard is conditional validity. If the adaptation information is denoted by \(A\), then for each possible value \(a\), the testing procedure should control its conditional probability of rejection under \(H_0\). Averaging valid conditional error rates over possible values of \(A\) then controls the unconditional type I error. If the nuisance estimate is related to the treatment contrast, however, conditioning on it can change the distribution of the test statistic. Blinding may conceal the treatment labels from the person making the choice, but concealment alone does not establish independence or conditional validity.

Sample Size Calculations Use Nuisance Quantities

For planning, nuisance estimates commonly enter a standard-error or information calculation. For example, for two independent groups with normally distributed outcomes, equal group sizes, common known standard deviation \(\sigma\), and a one-sided test at level \(\alpha\), the approximate per-group sample size for power \(1-\beta\) at a target difference \(\delta>0\) is

Here \(z_q\) is the \(q\)-quantile of the standard normal distribution. The formula makes clear why an estimated \(\sigma^2\) may change the planned sample size: larger variance requires more observations to achieve the same target precision and power. But this planning calculation is not itself a type I error argument. That argument concerns how the final rejection rule behaves under \(H_0\), after the sample size has been selected.

The Conditional Validity Argument

A convenient way to see the protection is through a one-sided inverse-normal combination test. Let \(P_1\) and \(P_2\) be stagewise p-values, and define \(Z_i=\Phi^{-1}(1-P_i)\), where \(\Phi\) is the standard normal cumulative distribution function. Suppose the adaptation information \(A\) determines the second-stage sample size and therefore determines weights \(w_1(A)\) and \(w_2(A)\). A combination statistic is

The exact argument below requires that, under \(H_0\), conditional on \(A\), the stagewise p-values are independent and uniform on \((0,1)\), and that the weights are fixed once \(A\) is given. In that case, conditional on \(A\), \(Z_1\) and \(Z_2\) are independent standard normal variables. Thus \(C\), conditional on \(A\), is standard normal. For a level-\(\alpha\) upper-tail test, reject when \(C>z_{1-\alpha}\). The conditional rejection probability is then \(\alpha\) for every value of \(A\). Averaging over \(A\) gives the unconditional result:

This is a specific sufficient condition, not a blanket guarantee for every nuisance-based rule. In particular, if the first-stage p-value is not uniform conditional on the information used to select the weights or sample size, the normal combination argument does not follow. For discrete tests, p-values may instead be super-uniform; the test then needs a calibration that accounts for that fact rather than assuming continuous uniformity without justification.

Event Rates and Blinded Information

Binary outcomes introduce a related issue. An event-rate estimate can affect the variance of a difference in proportions, and hence the planned information. A blinded pooled event rate may be attractive because it does not identify which treatment group has the higher observed rate. Nevertheless, pooled information is still computed from outcomes. It cannot be declared independent of the treatment statistic merely because treatment labels were hidden.

One way to make the reasoning explicit is to use an independent nuisance source, such as a prespecified external estimate or a separate blinded run-in sample, for the sample-size choice. Another is to use a test and conditioning argument that establishes conditional null validity given the information used in adaptation. For example, conditional tests for binary outcomes can condition on appropriate event totals; discreteness then often makes the p-values conservative rather than exactly uniform. The particular test and conditioning set must match the adaptation rule.

Why Outcome-Dependent Choices Can Break the Argument

The danger is not limited to deliberately choosing a sample size from a treatment-effect estimate. It arises whenever adaptation information carries information about the statistic used for testing and the final procedure treats that information as if it were irrelevant. Suppose the first-stage contrast tends to be positive under \(H_0\) simply through random variation, and a rule responds to that contrast by selecting a second-stage size or combination weight. Conditional on that choice, the distribution of the first-stage evidence is no longer the unconditional null distribution used by a fixed-design cutoff.

A common error is to say, “The second-stage sample size depends only on variance, so type I error is unchanged.” The missing step is a dependence check. If the variance estimate is statistically independent of the first-stage test statistic under the null, and the stagewise testing construction remains valid conditional on that estimate, the argument can work. If the estimate shares data with the test statistic, this independence or conditional calibration must be established—not assumed. Conversely, an outcome-dependent choice is not automatically invalid if the final procedure explicitly preserves conditional error; it is invalid to use a fixed-design calibration without showing that the calibration still applies.

Diagnostics, Alternatives, and Limitations

Practical diagnostics should focus on the joint behaviour of the adaptation statistic and the testing statistic under the null model. Simulation can be useful: generate data under representative null parameter values, repeat the full sample-size selection and testing algorithm, and estimate the rejection frequency. This is a check on the complete design, not a substitute for a proof when a formal guarantee is required. Look for unexpected association between the nuisance estimate and the stage-one signed statistic, changes in the conditional distribution of stagewise p-values across adaptation values, and differences between simulated and claimed conditional rejection rates.

The main alternatives are to use an independent nuisance estimate, to use a test that is valid conditional on the observed nuisance information, or to recalibrate the final test through conditional-error methods. The first can simplify the independence argument but may be less precise or less current. Conditioning can give rigorous control but may cost power, especially with discrete outcomes. Conditional-error recalibration can accommodate broader adaptations, but it requires computing and preserving the relevant conditional rejection probability. These approaches are not interchangeable without checking their assumptions.

Nuisance-based re-estimation also has limits beyond type I error. A variance estimate from a small interim sample can be unstable; a second-stage size based on it may be too small or unnecessarily large. A sample-size cap may prevent the planned power from being attained, even when type I error remains controlled. Misspecified variance or event-rate models can make the information calculation inaccurate. Therefore report the estimate used, the rule mapping it to sample size, any bounds or rounding, and the final testing calibration. Sample-size adaptivity can improve planning, but it cannot guarantee power under every nuisance value or repair a misspecified outcome model.

Common Pitfall

• Wrong reasoning: “The sample size is based on a blinded pooled variance or event rate, not the estimated treatment effect, so the nominal type I error is automatically preserved.”
• Correction: Show that the test remains calibrated conditional on the information used for adaptation, or use a procedure that explicitly preserves the relevant conditional error. Blinding may help prevent a direct effect-driven choice, but it does not itself prove conditional independence or conditional validity.