Maintaining A Fixed Final Test After Re-Estimation

What It Means to Keep the Final Test Fixed

A sample-size change need not change the primary hypothesis test. The trial can retain the same primary estimand, null and alternative hypotheses, analysis model, test direction, significance level, and final decision rule in standardized units. What changes is the amount of data used to calculate the final statistic and its standard error. The central qualification is important: writing down the familiar final statistic and cutoff does not, by itself, prove that the test has its advertised Type I error after an adaptive sample-size rule.

Consider a two-arm trial with a continuous outcome, equal allocation, and a treatment effect defined as the difference in population means, $\Delta=\mu_T-\mu_C$. For a one-sided superiority test, suppose the primary hypotheses are $H_0:\Delta\leq 0$ and $H_a:\Delta>0$, with significance level $\alpha$. A fixed final test evaluates the treatment contrast at the end of the trial using all observations in the final analysis population. It does not test a new hypothesis merely because the sample size changed.

Definition: A fixed final test after re-estimation retains the prespecified primary estimand, hypotheses, analysis method, tail, and nominal significance level after the sample size is updated. The final statistic is calculated using the final sample size and all data included in the prespecified analysis. The phrase “fixed test” describes the inferential target and rule; it does not guarantee Type I error control for every sample-size adaptation.

For known common standard deviation $\sigma$, with $n$ participants per arm, the usual final statistic is

Z_n=\frac{\bar{X}_T-\bar{X}_C}{\sigma\sqrt{2/n}}.

Under the fixed-sample normal model and $H_0:\Delta=0$, $Z_n\sim N(0,1)$. A one-sided level-$\alpha$ test rejects when $Z_n>z_{1-\alpha}$, where $z_{1-\alpha}$ is the standard normal quantile with cumulative probability $1-\alpha$. For a two-sided test, the rule is $|Z_n|>z_{1-\alpha/2}$. The test’s standardized cutoff stays the same if $n$ changes, provided the same $\alpha$, direction, and reference distribution remain appropriate.

Why the Standardized Test Can Stay the Same

The underlying logic is standardization. Under a fixed sample size and the stated model, the treatment-control difference has mean $\Delta$ and variance $2\sigma^2/n$. Under the null boundary $\Delta=0$, dividing by its standard deviation gives a standard normal statistic. Changing $n$ changes the standard error, but not the standard normal reference distribution at a prespecified fixed sample size.

Keep the target fixed.
Specify the same treatment contrast, null hypothesis, analysis population, and one- or two-sided alternative before the trial begins.

Update the precision for the final sample size.
Use the final number of analyzed participants per arm in the standard error, rather than the original planned number.

Apply the prespecified final rule.
Compare the resulting statistic with the critical value for the unchanged nominal $\alpha$, but only if that rule has valid Type I error under the adaptation.

A useful sufficient condition for validity is that, under the null, the sample-size choice is independent of the final standardized treatment contrast in a way that ensures its conditional null distribution remains the stated reference distribution. For example, if the final sample size is selected using an external pilot estimate independent of trial outcomes, then conditioning on the selected sample size leaves the trial’s fixed-sample null distribution unchanged. Other re-estimation rules require their own justification; independence should not be assumed merely because a decision is described as blinded.

Justification: Suppose $N$ is selected from a set of possible per-arm sample sizes using information independent of the trial’s treatment contrast under $H_0$, and $Z_N\mid N=n\sim N(0,1)$ for every allowed $n$. For a one-sided test with cutoff $c=z_{1-\alpha}$, $$ P_{H_0}(Z_N>c)=\sum_n P_{H_0}(Z_N>c\mid N=n)P(N=n) =\sum_n \alpha P(N=n)=\alpha. $$ Thus the nominal test retains its Type I error in this setting. The key is the conditional null distribution, not merely the fact that the same formula and cutoff are used.

Worked Example: An Independent Sample-Size Update

Suppose a hypothetical two-arm trial plans 60 participants per arm and will use a one-sided test at $\alpha=0.025$, with known $\sigma=10$ outcome units. An external pilot, conducted independently of the trial, leads the sponsor to increase the final target to 90 participants per arm. The final treatment-control difference is 3.4 outcome units. Assume independent normal outcomes and complete outcome data.

The final standard error is based on 90 participants per arm:

SE=10\sqrt{\frac{2}{90}}=1.491, \qquad Z_{90}=\frac{3.4}{1.491}=2.280.

The one-sided critical value remains $z_{0.975}=1.960$, because the primary test and $\alpha$ have not changed. Since $2.280>1.960$, the result rejects $H_0$; the one-sided p-value is approximately 0.0113. Under the independent-selection condition, the Type I error reasoning in the justification applies. The original target of 60 is not used in the final standard error.

The equivalent threshold on the raw treatment-control difference is not fixed. With 90 per arm it is $1.960(1.491)=2.922$ outcome units. At 60 per arm it would have been $1.960(10\sqrt{2/60})=3.578$ outcome units. The fixed element is the standardized critical value, not a particular raw difference.

The Final Statistic Uses the Final Analysis

When the sample size increases, the final statistic ordinarily uses all eligible observations, including those enrolled before the update, under the prespecified analysis plan. Replacing the denominator with the final sample size while dropping earlier observations, or reusing an interim statistic as if it were the final statistic, defines a different analysis. The protocol should identify the analysis population, handling of missing outcomes, estimator, variance estimator, and final sample-size convention.

With unknown common variance, the familiar fixed-sample procedure uses the pooled standard deviation $s_p$ and a $t$ statistic. For equal allocation with $n$ participants per arm, it is

t=\frac{\bar{X}_T-\bar{X}_C}{s_p\sqrt{2/n}}, \qquad df=2n-2.

Under independent normal observations with a common variance and a fixed sample size, this statistic has a $t$ distribution with $2n-2$ degrees of freedom under the null boundary. An adaptive rule can alter that reference distribution. Therefore, after re-estimation, the usual $t$ cutoff is justified only if the design and analysis establish that the adaptation preserves the required null distribution or otherwise calibrate the final test. The same caution applies to other outcome models and variance estimators.

Worked Example: Recomputing a Pooled-Variance Final Test

In a hypothetical trial, the final analysis includes 40 participants per arm. The observed treatment-control difference is 4.0 score points, and the pooled standard deviation is 12.5 points. For illustration, assume the sample size was fixed or that the adaptation has separately been shown to justify the usual pooled-$t$ reference distribution. Use a two-sided test at $\alpha=0.05$.

The standard error and statistic are

SE=12.5\sqrt{\frac{2}{40}}=2.795, \qquad t=\frac{4.0}{2.795}=1.431, \qquad df=40+40-2=78.

The two-sided critical value is approximately $t_{0.975,78}=1.991$. Since $|1.431|<1.991$, the test does not reject $H_0$; the two-sided p-value is approximately 0.156. The estimated difference is 4.0 score points, but this result does not provide statistically significant evidence of a nonzero difference at the 0.05 level. The assumption about the reference distribution matters: calculating a $t$ statistic does not establish that its ordinary $t$ distribution holds after any arbitrary re-estimation.

When Keeping the Formula Is Not Enough

If sample size depends on interim treatment-effect information, the rule selects the final sample size partly according to observed evidence. Conditional on that selection, the final data and statistic may no longer have the ordinary fixed-sample null distribution. Using the same nominal cutoff can then produce a Type I error different from $\alpha$. Similar concerns can arise with nuisance-parameter re-estimation based on trial data; whether a particular blinded rule preserves error control depends on its details and must be justified, not inferred from the label “blinded.”

A fixed final test is also different from repeatedly testing for efficacy. If there is no interim efficacy decision and only one final hypothesis test, there is no repeated efficacy testing to adjust for. If efficacy is assessed at interim analyses, the design needs a decision rule that accounts for those looks; simply retaining a conventional final cutoff does not undo earlier opportunities to reject. Group sequential boundaries and combination tests are alternatives when the design includes such interim decisions or when adaptation makes the ordinary final reference distribution unsuitable.

Worked Example: Random Sample Size Does Not Guarantee a Valid Test

Suppose a trial uses either 50 or 80 participants per arm. Under the null, the final standardized statistic would have a standard normal distribution at either fixed sample size. If an independent external rule chooses 50 per arm with probability 0.4 and 80 per arm with probability 0.6, then for a one-sided $\alpha=0.05$ test with cutoff 1.645,

P_{H_0}(Z_N>1.645) =0.4(0.05)+0.6(0.05)=0.05.

Now change the rule so that the larger sample is selected when the interim treatment contrast looks unusually favorable. The calculation above no longer follows: it relied on a 0.05 conditional rejection probability at each selected sample size. Selection based on interim evidence can change the conditional distribution of the final statistic. Without a derivation or design-based calibration for that rule, the nominal cutoff does not establish a 5% Type I error.

Diagnostics, Alternatives, and Limitations

Before interpreting the final p-value, check the connection between the adaptation and the final test. Confirm that the primary estimand, analysis population, test direction, significance level, and model are unchanged; verify that the final standard error uses the final data and sample size; and establish why the reference distribution remains valid under the actual re-estimation rule. A mismatch between the protocol’s planned sample-size rule and the rule implemented during conduct is especially important, because the nominal test may no longer describe the design that was run.

When independence or a suitable conditional distribution cannot be established analytically, design-based simulation under the null can assess calibration across plausible nuisance-parameter values. Such simulation should reproduce the full adaptation rule, randomization, stopping or cap rules, and final analysis. A single simulation scenario is not proof of general error control. Nor does a nonsignificant final result show that the adaptation was valid: Type I error is a property of the procedure’s repeated-sampling behavior.

Approach	Final decision rule	Main consideration
Fixed-sample final test after a justified update	Use the final statistic and the original nominal critical value.	Requires a valid null reference distribution under the sample-size rule.
Group sequential design	Use prespecified boundaries for interim and final efficacy looks.	Accounts for repeated efficacy testing; it is not equivalent to a single fixed final test.
Combination or conditional-error approach	Use a calibrated rule that incorporates information from stages or constrains the remaining rejection probability.	Can accommodate some adaptations, but its statistic and calibration must be specified for that design.

A fixed final test is attractive because it preserves a simple clinical question and a familiar interpretation. Its limitation is that simplicity of the final formula cannot substitute for a valid design-level argument. In particular, a standard normal or $t$ statistic computed at the realized sample size is not automatically a standard fixed-sample test when that sample size was chosen using information from the same trial.

Common Mistake

Wrong reasoning: “We kept the same $Z$ statistic and the same 0.05 cutoff, so the Type I error must still be 0.05.” This ignores how the random sample-size decision may depend on the data and alter the statistic’s null distribution.

Correction: Define the final test using the final analysis sample and the prespecified hypotheses, then justify its null distribution under the actual adaptation rule. If the required distributional condition does not hold, use an appropriately calibrated method rather than treating the unchanged cutoff as proof of validity.

Check Your Understanding

Answer each question, then compare with the solutions.

A two-sided test has $\alpha=0.05$. If the sample size increases but the test remains valid, what happens to its standardized critical value, and what happens to the raw-effect threshold?
Under what condition does the mixture argument establish that a random final sample size preserves a one-sided nominal level $\alpha$?
A researcher changes the sample size based on an interim treatment-effect estimate but keeps the usual final $Z$ cutoff. Is that enough to establish Type I error control? Why or why not?
For a final difference of 2.5 units, known $\sigma=8$, and $n=64$ per arm, calculate the one-sided $Z$ statistic and compare it with the 0.05 critical value.

Solutions. (1) The standardized cutoff remains $z_{0.975}=1.960$ if the same valid two-sided test is retained. The raw-effect threshold changes because the standard error changes with sample size. (2) It is sufficient that, for every allowed sample size, the conditional null rejection probability given the selected size is $\alpha$; then averaging those probabilities over the sample-size choices gives $\alpha$. (3) No. Selection based on interim treatment-effect information can alter the conditional null distribution of the final statistic, so the ordinary cutoff requires a separate validity argument or calibration. (4) $SE=8\sqrt{2/64}=1.414$, so $Z=2.5/1.414=1.768$. Since $1.768>z_{0.95}=1.645$, this fixed-sample one-sided calculation rejects at 0.05, subject to the stated test assumptions and a valid sample-size rule.