Changing the Second-Stage Test After Re-Estimation

Why a Re-Estimated Design May Need a Different Test

After an interim analysis, a design may change the second-stage sample size, the information planned for that stage, or the test used to analyse its data. The key question is not whether the replacement test resembles the planned one. It is whether, given everything known at the interim, the replacement can reject often enough under the null to exceed the error allowance left by the original design.

The earlier material on preserving conditional error gives the basic principle: control the probability of rejection after the interim, conditional on the interim information. Here the focus is the test change itself. The replacement may use a different statistic or test procedure, but its null behavior must be calibrated to the remaining conditional error. Validity is conditional on the information used to choose the new procedure, not merely validity of that procedure in a fixed design.

Definition: Let $H$ denote the full interim history available when the second-stage plan is changed, including the first-stage data and the sample-size decision. For a planned test with rejection indicator $\phi_0$, its conditional error at null parameter value $\theta$ is $$ A_\theta(H)=P_\theta(\phi_0=1\mid H). $$ It is the original test’s probability of eventual rejection, conditional on the observed interim history. A replacement test preserves the conditional-error bound if, for every $\theta$ in the null hypothesis and almost every possible $H$, its conditional rejection probability is no greater than $A_\theta(H)$.

This definition assumes that the original design and its rejection rule are specified, and that $H$ records all information used to adapt the design. Conditional probabilities must be computed under the null model. For a composite null, the condition must hold for every null parameter value, or a usable bound must be established that controls error over the whole null. A calculation at just one convenient nuisance-parameter value does not establish composite-null control.

The Error-Control Argument

Suppose the original test has Type I error at most $\alpha$ for every $\theta\in H_0$. Write $\phi_1$ for the rejection indicator of the replacement procedure. If its conditional rejection probability is bounded by the original conditional error, then the unconditional rejection probability cannot be larger than that of the original test.

\begin{aligned} P_\theta(\phi_1=1) &=E_\theta\!\left[P_\theta(\phi_1=1\mid H)\right]\\ &\leq E_\theta\!\left[A_\theta(H)\right]\\ &=P_\theta(\phi_0=1)\\ &\leq \alpha,\qquad \theta\in H_0. \end{aligned}

The first and third equalities are the law of total probability: averaging a conditional rejection probability over the distribution of $H$ recovers the unconditional rejection probability. This argument does not require the new test to have the original test’s form. It does require the conditional bound to hold under the null, after conditioning on all the information that affected the change.

A common practical implementation uses a replacement $p$-value. If $p_{\mathrm{new}}$ is conditionally valid under the null, meaning $P_\theta(p_{\mathrm{new}}\leq u\mid H)\leq u$ for every $u\in[0,1]$, then rejecting when $p_{\mathrm{new}}\leq A_\theta(H)$ uses no more than the conditional error. For a simple null, $A_\theta(H)$ may be directly computable. For a composite null, the cutoff must be available without knowing the true nuisance parameter and must still provide uniform control. One valid route is to use a computable conditional cutoff $C(H)$ that bounds the replacement’s conditional rejection probability and satisfies $E_\theta[C(H)]\leq\alpha$ for every $\theta\in H_0$.

Worked Example: Computing the Original Conditional Error

Consider a hypothetical one-sided test with two independent stagewise statistics. Under the simple null, $Z_1$ and $Z_2$ are independent standard normal random variables. The planned test rejects when $Z_{\mathrm{old}}=\sqrt{t_0}Z_1+\sqrt{1-t_0}Z_2>z_{1-\alpha}$. Take $\alpha=0.025$, $t_0=0.50$, and suppose the observed first-stage statistic is $Z_1=0.80$. These assumptions give a standard normal reference distribution for the planned statistic under the null.

Conditional on $Z_1=0.80$, the planned test rejects if $$ Z_2> \frac{z_{0.975}-\sqrt{0.50}(0.80)}{\sqrt{0.50}} = \frac{1.960-0.5657}{0.7071} \approx 1.972. $$

Thus the original conditional error is $$ A(0.80)=P(Z_2>1.972)\approx 0.0243. $$

The first-stage result has therefore left about $0.0243$ conditional probability of rejection under the null. A replacement second-stage test must spend no more than this allowance, conditional on this interim result. It is not appropriate simply to give the replacement the full nominal level $0.025$ for every interim history.

Using a Replacement Test and a Conditional P-Value

A second-stage $p$-value can come from a different statistic, endpoint, or analysis procedure, provided its null calibration is valid conditional on the interim history. In particular, if the sample size is selected using $H$, the null distribution used for the new test must account for that selection. Independent future observations often make the condition straightforward: given $H$, the new data have the stated null distribution and are not reused first-stage observations. Independence is not essential if a correct conditional distribution is available, but it cannot be assumed without justification.

Worked Example: Replacing the Test After Sample Size Re-Estimation

Suppose the hypothetical design above re-estimates the second-stage sample size after observing $Z_1=0.80$. Investigators then use a different second-stage score statistic for the same one-sided scientific question. Assume the new statistic $Z_{\mathrm{new}}$, calculated from fresh second-stage observations at the selected sample size, is standard normal under the simple null conditional on the interim history. This assumption must be justified by the sampling model and the construction of the statistic; it does not follow merely from having increased the sample size.

The one-sided conditional $p$-value is $p_{\mathrm{new}}=1-\Phi(Z_{\mathrm{new}})$, where $\Phi$ is the standard normal cumulative distribution function. Use the conditional-error cutoff $0.0243$ from the preceding calculation. If the observed replacement statistic is $Z_{\mathrm{new}}=2.10$, then $$ p_{\mathrm{new}}=1-\Phi(2.10)\approx 0.0179. $$

Since $0.0179<0.0243$, the design rejects at this interim history. In contrast, if $Z_{\mathrm{new}}=1.85$, then $$ p_{\mathrm{new}}=1-\Phi(1.85)\approx 0.0322, $$ so it does not reject. Conditional validity ensures that, under the null, the probability of crossing the $0.0243$ cutoff is at most $0.0243$. The sample-size change is allowed; the validity of the replacement statistic and the conditional-error calibration are what protect the error rate.

The same logic applies when the replacement statistic is not normally distributed. The design needs a $p$-value that is super-uniform under the null conditional on $H$: at every threshold $u$, the conditional probability that the $p$-value is at most $u$ must not exceed $u$. If the new test is discrete, for example, its $p$-value may be conservative rather than exactly uniform; conservative conditional $p$-values remain valid for this bound.

Conditions to Check Before Changing the Test

A valid change is not guaranteed by the fact that the second-stage analysis was described as “preplanned” after re-estimation, nor by the fact that each candidate test is valid in isolation. The following conditions make the claim precise.

Define the interim history.
Include the first-stage results, sample-size re-estimation, and every other interim quantity used to choose or calibrate the replacement. Conditioning on too little information can conceal selection.

Determine the available error.
Compute the planned test’s conditional rejection probability under the null, or establish a computable bound that is valid for every null parameter value.

Validate the new test conditionally.
Show that, given the interim history and the chosen sample size, the replacement $p$-value is super-uniform under the null, or otherwise show directly that the conditional rejection probability is bounded by the available error.

Apply the conditional cutoff.
Reject only when the replacement’s conditional rejection probability does not exceed the available bound. Then average over interim histories to obtain the unconditional Type I error guarantee.

For a composite null, each step must hold uniformly over the null values that matter. The conditional error itself can depend on nuisance parameters, such as a variance. If the design cannot calculate a parameter-specific cutoff, it needs a conservative bound that is both usable at the interim and sufficient to control the unconditional error. Taking a pointwise worst case over nuisance values is not automatically enough: the resulting random cutoff must also have the required error control when averaged under each null value.

Worked Example: Why Selecting a Test From Stage-Two Results Can Fail

Suppose the available conditional error is $0.025$, and two one-sided replacement tests are applied to the same second-stage data. Under the null and conditional on the interim history, suppose their $p$-values $p_1$ and $p_2$ are independent uniform random variables. Each test is individually valid. However, choosing whichever test produces the smaller $p$-value and rejecting if that value is at most $0.025$ gives $$ P(\min(p_1,p_2)\leq 0.025\mid H) =1-(1-0.025)^2 =0.0494. $$

The conditional rejection probability is about $0.0494$, almost twice the available $0.025$. The mistake is to infer that a data-selected test retains its individual calibration. One simple correction in this example is to reject only if either $p$-value is at most $0.025/2=0.0125$. The union bound then gives a conditional rejection probability no greater than $0.0125+0.0125=0.025$. Other valid calibrations are possible, but the selection rule must be included in the conditional error calculation.

Diagnostics, Alternatives, and Limitations

Several checks are especially useful when reviewing an adaptive analysis:

Audit the information used for adaptation. List the data and interim decisions that affected the sample size, endpoint, statistic, or analysis method. The conditional validity claim must account for all of them.
Check the null distribution at the chosen sample size. A statistic standardized for a fixed sample size may not have the same conditional distribution after sample size selection. Re-estimation can change information, variance estimation, or the distribution of a studentized statistic.
Check for data reuse. A replacement test that reuses first-stage observations may not have a conditionally valid second-stage $p$-value. If dependence is present, it must be modelled in the conditional calibration.
Check the whole rejection rule. Choosing among tests, endpoints, or transformations based on second-stage results is part of the test, not an innocuous reporting choice. Calibration of each option alone does not calibrate their selection.
Check uniformity for a composite null. A conditional calculation at one nuisance-parameter value may fail at another. The claimed bound must cover the null values for which Type I error control is required.

Keeping the original second-stage test is the simplest option when its statistic and reference distribution remain valid after the sample-size change. It avoids a test-selection issue, but it does not by itself justify using the original fixed-design cutoff after the information structure changes. Conditional-error recalibration is more flexible: it allows a different second-stage procedure if that procedure respects the remaining conditional allowance. Combination approaches instead build a final test from valid stagewise evidence under a specified calibration; they are another design strategy, not a reason to treat an arbitrary replacement $p$-value as valid.

Conditional-error control also has limits. It does not repair a misspecified null distribution, invalid second-stage $p$-values, or unrecorded adaptation. A valid bound can be conservative, especially with discrete data or uncertain nuisance parameters. And preserving Type I error does not guarantee adequate power, unbiased effect estimation, or a useful confidence interval. Those properties require their own assessment.

Key takeaway: A second-stage test can change after sample size re-estimation when its conditional rejection probability, given the complete interim history, is no greater than the original test’s conditional error (or another valid conditional-error bound). The test’s conditional null calibration and the cutoff together—not the test’s fixed-design validity alone—deliver Type I error control.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

What information should be included in the interim history when calculating a conditional-error bound?
At a particular interim history, the original conditional error is $0.018$. A replacement test has a conditionally valid $p$-value of $0.021$. Does the conditional-error rule reject?
Why does a valid fixed-sample second-stage test not automatically remain valid after sample-size re-estimation?
Two independent uniform conditional $p$-values are searched, and the smaller is compared with $0.02$. What is the conditional probability of rejection?

Solutions. (1) Include the first-stage data and every interim result or decision used to set the sample size or choose, modify, or calibrate the replacement test. (2) No. The $p$-value $0.021$ exceeds the available conditional error $0.018$, so the replacement rule does not reject. (3) Re-estimation can change the conditional null distribution, and the test may also depend on information used to choose the sample size. Validity must be checked conditional on that information at the selected sample size. (4) The probability is $1-(1-0.02)^2=0.0396$, which exceeds $0.02$; selecting the smaller $p$-value is not protected by either test’s individual validity.