Flexible Stage-Two Information Under Combination Testing

Why Stage-Two Information Can Change

An interim analysis may lead investigators to increase or decrease the planned stage-two sample size, and therefore the information available at the final analysis. Such adaptation need not invalidate a combination test. The key is not that stage-two information stays fixed; it is that the stage-two p-value remains valid conditional on the interim data and the selected stage-two design. That distinction allows the test to account for a data-dependent sample size without pretending that the selection never occurred.

Conditional validity is a property of the stage-two test, not a blanket guarantee for every way of combining its p-value with the interim result. In particular, a random information-based weight or a changed rejection boundary requires its own justification. The combination rule and its calibration must be considered alongside the stage-two p-value.

Definition: Let $\mathcal{F}_1$ denote the information available at the interim analysis, including any stage-two design choices made from it. A stage-two p-value $p_2$ is conditionally valid under $H_0$ if, for every $u$ with $0 \le u \le 1$, $$ P(p_2 \le u \mid \mathcal{F}_1) \le u. $$ If equality holds for every $u$, the p-value is conditionally uniform. A conditionally valid p-value is super-uniform; a conditionally uniform p-value has the exact uniform null distribution even after conditioning on the interim information.

The relevant conditioning includes the selected sample size or information level. If the stage-two plan is chosen using stage-one data alone, the stage-two observations have not yet been used to make that choice. A stage-two test can therefore retain its null calibration at the selected design, provided its model and test assumptions hold. This is different from selecting the design after looking at stage-two outcomes and then using an ordinary fixed-design p-value.

Conditions for Conditional Validity

A useful way to assess a proposed adaptation is to write down what is known when the design is chosen and what remains random afterward. Under the null hypothesis, conditional validity typically requires the following:

Interim-measurable selection: the stage-two sample size or information is chosen using $\mathcal{F}_1$, before the stage-two outcomes used in $p_2$ are observed.
Valid stage-two calibration: for each design that could be selected, the stated stage-two test has a valid null p-value at that design. The test must use the right reference distribution and account for relevant nuisance parameters.
Suitable separation of data: stage-two outcomes must have the stated conditional null distribution given $\mathcal{F}_1$ and the selected design. Independence of new stage-two observations from stage one is a common sufficient condition.
Appropriate handling of further monitoring: if stage-two outcomes themselves determine when to stop or which analysis to report, the p-value must account for that additional selection. A fixed-sample p-value does not automatically do so.

These are assumptions to verify, not consequences of calling a design adaptive. For example, a nominal z-test may fail to provide a conditionally valid p-value if its variance estimate, reference distribution, or data-dependent analysis is not justified under the null after conditioning. Likewise, a sample size selected using stage-one data is not a problem by itself, but selecting it after inspecting the stage-two outcomes changes the calibration question.

Worked Example: Selecting a Stage-Two Sample Size at Interim

Suppose a one-sided test concerns a treatment-minus-control mean difference, with $H_0:\mu=0$. For illustration, assume independent normally distributed outcomes with known standard deviation $\sigma=10$ units in each group, and use the standard error $10/\sqrt{n}$ for the stage-two contrast. At interim, investigators use only stage-one data to choose $n=40$ for stage two. The stage-two estimated difference is $2.4$ units.

Conditional on the interim data and the selected $n=40$, assume the new stage-two observations are independent of stage one and the stated normal model holds. Under $H_0$, the stage-two z-statistic is standard normal. Its observed value is $$ z_2=\frac{2.4}{10/\sqrt{40}}=\frac{2.4}{1.5811}=1.518. $$

The one-sided p-value is $p_2=1-\Phi(1.518)\approx 0.0645$. Its null distribution is uniform conditional on the interim information because the normal z-test is exact for this known-variance model at the selected sample size. The numerical p-value does not prove the assumptions; the justification comes from the selection timing, the independent new data, and the correct null model.

Fixed-Weight Combination With Flexible Information

For a one-sided test with evidence in the positive direction, transform each stagewise p-value to a normal score: $Z_i=\Phi^{-1}(1-p_i)$, where $\Phi$ is the standard normal cumulative distribution function. A fixed-weight inverse-normal combination statistic is $Z_C=w_1Z_1+w_2Z_2$, with prespecified constants $w_1,w_2\ge0$ and $w_1^2+w_2^2=1$. Reject $H_0$ at one-sided level $\alpha$ if $Z_C>z_{1-\alpha}$, where $z_{1-\alpha}=\Phi^{-1}(1-\alpha)$.

The weights in this rule are fixed rather than recomputed from the selected stage-two information. Thus stage-two information may vary while the rule for combining the two p-values remains the same. Provided $p_1$ is valid under $H_0$, and $p_2$ is conditionally valid given $\mathcal{F}_1$, this fixed-weight rule controls Type I error. Exact conditional uniformity gives the familiar exact standard-normal calibration when the first-stage p-value is also exact.

Result: Suppose $Z_1=\Phi^{-1}(1-p_1)$ is standard normal under the null, and $Z_2=\Phi^{-1}(1-p_2)$ is conditionally standard normal given $\mathcal{F}_1$. If $w_1,w_2$ are fixed, nonnegative constants satisfying $w_1^2+w_2^2=1$, then $Z_C=w_1Z_1+w_2Z_2$ is standard normal under $H_0$, even if the stage-two information is selected from $\mathcal{F}_1$. For super-uniform p-values, the same upper-tail rejection rule is conservative rather than necessarily exact.

The justification uses conditional distributions. Given $\mathcal{F}_1$, $Z_1$ is already determined, while $Z_2$ has a standard normal distribution under the exact conditional-validity assumption. Therefore, $$ P(Z_C>c\mid\mathcal{F}_1) =1-\Phi\!\left(\frac{c-w_1Z_1}{w_2}\right) $$ when $w_2>0$. Averaging over the stage-one data gives the standard-normal distribution for the weighted sum: conditional on $\mathcal{F}_1$, $Z_2$ is independent of the fixed value $Z_1$, and the two normal scores have the required standard normal marginal distributions. Equivalently, $Z_1$ and $Z_2$ are independent standard normal variables, so the variance of their fixed-weight sum is $w_1^2+w_2^2=1$. The selected stage-two information does not alter this argument because it is already included in the conditioning for $p_2$.

Worked Example: Combining P-Values After Information Re-Estimation

Suppose a one-sided design uses fixed equal weights, $w_1=w_2=1/\sqrt{2}$. Stage-one and conditionally valid stage-two p-values are $p_1=0.20$ and $p_2=0.04$, respectively. The stage-two information was selected using interim data, but the p-value is calibrated conditional on that selected design.

The normal scores are $Z_1=\Phi^{-1}(0.80)=0.8416$ and $Z_2=\Phi^{-1}(0.96)=1.751$. Thus $$ Z_C=\frac{0.8416+1.751}{\sqrt{2}}=1.833. $$

At one-sided $\alpha=0.05$, the critical value is $z_{0.95}=\Phi^{-1}(0.95)=1.645$. Since $1.833>1.645$, reject $H_0$ at the 0.05 level. The combined p-value is $1-\Phi(1.833)\approx0.0334$. Under the stated assumptions, this is evidence against the null in the positive direction; it does not establish that the treatment caused the difference.

Why Random Weights Need Separate Calibration

It is tempting to set the weights from the realized information, for example using $w_1=\sqrt{I_1/(I_1+I_2)}$ and $w_2=\sqrt{I_2/(I_1+I_2)}$, where $I_1$ and $I_2$ are stagewise information. If $I_2$ is chosen using stage-one data, these weights are themselves functions of $\mathcal{F}_1$. Conditional validity of $p_2$ still holds, but the fixed-weight normal-sum argument does not apply: conditional on $\mathcal{F}_1$, the first-stage contribution $w_1Z_1$ is already determined and its weight may depend on $Z_1$. A random weighted sum need not have the standard normal null distribution.

A simple example shows why. Suppose $Z_1$ and $Z_2$ are independent standard normal variables under the null, and $Z_2$ is conditionally valid. Choose the rule $Z_C=Z_1$ if $Z_1>1.645$, and $Z_C=Z_2$ otherwise. This corresponds to selecting one of two endpoint weight pairs based on interim data. At a nominal one-sided level of 0.05, the rejection probability is $$ P(Z_1>1.645)+P(Z_1\le1.645)P(Z_2>1.645) =0.05+(0.95)(0.05)=0.0975. $$

The stage-two p-value remains valid in this example, yet the adaptive combination rule nearly doubles the nominal Type I error. The error is in applying the fixed standard-normal cutoff to a statistic whose weights were chosen from the first-stage result.

Worked Example: Preserving Conditional Error at a Given Interim Result

Consider an original equal-weight inverse-normal test at $\alpha=0.05$, with $Z_C=(Z_1+Z_2)/\sqrt{2}$. At an interim result $Z_1=0$, its rejection condition is $Z_2/\sqrt{2}>1.645$, or $Z_2>1.645\sqrt{2}=2.326$. Under a conditionally standard normal $Z_2$, the original conditional rejection probability at this interim value is approximately $1-\Phi(2.326)=0.0100$.

If a revised procedure at this interim result uses $Z_2$ alone, it cannot use the ordinary 1.645 cutoff and still preserve the original conditional error: that would give conditional rejection probability 0.05. Instead it must require $Z_2>2.326$, equivalently $p_2<0.0100$, at this particular interim value. This calculation illustrates the principle; a full adaptive rule must calibrate the boundary for every relevant interim result, not just this one.

Key distinction: A conditionally valid stage-two p-value permits the stage-two design to depend on interim data without invalidating that p-value. It does not, by itself, validate data-dependent weights, a changed combination function, or a new cutoff. Those changes require a combination rule with its own null calibration or a rule that preserves the original conditional error.

Checking the Design and Comparing Approaches

A practical review should trace the information flow. Record exactly which data determine the stage-two information, whether any stage-two outcomes were examined first, and which null distribution is used for $p_2$ at every possible selected design. Then check whether the combination weights and rejection boundary are fixed or depend on interim results. A correct conditional p-value cannot compensate for a combination rule whose calibration is wrong.

Approach	What can vary?	Calibration requirement
Fixed-weight combination	Stage-two information can be selected from interim data	Stagewise p-values must be valid, with the stage-two p-value valid conditional on the interim information
Information-based adaptive weights	Stage-two information and the weights may depend on interim data	Do not automatically use a fixed-weight cutoff; justify the null distribution or recalibrate the rule
Conditional-error approach	The remaining design or test may change	At each interim state, the revised procedure must not exceed the original conditional rejection probability under the null

The principal diagnostics are failures of timing or calibration: stage-two information chosen after examining stage-two outcomes, a test whose null distribution changes with the selected design but is treated as fixed, or weights that depend on interim data but are combined using an unadjusted fixed-design cutoff. Also check whether the target hypothesis is composite. Conditional validity must hold under the null values relevant to the intended Type I error guarantee; validity only at an unjustified single parameter value is insufficient.

This approach has limits. Conditional validity can be difficult to establish when the stage-two test uses estimated nuisance parameters, complex dependence, or additional unplanned monitoring. Moreover, holding weights fixed may not be the most powerful choice for every adaptation. The appropriate comparison is not “flexible versus inflexible” alone: it is a simple, valid fixed-weight rule versus a potentially more tailored rule whose conditional or unconditional calibration has been demonstrated.

Common Mistakes

“The sample size is adaptive, so the stage-two p-value is invalid.” Not necessarily. If the information is selected from interim data and the stage-two p-value is valid conditional on that selection, the p-value can remain valid.
“The stage-two p-value is valid, so any combination rule is valid.” No. The random-weight example has a valid $p_2$ and still has Type I error 0.0975 at a nominal 0.05 level.
Using the planned information instead of the selected design in the stage-two calibration. The p-value must correspond to the actual selected design and its null distribution.
Ignoring later stage-two monitoring. If stage-two outcomes trigger further stopping or selection, a fixed-sample p-value may no longer be conditionally valid.

Check Your Understanding

Answer each question, then compare with the brief solutions below.

What does conditional validity of $p_2$ mean mathematically?
Interim data select the stage-two sample size, but no stage-two outcomes are inspected before that choice. Name two further conditions needed for a valid stage-two p-value.
Why can fixed inverse-normal weights support changing stage-two information when the stage-two p-value remains conditionally valid?
At $Z_1=0$, the original equal-weight test rejects when $Z_C>1.645$. What stage-two z-score cutoff preserves its conditional error if the revised test uses $Z_2$ alone?
Explain why a valid conditional stage-two p-value does not automatically justify information-based random weights.

Solutions. (1) For every $u\in[0,1]$, $P(p_2\le u\mid\mathcal{F}_1)\le u$. (2) The stage-two test must have valid null calibration at the selected sample size, and the new stage-two data must have the required conditional null distribution given the interim information; independence of new observations is a common sufficient condition. (3) The fixed weights preserve the standard-normal combination calibration when the stagewise p-values have the stated null distributions and dependence structure. The selected information is included in the conditional calibration of $p_2$. (4) $Z_2>1.645\sqrt{2}=2.326$, giving a conditional rejection probability of about 0.0100. (5) Random weights can depend on the first-stage statistic, so the weighted sum need not have the standard normal distribution used by the fixed-weight cutoff; the full adaptive rule needs valid recalibration.