Conditional Validity After Data-Dependent Stage Changes

Why Conditional Validity Matters

An adaptive trial may use first-stage results to change the second-stage sample size, select an endpoint, or choose which treatment comparison to continue. Such a change makes the later-stage analysis depend on earlier data. The key requirement is not that the two stagewise p-values be independent without qualification. It is that the later-stage p-value remain valid conditional on the information used to make the change.

This condition connects a practical design rule to the combination-test principle. If the second-stage p-value is conditionally valid, a fixed-weight combination test can control the global null rejection probability even when the second-stage procedure was chosen adaptively. But conditional validity does not automatically justify changing the combination weights after seeing the first-stage results; that changes the calibration problem.

Definition: A p-value $P_2$ is conditionally super-uniform under a null hypothesis if, for every $u\in[0,1]$, $$ P(P_2\leq u\mid\mathcal{F}_1,D_2)\leq u $$ under that null. Here $\mathcal{F}_1$ denotes the information available at the end of stage 1, and $D_2$ denotes the second-stage design choice, which is made using that information. The condition must hold for every allowed design choice and, for a composite null, for every parameter value in the null.

Since $D_2$ is determined by $\mathcal{F}_1$, conditioning on both emphasizes that the p-value is valid after the actual adaptation. It does not require the design choice to be independent of the first-stage data. It does require the conditional analysis to account for the choice: for example, if an endpoint is selected from several candidates, the reported p-value must account for that selection unless the conditional procedure itself already does so.

How Conditional Validity Protects a Combination Test

Consider two stages. Let $P_1$ be a valid first-stage p-value and let $P_2$ be conditionally super-uniform given $\mathcal{F}_1,D_2$. First consider a fixed-weight inverse-normal combination test. Write $Z_i=\Phi^{-1}(1-P_i)$, where $\Phi$ is the standard normal cumulative distribution function. For prespecified weights $w_1,w_2>0$ with $w_1^2+w_2^2=1$, the combined statistic is $C=w_1Z_1+w_2Z_2$. The critical value $c_\alpha$ is calibrated so the test has level $\alpha$ when the stagewise p-values are independent uniform variables.

For any observed first-stage information, rejection requires the second-stage p-value to be below a threshold determined by the first-stage result. Conditional super-uniformity bounds the probability of crossing that threshold by the threshold itself. Because the threshold is a nonincreasing function of $P_1$, a super-uniform $P_1$ is no more likely to produce rejection than a uniform p-value. Thus the independent-uniform calibration is conservative or exact under the stated conditions.

P(\text{reject}\mid\mathcal{F}_1,D_2) \leq 1-\Phi\!\left(\frac{c_\alpha-w_1\Phi^{-1}(1-P_1)}{w_2}\right)

The expression on the right is the threshold for $P_2$. As $P_1$ increases, its inverse-normal evidence decreases, the required second-stage evidence increases, and the threshold for $P_2$ decreases. This monotonicity is essential: it lets first-stage super-uniformity bound the overall rejection probability by the independent-uniform reference probability.

The same reasoning applies to Fisher combination. For two p-values, the Fisher statistic is $T=-2\log(P_1P_2)$, and the usual level-$\alpha$ critical value $c_\alpha$ is obtained from the $\chi^2$ distribution with 4 degrees of freedom under independent uniform p-values. Conditional on $P_1$, rejection is the event $P_2<\exp(-c_\alpha/2)/P_1$, with the threshold capped at 1. Conditional super-uniformity bounds that event just as it bounds any lower-tail event for $P_2$. The resulting rejection bound is nonincreasing in $P_1$, so the same first-stage argument applies.

Key result: For the two-stage setting above, a combination test calibrated under independent uniform stagewise p-values controls type I error when each stagewise p-value is super-uniform under the null conditional on the information and design choices available before that stage, provided that, conditional on $P_1=p$, rejection is a lower-tail event in $P_2$ and its conditional rejection-probability bound is nonincreasing in $p$. The combination rule and its weights must be fixed independently of the stagewise results, or separately calibrated for their adaptive selection.

Worked Examples

Worked Example: Fisher Combination After an Adaptive Sample-Size Change

Suppose a hypothetical trial uses a one-sided Fisher combination test at level $0.05$. After stage 1, the team may choose one of two second-stage sample sizes using $\mathcal{F}_1$. Assume the second-stage test is constructed so that, under the null and conditional on either chosen sample size and all stage-1 information, $P_2$ is super-uniform. The first-stage p-value is also super-uniform. The Fisher critical value is approximately $9.488$, the 95th percentile of a $\chi^2$ distribution with 4 degrees of freedom.

Suppose the observed first-stage p-value is $P_1=0.20$. The Fisher statistic exceeds its critical value when

-2\log(0.20P_2)>9.488 \quad\Longleftrightarrow\quad P_2<\frac{\exp(-9.488/2)}{0.20}.

Since $\exp(-4.744)\approx0.00870$, the conditional rejection threshold is $0.00870/0.20\approx0.0435$. Conditional on the first-stage information and the selected sample size, the probability of rejection is therefore at most $0.0435$, by conditional super-uniformity. This calculation describes the conditional bound for this observed first-stage result; it is not itself the overall type I error rate. Integrating the corresponding bound over all first-stage outcomes, using the valid first-stage p-value, gives the overall level-$0.05$ control associated with the Fisher calibration.

The adaptation is permitted because the second-stage p-value is valid after conditioning on the chosen sample size. A sample-size-dependent test that did not have this property would not be justified by this argument.

Worked Example: Conditional Validity After Selecting an Endpoint

Suppose a hypothetical study considers two candidate endpoints at stage 2 and, based on stage-1 data, reports the smaller of their one-sided p-values, $P_{21}$ and $P_{22}$. Assume each candidate p-value is conditionally super-uniform under the null given $\mathcal{F}_1$. A Bonferroni-adjusted p-value is

P_2=\min\{1,\,2\min(P_{21},P_{22})\}.

For $0\leq u<1$, the event equality and union bound give

\begin{aligned} P(P_2\leq u\mid\mathcal{F}_1) &=P\left(\min(P_{21},P_{22})\leq \frac{u}{2}\,\middle|\,\mathcal{F}_1\right)\\ &\leq P(P_{21}\leq u/2\mid\mathcal{F}_1) +P(P_{22}\leq u/2\mid\mathcal{F}_1)\\ &\leq \frac{u}{2}+\frac{u}{2}=u. \end{aligned}

The event equality follows because $u<1$: then $P_2\leq u$ is equivalent to $2\min(P_{21},P_{22})\leq u$. At $u=1$, that equivalence need not hold, since $P_2\leq1$ always. Instead, directly $P(P_2\leq1\mid\mathcal{F}_1)=1\leq1$. Thus $P_2$ is conditionally super-uniform over the full interval $u\in[0,1]$, and it can serve as the second-stage input to a combination test. Reporting the unadjusted minimum would not, in general, have this conditional validity.

Worked Example: A Data-Dependent Weight Can Break Calibration

Let $Z_1,Z_2$ be independent standard normal statistics under a hypothetical global null, and consider a one-sided 5% test with critical value $1.645$. If the first-stage result is positive, choose weights $w_1=1,w_2=0$; otherwise choose $w_1=0,w_2=1$. The weights always satisfy $w_1^2+w_2^2=1$, but they depend on the first-stage result. The resulting statistic is $C=Z_1$ when $Z_1>0$, and $C=Z_2$ when $Z_1\leq0$.

The probability of rejection is

\begin{aligned} P(C>1.645) &=P(Z_1>1.645)+P(Z_1\leq0,\ Z_2>1.645)\\ &=0.05+(0.50)(0.05)=0.075. \end{aligned}

The test rejects 7.5% of the time, not 5%. Each underlying stage statistic is valid, but the weights select evidence using $Z_1$, so the fixed-weight normal calibration no longer applies. The remedy is not simply to assert conditional validity of $P_2$; the adaptive weighting rule itself must be incorporated into a valid calibration, or the weights must be fixed in advance.

Assumptions, Diagnostics, and Limits

The conditional-validity argument depends on several explicit conditions. The null hypothesis must be the one for which each stagewise p-value is claimed valid; the stage-2 test must remain valid for every allowed adaptation and every null parameter value; and the stage-1 p-value must itself be super-uniform. In addition, the combination rule and its reference calibration must match the weights and stage structure actually used. If conditional validity is only approximate, for example from a large-sample approximation, the resulting error control is approximate as well.

Useful design diagnostics include checking that the stage-2 analysis is specified for each possible adaptive choice, confirming whether selection among endpoints or treatments is multiplicity-adjusted, and verifying that the final combination weights are genuinely prespecified. Simulation under representative null parameter values can reveal inflated rejection rates, but it does not replace a justification of conditional validity. A test that is well calibrated on average across design choices may still fail conditionally after a particular choice, which is exactly where the combination argument needs its guarantee.

Conditional validity is a powerful but limited guarantee. It permits adaptation of later-stage procedures when their p-values remain valid after the adaptation. It does not make arbitrary data-dependent choices harmless, and it does not license treating dependent p-values as independent without an argument. In particular, changing sample size may be compatible with the result if the stagewise test remains conditionally valid, while selecting an endpoint without adjustment may not be. Data-dependent weights require a calibration that reflects their selection rule; the usual fixed-weight reference distribution cannot simply be reused.

Common mistake: “Each stage has a valid p-value, so any combination rule remains valid after adaptation.” This is too broad. Later-stage validity must hold conditional on the information and design choices, and the combination rule must have the stated calibration. In particular, validity of the stagewise p-values alone does not justify weights chosen after looking at earlier results.

Check Your Understanding

Answer each question, then compare with the brief solutions below.

State the conditional super-uniformity requirement for a stage-2 p-value.
For a Fisher test with critical value $c$ and observed $P_1=p$, what threshold must $P_2$ cross to reject?
Why does selecting the smaller of two unadjusted endpoint p-values generally fail the conditional-validity requirement?
Does changing the second-stage sample size automatically invalidate a combination test? State the necessary qualification.
Why is a prespecified fixed-weight inverse-normal calibration not automatically valid when weights depend on $Z_1$?

Solutions. (1) For every $u\in[0,1]$, require $P(P_2\leq u\mid\mathcal{F}_1,D_2)\leq u$ under the null. (2) Rejection occurs when $P_2<\exp(-c/2)/p$, with the threshold effectively capped at 1. (3) Even if each endpoint p-value is conditionally valid, taking their minimum increases the chance of a small reported value; a valid adjustment such as the Bonferroni adjustment is needed. (4) No. It is permitted by this argument if the stage-2 p-value remains null-valid conditional on the first-stage information and the selected sample size, and the combination calibration remains appropriate. (5) Data-dependent weights can select a more favorable combination of stage evidence, changing the null distribution of the combined statistic; the fixed-weight reference calibration then need not control the stated level.

Conditional Validity After Data-Dependent Stage Changes

What You'll Learn