Proving Type I Error Control for a Combination Test

Why a Combination-Test Proof Needs More Than a Formula

In a seamless phase 2/3 trial, phase 2 data may help determine which treatment proceeds, while data from both phases contribute to the confirmatory test. A prespecified combination function provides a way to combine stagewise evidence. But writing down that function does not, by itself, establish Type I error control. The proof depends on the validity of the stagewise p-values under the null and on how selection affects their distributions.

The central argument is that valid stagewise evidence cannot, under the stated conditions, make the combined statistic more extreme under the null than its reference distribution allows. Selection matters because a treatment chosen for promising results may have unusually small unadjusted phase 2 p-values. The proof must account for that feature rather than assume it away.

For the inverse-normal combination test, each stagewise p-value is transformed to a one-sided standard-normal score, \(Z_k=\Phi^{-1}(1-p_k)\), where \(\Phi\) is the standard normal cumulative distribution function. For two stages, the combined statistic is

The combined p-value is \(p_{\mathrm{comb}}=1-\Phi(Z_{\mathrm{comb}})\), and the test rejects when \(p_{\mathrm{comb}}\leq\alpha\), equivalently when \(Z_{\mathrm{comb}}\geq\Phi^{-1}(1-\alpha)\). The weights must be fixed in advance for the stated calibration. Their squared sum is one so that a weighted sum of independent standard normal scores has variance one.

Conditions for the Error-Control Argument

Consider one confirmatory null hypothesis for the treatment that proceeds. The relevant assumptions are as follows.

• Stagewise validity. Under this null, \(p_1\) is super-uniform. It must remain valid for the treatment actually selected, not merely for a treatment considered before selection.
• Conditional validity of later evidence. Given all phase 2 data used for selection and the transition decision, \(p_2\) is super-uniform under the null for the selected treatment. In particular, phase 2 evidence must not be made more favorable by conditioning on the selection rule.
• Prespecified combination rule. The combination function, weights, direction of testing, and rejection threshold are specified without using the observed p-values to choose a more favorable rule.

Conditional validity of \(p_2\) is a common consequence of using independent increments: conditional on the earlier data, the later-stage test retains its null calibration. The assumptions do not say that \(p_1\) must be uniform conditional on every detail of phase 2 selection. Instead, its required validity must be established for the selected hypothesis under the actual selection procedure. As covered in the earlier material on valid stagewise p-values after treatment selection, this may require a selection-adjusted p-value or a selection rule that is independent of the stagewise test statistic.

A key limitation follows immediately: independent phase 3 data do not repair an invalid selected phase 2 p-value automatically. If selection picks the treatment with the smallest unadjusted phase 2 p-value, that selected p-value can be anti-conservative under the null. The combination-test proof must not use the super-uniformity of that p-value unless it has been justified.

Why the Inverse-Normal Test Controls Type I Error

First suppose both p-values are independent uniform random variables under the null. Then \(Z_1\) and \(Z_2\) are independent standard normal random variables. Their weighted sum has mean zero and variance \(w_1^2+w_2^2=1\), so \(Z_{\mathrm{comb}}\) is standard normal. Thus the probability of exceeding \(\Phi^{-1}(1-\alpha)\) is exactly \(\alpha\).

The same conclusion becomes an upper bound when stagewise p-values are super-uniform rather than exactly uniform. A super-uniform p-value produces a normal score that is stochastically no larger than a standard normal score: for any real \(z\),

For the second stage, this bound holds conditionally on the phase 2 data and the selection decision. To see the full argument, fix a value of \(Z_1\) and let \(c=\Phi^{-1}(1-\alpha)\). If \(w_2>0\), rejection requires

Conditional super-uniformity bounds the probability of this event by the corresponding standard-normal upper tail. That bound is an increasing function of \(Z_1\): larger first-stage scores can only increase the conditional chance of rejection. Since the marginal distribution of \(Z_1\) is stochastically no larger than a standard normal distribution, averaging the bound over \(Z_1\) gives no more rejection probability than if both scores were independent standard normals. In that reference case the weighted sum is standard normal, giving probability \(\alpha\). Therefore the actual null rejection probability is at most \(\alpha\). If one weight is zero, the test reduces to the other stage’s valid test, and the same control follows directly.

Fisher Combination and Its Calibration

A second common combination function is Fisher’s statistic,

If the \(K\) p-values are independent and exactly uniform under the null, then each \(-2\log(p_k)\) has a chi-square distribution with 2 degrees of freedom, and their sum has a chi-square distribution with \(2K\) degrees of freedom. The combined p-value is the upper-tail probability from that chi-square distribution. Under independent super-uniform stagewise p-values, the statistic is stochastically no larger than its uniform-p-value reference counterpart, so the same upper-tail calibration is conservative. For adaptive use, the validity argument must still accommodate selection and dependence; for example, conditional validity of each later p-value given earlier data supports the sequential argument.

Selection and the Scope of the Claim

The following hypothetical example illustrates one way selection can be compatible with the proof. Suppose two candidate treatments are under consideration. A prespecified phase 2 screening measure, independent under the null of each treatment’s stagewise test statistic, determines which treatment proceeds. For the selected treatment, the stage 1 p-value is therefore uniform conditional on which treatment was selected, and the later-stage p-value is conditionally valid given the screening and phase 2 data. The selected p-values are \(p_1=0.10\) and \(p_2=0.02\), and the inverse-normal test uses equal weights \(w_1=w_2=1/\sqrt{2}\).

The contrast is important. If the phase 2 rule instead selects whichever treatment has the smallest unadjusted phase 2 p-value, that p-value’s distribution for the selected treatment is generally not uniform under the null. One cannot apply the theorem by simply labeling it \(p_1\). A valid selection-adjusted procedure or another justification of selected-stage validity is needed before using the combination-test guarantee.

Diagnostics, Alternatives, and Limitations

A practical audit should check the assumptions that enter the proof, not just whether the arithmetic is correct. Confirm that the weight vector was fixed before examining the stagewise p-values; verify how the stagewise p-value for the selected treatment was calibrated; and check that later-stage validity holds conditional on the information used at the transition. A discrepancy between the planned and implemented selection rule, or reuse of data in a way that compromises conditional validity, requires renewed justification.

Combination tests differ from simple stagewise alpha allocation. An alpha-allocation rule assigns rejection thresholds to stages, whereas a combination test maps the stagewise p-values through one prespecified function and applies a single calibrated threshold. Fisher’s test can respond strongly when several stages provide small p-values; the inverse-normal test offers explicit weights that control how strongly each stage contributes. Neither choice removes the need to justify the stagewise p-values. Conditional-error methods provide another framework for preserving a specified error bound after adaptations, but they are not a substitute for checking the assumptions of the chosen combination function.

The theorem is about a single null hypothesis with the required validity conditions. It does not, by itself, control error across several candidate treatments, endpoints, or claims. Nor does it establish that the estimand is appropriate, that the trial is adequately powered, or that the selection rule is clinically sensible. Those questions require their own prespecified design and analysis arguments.

Common Mistakes

• Assuming later-stage independence is sufficient. Independent phase 3 data can yield a conditionally valid \(p_2\), but an invalid selected \(p_1\) can still undermine the combined test. Justify both stagewise inputs.
• Treating a selected p-value as automatically uniform. Selection based on the same evidence can favor small p-values. Establish validity for the selected treatment using the actual selection procedure.
• Changing weights after seeing results. Choosing weights to emphasize the more favorable stage changes the test. Use prespecified weights or a separately justified procedure.
• Confusing a controlled error rate with the observed p-value. Type I error control describes a repeated-trial probability under the null; it is not the probability that the null is true for this trial.