Stagewise Error Rates and Final Test Calibration

Why the Final Cutoff Is Not a Stagewise Cutoff

A two-stage adaptive trial produces evidence at two points in time. Each stage may have its own nominal significance threshold, but those thresholds do not automatically define the type I error rate of the final test. The final decision depends on the rule applied to the evidence across stages: for example, whether success means crossing a cutoff at either stage, or whether the two stagewise p-values are combined and compared with one final cutoff.

This distinction matters because type I error belongs to the whole decision rule. A test that rejects whenever either of two independent stagewise p-values is below 0.05 does not have an overall type I error of 0.05. By contrast, a properly calibrated combination test uses a single final threshold chosen to control the probability of rejecting the global null across the two stages.

Two-Stage Inverse-Normal Calibration

Consider a one-sided test with null hypothesis \(H_0\) and alternative hypothesis \(H_a\) in the prespecified direction. Let \(p_1\) and \(p_2\) be the stagewise p-values. For the exact normal calibration below, assume that under \(H_0\), each \(p_i\) has the continuous Uniform(0,1) distribution and that \(p_1\) and \(p_2\) are independent. The test procedures and direction must be specified so that small p-values consistently indicate evidence for \(H_a\). These assumptions commonly correspond to valid continuous stagewise tests based on independent data from the two stages.

Transform each stagewise p-value to a normal score using \(Z_i=\Phi^{-1}(1-p_i)\), where \(\Phi\) is the standard normal cumulative distribution function. Let \(w_1\) and \(w_2\) be fixed, prespecified nonnegative weights satisfying \(w_1^2+w_2^2=1\). The two-stage inverse-normal combination statistic is \(Z_{\mathrm{comb}}=w_1Z_1+w_2Z_2\).

The normalization of the weights is essential to this reference distribution. The weights may reflect a chosen combination rule, but once selected they must be fixed independently of the observed stage results for this calibration. The final cutoff is determined by the overall \(\alpha\), not by applying each \(w_i\) or each stagewise threshold as a separate significance level.

Why the Calibration Works

The key facts follow directly from the assumed null distributions. If \(p_i\) is Uniform(0,1), then for any real \(z\),

Thus \(Z_i\) is standard normal. Independence of the p-values implies independence of their transformed scores. A weighted sum of independent normal variables is normal; its mean is zero and its variance is \(w_1^2+w_2^2=1\). Hence \(Z_{\mathrm{comb}}\sim N(0,1)\) under \(H_0\). The upper-tail probability at \(z_{1-\alpha}\) is \(\alpha\), which gives the stated overall type I error.

This argument also shows why exact uniformity is not a cosmetic assumption. A discrete or conservative p-value is generally not exactly Uniform(0,1); its transformed score need not be standard normal. If independent stagewise p-values are only superuniform—that is, \(P(p_i\leq t)\leq t\) for every \(t\) between zero and one—the normal-score combination can be conservative under suitable independent-stage conditions, but the exact standard-normal claim no longer follows from the argument above. Its actual null calibration should be established for the p-value distributions and decision rule being used.

Stagewise Thresholds Define Different Rules

To see why threshold labels alone do not determine overall error, consider a different rule: reject if either of two independent, exactly uniform stagewise p-values is at most 0.05. Under \(H_0\), the probability of no rejection is the probability that both exceed 0.05. Independence gives

So applying a 0.05 cutoff at each stage and rejecting at either one yields a 0.0975 overall type I error in this example, not 0.05. One alternative is to use 0.025 at each stage with the same “either stage” rule. Its overall type I error is \(1-(1-0.025)^2=0.049375\), which is slightly below 0.05. This is a separate multiple-testing rule; its thresholds should not be confused with the final cutoff of a combination statistic.

A trial could also have a formal interim efficacy boundary that permits early rejection and a later combination test for participants who continue. In that case, the global decision rule includes both the early-rejection event and the later rejection event. The final combination cutoff alone does not automatically calibrate the probability of rejection across both opportunities. The early and final parts must be calibrated as one rule.

Applying the Rule Reliably

A calibrated decision requires more than plugging two p-values into a formula. Before analysis, specify the one-sided direction, the stagewise tests, the combination weights, the overall \(\alpha\), and whether any separate interim rejection rule is allowed. Check that the two stagewise p-values have the null properties assumed for the chosen calibration and that their dependence structure supports the reference distribution.

Diagnostics include checking that the weights square to one, that the final cutoff matches the intended one-sided overall level, and that the combination statistic and combined p-value lead to the same decision. If a stagewise p-value is discrete, heavily rounded, or computed from data that overlap with another stage, the exact independent-uniform derivation may not apply. In those cases, use a calibration justified for the actual null distributions and dependence rather than treating the normal reference as automatic.

Common Mistakes and Limits

• Calling each stage’s cutoff the overall level. A 0.05 cutoff at each of two opportunities does not generally produce a 0.05 overall error rate. State the complete rejection rule and evaluate its null rejection probability.
• Comparing the combined statistic with a stagewise cutoff. The statistic \(Z_{\mathrm{comb}}\) is compared with \(z_{1-\alpha}\) for the chosen overall level, not with a cutoff selected independently for either stage.
• Assuming every valid p-value is exactly uniform. Validity may mean only that a p-value is conservative. Exact normal-score calibration requires exact uniformity and independence under the stated setup; other cases require appropriate null calibration.
• Changing weights after seeing the results. The standard calibration above assumes prespecified weights. Choosing them in response to observed evidence changes the decision procedure and is not covered by this derivation.

The inverse-normal rule is one way to combine stagewise evidence, not a universal description of every adaptive trial. Its exact cutoff follows from its specified assumptions and statistic. Alternative combination rules can have different statistics and reference distributions, so their final thresholds must be calibrated for those rules rather than borrowed from the inverse-normal test.