Discrete Stagewise P-Values and Calibration

Why discreteness matters when combining evidence

Combination tests often use stagewise p-values as inputs to a final statistic. A familiar calibration argument treats each stagewise p-value as uniform under the null hypothesis. That argument is exact for continuous test statistics under suitable conditions, but many clinical trial outcomes are discrete: event counts, binary responses, and rank statistics with ties can yield only a finite set of possible p-values. Exact tail probabilities then tend to be conservative: small p-values occur no more often than their nominal probabilities permit.

This distinction matters because a combination statistic may be calibrated against the distribution it would have if its inputs were independent uniform random variables. If actual stagewise p-values are independent but discrete and conservative, the nominal rejection threshold will generally control the type I error rate at or below its target, rather than attain it. This is a size issue, not a reason to replace valid p-values with smaller ones. Calibration can account for the discrete null distribution when attaining the nominal size is important.

For example, a nonrandomized one-sided exact p-value may be defined as the null probability of an outcome at least as extreme as the observed outcome, with ties included in the tail. Because outcomes arrive in jumps, a tail probability may move from above a chosen cutoff to below it without ever equalling it. The resulting test rejects at a rate no greater than its nominal level. For a composite null, validity requires the super-uniform inequality under every null parameter value, or under the relevant supremum over those values.

Effect on the inverse normal combination test

For two stages, let \(P_1\) and \(P_2\) be stagewise p-values, and use the upper-tail normal score \(Z_i=\Phi^{-1}(1-P_i)\), where \(\Phi\) is the standard normal cumulative distribution function. With positive weights \(w_1,w_2\) satisfying \(w_1^2+w_2^2=1\), the inverse normal statistic is \(C=w_1Z_1+w_2Z_2\). The nominal continuous-reference test rejects for \(C>z_{1-\alpha}\), where \(z_{1-\alpha}=\Phi^{-1}(1-\alpha)\).

If the p-values are independent and uniform under \(H_0\), then \(Z_1,Z_2\) are independent standard normal variables and \(C\) is standard normal. If instead each p-value is independent and super-uniform, the transformed scores are stochastically no larger than standard normal scores. Since the weights are positive, the weighted sum is likewise stochastically no larger than the standard normal reference. Thus the continuous-reference cutoff controls size, but may be conservative. The argument depends on independence under the null; it does not establish validity for arbitrary dependence.

The key step can be justified by a quantile coupling. Let \(U_i\) be independent uniform random variables on \([0,1]\). Super-uniformity means that a draw from the distribution of \(P_i\) is stochastically at least as large as a uniform draw. Using the same uniform variable to construct both draws through their quantile functions gives \(P_i\geq U_i\) in this coupling. Since \(p\mapsto\Phi^{-1}(1-p)\) is decreasing, the corresponding normal score from \(P_i\) is no larger than the score from \(U_i\). The latter scores are independent standard normals. Multiplying by positive weights and adding preserves the inequality, which proves the stated size bound.

Calibration with a discrete null distribution

When the trial design and null distribution are sufficiently specified, the combination statistic can instead be calibrated directly. Enumerate or otherwise compute its null distribution, then choose a rejection region whose probability under \(H_0\) is no greater than \(\alpha\). Because the distribution is discrete, the attainable rejection probabilities may skip over \(\alpha\). A nonrandomized test should use the largest available rejection region that stays within the target; it may remain conservative. Randomization at a boundary can attain the target exactly, but must be specified in advance and is not always desirable in clinical practice.

Direct calibration is only as reliable as the null distribution used to obtain it. Enumeration must reflect the actual stagewise test definitions, sample sizes, allocation, nuisance parameters where relevant, and any adaptation that affects the second stage. If adaptation changes the stage-two design, its p-value must still be valid under the appropriate null conditional on the information used to adapt the design. Independence or conditional independence assumptions used by the calculation must also hold. A nominal cutoff cannot repair a misspecified null distribution.

Consequences for Fisher calibration

The same discreteness issue applies when a Fisher combination test is compared with its continuous-uniform reference distribution. For two independent uniform p-values, the statistic \(F=-2\{\ln(P_1)+\ln(P_2)\}\) has a chi-square reference distribution with four degrees of freedom. If the stagewise p-values are instead independent and super-uniform, the decreasing transformation \(-2\ln(p)\) gives contributions stochastically no larger than those from uniform p-values. Consequently, use of the continuous chi-square cutoff is conservative or exact, under those conditions. This statement concerns size calibration; it does not require treating the continuous reference as the exact discrete null distribution.

Diagnostics, alternatives, and limitations

Before applying a continuous-reference calibration, check whether the stagewise p-values are valid under the relevant null and whether the assumptions about their joint distribution are justified. Useful diagnostics include enumerating the possible stagewise outcomes in a small design, comparing attainable p-values with their null probabilities, and simulating the full null design—including adaptation—when exact enumeration is impractical. Simulation estimates size with Monte Carlo error; it is not a proof of validity, so report the number of replicates and uncertainty when relying on it.

The main approaches differ in what they prioritize:

A common mistake is to reason, “Exact p-values are conservative, so replacing them by mid-p values will make the combination test less conservative without consequence.” That conclusion is not justified: a mid-p value can have \(P(P\leq u)>u\) for some \(u\), so it need not be a valid p-value. The stochastic-order argument for the inverse normal and Fisher calibrations then no longer applies. Any alternative input or calibration requires its own type I error justification.

A second limitation is that discreteness is not the only source of size error. Dependence between stages, invalid stagewise tests, unaccounted selection, or use of a reference distribution that does not match the design can all undermine calibration. Conversely, a discrete p-value does not by itself imply that a test is unusable: a conservative reference test can be valid, and a design-specific calibration can be used when justified. The choice should be made with the intended error-rate guarantee and the trial’s operational constraints in view.