Calibrating a Combination-Test Rejection Threshold

Why a Rejection Threshold Must Be Calibrated

A combination test maps stagewise evidence to a single statistic and rejects the null hypothesis when that statistic enters a specified rejection region. Choosing the function is not enough: the threshold must also be chosen so that the probability of rejection under the null is controlled at the intended significance level \(\alpha\). A threshold that looks plausible on the scale of the statistic may yield a much larger or smaller type I error rate than intended.

The calibration calculation uses the null distribution of the combination statistic. That distribution depends on the null behavior of the stagewise p-values and, in particular, on their joint distribution. If the stages are independent and each p-value is exactly uniform under the null, the joint null distribution is known. If the p-values are dependent, discrete, or only conservatively valid, the calibration must account for that fact rather than automatically using the independent-uniform reference.

The type I error rate is the probability of rejecting \(H_0\) when \(H_0\) is true. Calibration is a statement about that probability under the specified null model; it is not a claim that a particular observed result has probability \(\alpha\) of being wrong. Throughout the examples, \(\alpha=0.05\), and smaller values of the combination statistic indicate stronger evidence, so the rejection rule is \(T\leq c\).

From a Null Distribution to a Threshold

Suppose the stagewise p-values are independent and exactly uniform on \([0,1]\) under the null. Then their joint null density is 1 on the unit cube. For a specified combination function \(T=g(P_1,\ldots,P_K)\), the probability of rejection at a lower-tail threshold \(c\) is the volume of the part of that cube where \(g(p_1,\ldots,p_K)\leq c\). Calibration means choosing \(c\) so that this volume equals \(\alpha\), when possible.

Here \(\mathbf{1}\{\cdot\}\) equals 1 when the condition in braces holds and 0 otherwise. The equation is a direct expression of the test’s type I error rate. It also makes clear why a threshold cannot be transferred from one combination function to another without checking the null distribution: changing \(g\) changes the region being integrated.

If the distribution of \(T\) is continuous, solving the tail equation gives an exact threshold. If it has point masses, one may be unable to attain exactly \(\alpha\) with a nonrandomized rule. A conservative threshold instead ensures the rejection probability is no greater than \(\alpha\). A randomized decision at the boundary can sometimes attain equality, but it must be part of the prespecified testing procedure.

Justifying Conservative Calibration

The independent-uniform calculation is exact only when its assumptions hold. For a broader validity argument, suppose each stagewise p-value is valid under the null, meaning \(P_{H_0}(P_i\leq t)\leq t\) for every \(t\in[0,1]\). This permits conservative p-values, whose null distributions may put less probability than a uniform distribution in small-p-value regions.

Validity of the individual p-values alone does not determine the joint distribution for an arbitrary combination function. A useful sufficient condition is that the p-values are independent and the statistic is nonincreasing in each p-value: making any one p-value smaller cannot make the statistic less persuasive. In that case, independent valid p-values are at least as large, in the stochastic sense, as independent uniform p-values. The combination statistic is consequently no more likely to enter a lower-tail rejection region than it would be under independent uniforms. A threshold calibrated using the independent-uniform reference therefore controls the type I error rate at or below \(\alpha\).

The argument can be seen by coupling each valid p-value with a uniform random variable. A valid p-value has a distribution that is stochastically no smaller than uniform, so the p-value can be coupled to be at least as large as its uniform counterpart. Applying a statistic that is nonincreasing in each coordinate then makes the statistic no smaller than its uniform-reference counterpart. Therefore its probability of being at or below any fixed lower-tail threshold cannot increase.

When the Reference Distribution Is Not Simple

For a more complicated combination function, an analytic distribution may be difficult to derive. The same calibration principle still applies: evaluate the statistic under its null distribution and choose a threshold with the desired null tail probability. Numerical integration or simulation from a justified null model can estimate that distribution. With simulation, the chosen threshold is itself subject to Monte Carlo error; more simulated null trials make the estimated tail probability more precise, but do not repair a misspecified null model.

In an adaptive trial, later-stage data collection or other design choices may depend on earlier information. Calibration must reflect the resulting joint null behavior of all stagewise p-values. If a later-stage p-value is conditionally valid given the information used to make the adaptation, that conditional validity can support the required null reference distribution. It is not enough to assert that the p-values are individually uniform while ignoring dependence or conditioning.

A practical check is to apply the full prespecified procedure under the null, including its adaptation rules, and verify that the simulated rejection frequency is near the target within expected simulation error. A frequency that is systematically too high can indicate a wrong joint null model, a coding error, or a threshold taken from the wrong tail. A frequency that is too low may reflect conservative p-values, a conservative threshold, or unnecessary loss of information.

Comparisons, Diagnostics, and Limitations

Combination-test calibration differs from setting separate stagewise cutoffs and treating either one as sufficient for rejection. The latter approach needs its own multiplicity control; it does not automatically produce an overall type I error rate of \(\alpha\). It also differs from a test based on pooled trial data: a combination test calibrates the distribution of its prespecified stagewise statistic, whereas a pooled-data test uses a statistic and null distribution derived from the pooled analysis. Neither approach is universally preferable; the choice depends on the design, assumptions, and inferential objective.

Calibration is only as reliable as its inputs. Important failure modes include using a threshold derived for independent p-values when the p-values are dependent, applying a continuous-distribution calculation to a discrete statistic without addressing ties, and using a reference distribution that omits an adaptive design feature. A threshold selected after looking at the observed results can also invalidate the nominal error control: the function and threshold should be specified without using the trial’s accumulating evidence to choose a favorable cutoff.

A common mistake is to say, “Each stage uses a 0.05 p-value, so the combination test has a 0.05 type I error rate.” That reasoning ignores the event defining overall rejection and the joint null distribution of the stagewise p-values. For example, rejecting when either of two independent p-values is at most 0.05 has type I error \(1-(1-0.05)^2=0.0975\), not 0.05. The correction is to calculate the probability of the complete rejection event under the null and choose its threshold accordingly.