Deriving the Combination Test Principle

Why a Combination Test Needs a Calibration Argument

In a multi-stage trial, each stage can provide a p-value for the same null hypothesis. A combination test turns those stagewise p-values into one final decision. The key validity question is not simply whether each p-value is valid on its own. It is whether the rule that combines them has a rejection probability no greater than the prespecified significance level under the null.

This distinction matters because stagewise p-values can be dependent. Even if each has a uniform null distribution, the distribution of a function of them depends on their joint null distribution. A combination function and a cutoff therefore form a test only when their null calibration is justified.

Definition: Let $P_1,\ldots,P_K$ be the stagewise p-values, and let $C(P_1,\ldots,P_K)$ be a prespecified combination statistic, oriented so that smaller values provide stronger evidence against $H_0$. A rule that rejects when $C(P_1,\ldots,P_K)\le c$ is a level-$\alpha$ test if, under the null, $$ P\bigl(C(P_1,\ldots,P_K)\le c\bigr)\le \alpha. $$ For a composite null hypothesis, this inequality must hold for every parameter value covered by $H_0$, or for the least-favourable null distribution used to calibrate the rule.

The combination function $C$, its inputs, and the rejection rule must be specified without using the observed stagewise p-values to choose a more favourable function or cutoff. The null distribution used for calibration must also reflect the trial's design, including any adaptation that affects how the stagewise p-values are generated.

From a Joint Null Distribution to a Test

Write $F_0(c)=P_0(C(P_1,\ldots,P_K)\le c)$ for the lower-tail null probability of the combination statistic. If the joint null distribution of the stagewise p-values is known, then $F_0$ can be calculated or otherwise bounded. Choose a cutoff $c_\alpha$ such that $F_0(c_\alpha)\le\alpha$, and reject when the observed combination statistic is at most $c_\alpha$.

P_0\bigl(C(P_1,\ldots,P_K)\le c_\alpha\bigr) =F_0(c_\alpha)\le\alpha.

This is the core derivation: the probability of rejection under the null is exactly the probability that the statistic falls in its prespecified rejection region. Calibration places an upper bound of $\alpha$ on that probability. If $F_0$ is continuous and the cutoff can be chosen to satisfy $F_0(c_\alpha)=\alpha$, the test uses the full nominal type I error rate. If the distribution has discrete jumps, a nonrandomized cutoff may give a probability below $\alpha$; that is still a valid, possibly conservative, level-$\alpha$ test.

Theorem: Suppose the combination statistic and rejection region are prespecified, and the null calibration satisfies $P_\theta(C\le c_\alpha)\le\alpha$ for every $\theta$ in $H_0$. Then the test has type I error rate at most $\alpha$. This conclusion follows directly because a type I error is precisely rejection when $H_0$ is true.

A stagewise p-value is valid under a null when its lower-tail probabilities are no larger than those of a uniform random variable: $P_0(P_i\le u)\le u$ for every $0\le u\le1$. That property describes each input separately. It does not by itself specify the joint distribution of all inputs, so it cannot justify every possible combination function and cutoff.

Worked Example: Calibrating the Minimum of Two Independent P-Values

Suppose a hypothetical two-stage trial has exact, independent stagewise p-values under a simple null hypothesis. Thus $P_1$ and $P_2$ are independent and each is uniform on $[0,1]$. Consider the prespecified statistic $C=\min(P_1,P_2)$, and set $\alpha=0.05$.

The event that the minimum is at most $c$ occurs if at least one p-value is at most $c$. By independence, $$ P_0(C\le c)=1-P_0(P_1>c,\ P_2>c) =1-(1-c)^2. $$

Set this probability equal to 0.05 to find the largest exact cutoff: $$ 1-(1-c)^2=0.05,\qquad c=1-\sqrt{0.95}=0.02532. $$

For hypothetical observed values $p_1=0.018$ and $p_2=0.071$, the statistic is $C=\min(0.018,0.071)=0.018$. Since $0.018\le0.02532$, the calibrated test rejects $H_0$ at level 0.05. The conclusion is about the test's null rejection probability; it is not a statement that the null has a particular probability of being true.

Why Marginal Validity Is Not Enough

The preceding calculation used a joint-distribution assumption. To see why it matters, consider two exact uniform p-values that are perfectly dependent: $P_1=P_2=U$, where $U$ is uniform on $[0,1]$ under the null. Each p-value is valid, but the pair does not have the independent joint distribution used in the previous example.

Worked Example: Calibrating a Product Under a Different Joint Null

In a hypothetical trial, use the prespecified combination statistic $C=P_1P_2$, with $P_1=P_2=U$ under the null. Then $C=U^2$. For $0\le c\le1$, $$ P_0(C\le c)=P_0(U^2\le c)=P_0(U\le\sqrt{c})=\sqrt{c}. $$

At $\alpha=0.05$, the calibrated cutoff solves $\sqrt{c}=0.05$, so $c_\alpha=0.05^2=0.0025$. If the hypothetical observed p-values are both 0.04, their product is $0.04\times0.04=0.0016$, which is below 0.0025; the calibrated test rejects.

If an analyst instead used the cutoff 0.05 without recalibrating, the null rejection probability would be $P_0(U^2\le0.05)=\sqrt{0.05}=0.2236$, not 0.05. The example shows that valid marginal p-values alone do not validate a chosen product cutoff. The joint null distribution must support that cutoff.

The same principle applies when the joint null distribution is not known exactly. A valid test can still be obtained if a conservative bound on the rejection probability is available. For instance, a union bound gives a simple rule that does not require stagewise independence.

Worked Example: A Dependence-Robust Bonferroni Combination

Suppose two stagewise p-values are each valid under the global null, but their dependence is unspecified. Reject if either p-value is at most $\alpha/2$, with $\alpha=0.05$. This is equivalent to rejecting when $C=2\min(P_1,P_2)\le0.05$.

Under the null, the rejection event is the union of the two events $P_1\le0.025$ and $P_2\le0.025$. Therefore, $$ P_0(\text{reject}) \le P_0(P_1\le0.025)+P_0(P_2\le0.025) \le0.025+0.025=0.05. $$

The argument uses only the validity of each p-value and the union bound, not independence. For hypothetical observations $p_1=0.030$ and $p_2=0.040$, neither is at most 0.025, so the rule does not reject. Equivalently, the Bonferroni-adjusted combined value is $\min(1,2\min(0.030,0.040))=0.060$, which exceeds 0.05. This rule may be more conservative than a test calibrated to a known joint null distribution, but its validity does not depend on knowing that distribution.

Assumptions, Checks, and Alternatives

A derivation is only as sound as its null model. Before using a combination rule, check the following points:

Common hypothesis: the stagewise p-values address the same null hypothesis, or the combination rule specifies precisely how distinct stagewise hypotheses define the final null.
Valid inputs: each stagewise p-value has its claimed null validity for the data and analysis actually used at that stage. If validity is conditional on earlier information, the conditioning statement must match the trial's adaptation process.
Appropriate joint calibration: the joint null distribution or a conservative bound accounts for dependence between stagewise p-values. Marginal uniformity alone is not a joint-distribution assumption.
Prespecification: the combination function and cutoff are not selected after inspecting the p-values unless that selection is itself included in the calibration.
Uniform control for composite nulls: if $H_0$ contains nuisance parameter values, the size bound holds across those values, not merely at one convenient parameter setting.

There are two broad calibration choices. If a defensible joint null distribution is available, calibrating the chosen function to that distribution can use more information and may be less conservative. If dependence is unknown, a bound such as Bonferroni can protect the type I error rate with weaker assumptions, often at the cost of power. These are alternatives for establishing the required probability bound; neither makes an incorrectly specified stagewise p-value valid.

Diagnostics should focus on the ingredients of that bound. Compare the assumed stagewise data-generating process with the prespecified analysis, verify that stage membership and any adaptation are handled as planned, and check calculations of the null rejection probability by derivation or simulation when appropriate. Simulation can reveal implementation errors or sensitivity to plausible null configurations, but a finite simulation alone does not prove a probability bound. In particular, repeatedly trying combination functions and reporting only the most favourable result is an unaccounted selection step.

Common pitfall: “Both stagewise p-values are valid, so any function of them is automatically a valid final p-value.” This is wrong: the marginal properties do not determine the null probability of every joint rejection region. The correction is to derive or conservatively bound $P_0(C\le c)$ under the relevant joint null, then choose a cutoff whose bound is no greater than $\alpha$.

Scope of the Principle

The combination-test principle is a calibration result, not a guarantee that every combination rule is powerful, scientifically appropriate, or robust to misspecification. A valid rule controls the probability of a false rejection under its stated null assumptions. It does not by itself supply an effect estimate, establish clinical importance, or make a post hoc choice of hypothesis valid.

The general recipe is deliberately simple: specify the stagewise inputs, specify a function and rejection region, establish the null probability of that region, and ensure it is at most $\alpha$. More specialized combination functions require their own null-distribution arguments. When those arguments rely on features of the stagewise design, those features must be justified rather than inferred merely from the fact that several stages were run.

Check Your Understanding

Answer each question, then compare with the brief solutions below.

What probability inequality must hold for a rule that rejects when $C\le c_\alpha$ to be level $\alpha$?
Two p-values are each marginally uniform under the null. Is that enough to calibrate the cutoff for an arbitrary function of the pair? Explain.
Under a null model, $P_1=P_2=U$ with $U$ uniform. For $C=P_1P_2$, find the cutoff giving null rejection probability 0.01.
Two valid p-values are used with the rule “reject if either is at most 0.02.” What upper bound does the union bound give for the null rejection probability?
Name one reason a simulation check cannot, by itself, establish that the test is level $\alpha$.

Solutions. (1) The null probability $P_0(C\le c_\alpha)$ must be no greater than $\alpha$, uniformly over null parameter values when the null is composite. (2) No. Marginal distributions do not determine the joint distribution, and the probability of a rejection region formed from both p-values can depend on their dependence. (3) Since $C=U^2$, $P_0(C\le c)=\sqrt{c}$; set $\sqrt{c}=0.01$, giving $c=0.0001$. (4) The bound is $0.02+0.02=0.04$, so the rule is level 0.04 or more generally level 0.05. (5) A finite simulation estimates a rejection probability with Monte Carlo uncertainty and cannot prove the required bound for all null parameter values or unexamined design configurations.