Combination Tests for Adaptive Sample Size Designs

Why Combine Stagewise Evidence?

In a two-stage study, investigators may use interim data to change the planned second-stage sample size. That flexibility can make the original fixed-design test inappropriate: its reference distribution and rejection boundary may depend on the original allocation of information. A combination test instead reduces the evidence from each stage to a stagewise p-value and applies a prespecified rule to the resulting pair.

The key is not that adaptation leaves the data distribution unchanged. It does not. Rather, the test must use stagewise p-values that remain valid under the null, including after accounting for the rule that selected the second-stage design. Calibration of the combination rule then controls the overall Type I error.

Definition: A combination test is a test that calculates stagewise p-values \(p_1,\ldots,p_K\) and rejects \(H_0\) when a prespecified function or rejection rule applied to those values falls in a calibrated rejection region. A p-value is valid under \(H_0\) if it is super-uniform: for every \(u\in[0,1]\), \(P(p_i\le u\mid H_0)\le u\). It is exactly uniform when equality holds for every \(u\).

For adaptive sample-size re-estimation, validity of the second-stage p-value needs particular care. Let \(\mathcal{F}_1\) denote all information available after stage one, including the interim observations and the selected second-stage sample size. A sufficient condition is that, under \(H_0\), \(p_2\) is conditionally super-uniform given \(\mathcal{F}_1\):

P(p_2\le u\mid\mathcal{F}_1,H_0)\le u \qquad \text{for every }u\in[0,1].

This condition says that once the interim history and the selected design are fixed, the new stage-two data still produce a valid p-value. It is commonly supported by using only new stage-two observations in the stage-two test, with a correctly specified null distribution for the chosen sample size. Reusing stage-one data in a way that ignores its role in selecting the design can invalidate this condition.

Calibrating a Combination Rule

A combination rule specifies which pairs of stagewise p-values lead to rejection. For example, a rule might reject when at least one p-value is sufficiently small. Its rejection region is coordinatewise decreasing if making either p-value smaller, while leaving the other unchanged, cannot turn a rejection into a non-rejection. Such regions are especially useful because super-uniform p-values place no more probability in small-p-value regions than uniform p-values do.

Suppose first that \(p_1\) and \(p_2\) are independent uniform random variables under \(H_0\). A combination rule with rejection region \(R\subseteq[0,1]^2\) has null rejection probability equal to the area of \(R\). Choose \(R\) so this area is at most the desired significance level \(\alpha\). If the p-values are independent and super-uniform instead, a coordinatewise decreasing region has rejection probability no greater than under independent uniform inputs.

Key result: Suppose \(p_1\) is super-uniform under \(H_0\), \(p_2\) is conditionally super-uniform given \(\mathcal{F}_1\) under \(H_0\), and the combination rejection region is coordinatewise decreasing. Then the rejection probability is no greater than the rejection probability obtained by applying that same rule to two independent uniform p-values. Thus, a rule calibrated to have uniform-input rejection probability at most \(\alpha\) controls the overall Type I error at level \(\alpha\).

The argument is unconditional; it does not assert that rejection probability given every interim history is at most \(\alpha\). For a decreasing region, fixing \(p_1=x\) leaves a section of the form \(0\le p_2\le b(x)\), where \(b(x)\) is nonincreasing in \(x\). Conditional super-uniformity of \(p_2\) bounds the conditional probability of entering that section by its length, \(b(x)\). Averaging over \(p_1\), super-uniformity of \(p_1\) bounds the expectation of this decreasing function by its expectation under a uniform p-value. This gives the stated joint bound. The distinction matters: an interim history can make rejection certain even though the overall rejection probability is controlled.

Worked Examples

Worked Example: Calibrating a Minimum-P Combination Rule

Consider the rule that rejects when \(\min(p_1,p_2)\le c\). Choose \(c\) to give a two-stage test with \(\alpha=0.05\) when the stagewise p-values are independent uniforms under \(H_0\).

Under independent uniforms, the probability of not rejecting is the probability that both p-values exceed \(c\):

P(\min(p_1,p_2)>c)=(1-c)^2.

Set the rejection probability equal to \(0.05\) and solve:

1-(1-c)^2=0.05, \qquad c=1-\sqrt{0.95}=0.0253206.

Thus the rule rejects if either stagewise p-value is at most approximately \(0.02532\). Under independent uniform p-values its Type I error is exactly \(0.05\). Under independent super-uniform p-values, or under the conditional validity conditions above for this decreasing region, its Type I error is at most \(0.05\); equality is not guaranteed.

For instance, if \(p_1=0.01\), rejection occurs whatever value \(p_2\) takes. The conditional rejection probability given this \(p_1\) is therefore 1, not 0.05. That does not contradict overall error control: calibration bounds the probability averaged over the null distribution of the stage-one evidence, not the rejection probability for every fixed interim result.

Worked Example: Testing After an Adapted Second-Stage Sample Size

Suppose the one-sided hypotheses concern a beneficial treatment effect \(\delta\): \(H_0:\delta\le 0\) versus \(H_a:\delta>0\). Assume observations in the two stages are independent and normally distributed with known standard deviation \(0.20\) outcome units. Stage one gives \(p_1=0.12\). Under a prespecified adaptation rule, that interim result leads to a stage-two sample size of \(n_2=80\) per treatment group. The stage-two-only estimated treatment difference is \(0.045\) outcome units.

Under the boundary null \(\delta=0\), the standard error of the stage-two difference is

SE=0.20\sqrt{\frac{1}{80}+\frac{1}{80}} =0.03162\text{ outcome units}.

The stage-two statistic and its one-sided p-value are

z=\frac{0.045}{0.03162}=1.423, \qquad p_2=1-\Phi(1.423)\approx 0.0774.

The chosen sample size depends on the interim result, but the calculation uses new stage-two data and the stated known-variance normal model. Conditional on the stage-one history and the selected sample size, the stage-two statistic has the specified null distribution, so \(p_2\) is conditionally valid under \(H_0\). With the minimum-p rule from the previous example, neither \(0.12\) nor \(0.0774\) is at most \(0.02532\), so the combined test does not reject \(H_0\) at the 0.05 level. This is a test decision, not evidence that the null hypothesis has been proved.

Worked Example: Why Uncalibrated Stagewise Thresholds Inflate Error

Suppose two independent stagewise p-values are uniform under \(H_0\), but a proposed rule rejects if \(p_1\le0.0253\) or \(p_2\le0.0500\). Independence gives the null rejection probability

$$ 1-(1-0.0253)(1-0.0500) =1-(0.9747)(0.95) =0.074035. $$

The Type I error is approximately \(0.07404\), or \(7.404\%\), rather than \(5\%\). The first threshold is close to the calibrated minimum-p threshold from the earlier example, but adding a second, larger threshold creates a larger rejection region. The rule is not a valid 0.05 test as written.

One simple correction is Bonferroni allocation: reject if either p-value is at most \(0.025\). The union bound gives

P(p_1\le0.025\ \text{or}\ p_2\le0.025\mid H_0) \le 0.025+0.025=0.05.

This bound does not require independence, provided each stagewise p-value is valid in the relevant sense. It may be conservative; the minimum-p rule calibrated to \(0.0253206\) uses the independence and uniform-reference calculation for a less conservative cutoff.

Validity Conditions and Diagnostics

A combination test controls Type I error only if both its calibration and its inputs are appropriate. Before relying on a design, check the following:

Null validity: Each stagewise p-value must be valid for the null hypothesis being tested, with a direction and test statistic consistent across stages.
Adaptation accounted for: If stage two is selected using interim data, its p-value must remain valid conditional on the information and design selection used for that choice, or the combination rule must otherwise be justified for the actual joint distribution.
Correct reference calibration: An exact Type I error calculation based on independent uniform inputs is exact only when those assumptions hold. Merely valid p-values generally yield an upper bound, not equality.
Prespecified rule: The combination function, rejection region, and any choices that affect calibration should be specified before examining the combined evidence.
Compatible hypotheses: Stagewise tests must address the same null and alternative. Combining p-values from different directions or different hypotheses does not automatically test a single coherent claim.

Warning signs include using the original fixed-design critical value after changing the sample size, calculating a stage-two p-value from all observations as if the sample size had been fixed in advance, or reporting a nominal error rate without matching it to the actual combination rule. Simulation under \(H_0\) can be a useful diagnostic: repeatedly generate data, apply the complete adaptation and testing procedure, and estimate the overall rejection rate. Simulation is not a substitute for establishing validity, but a rate materially above \(\alpha\) can reveal a coding or calibration error.

How Combination Testing Fits With Other Error-Control Approaches

A fixed-design test uses its planned sampling scheme and critical value. When the sample size changes based on interim data, that original calibration may no longer apply. The earlier material on conditional error takes a different route: it evaluates the remaining allowable rejection probability after interim data and restricts subsequent decisions to preserve that allowance. Combination testing instead constructs a final test from stagewise evidence using a calibrated rule. These approaches share the objective of controlling the overall Type I error, but their calculations and design requirements differ.

A simple Bonferroni combination rule is transparent and robust to dependence when each p-value is valid, though it can lose power. Rules calibrated under independence can be more efficient when their dependence and validity assumptions are justified. No combination rule repairs an invalid stagewise p-value, and no rule guarantees a particular power or sample size merely by controlling Type I error.

Common Mistakes and Limitations

Claiming pointwise conditional error control from unconditional calibration. A small \(p_1\) can make rejection certain under a minimum-p rule. Control is over the full null distribution of interim data, not necessarily for each fixed interim history.
Equating valid with exactly uniform. Super-uniform inputs can make a calibrated rejection probability smaller than \(\alpha\). Use equality only when the assumed uniform distribution is justified.
Ignoring the adaptation mechanism. A second-stage p-value that is valid for a fixed sample size is not automatically conditionally valid after a sample size is selected using interim information.
Treating a favorable result as proof of the alternative. Rejection controls a long-run false-positive rate under \(H_0\); it does not give the probability that \(H_0\) is true.

Combination tests also have limitations. Their performance depends on the chosen combination rule, and a rule that controls Type I error may still be inefficient for the scientific alternative of interest. More complex settings may require careful treatment of nuisance parameters, dependence, multiple adaptations, or composite null hypotheses. Those issues must be resolved in the design and validation of the particular test rather than assumed away by the phrase “combination test.”

Check Your Understanding

Answer the questions, then compare your reasoning with the solutions.

For independent uniform \(p_1,p_2\), what cutoff \(c\) gives a 0.05 test that rejects when \(\min(p_1,p_2)\le c\)?
If a combination rule has 0.05 rejection probability under independent uniform inputs, what can be concluded when its inputs are independent and super-uniform?
Why does a conditionally valid stage-two p-value matter when \(n_2\) is chosen from interim data?
For independent uniform p-values, calculate the Type I error of the rule that rejects if \(p_1\le0.02\) or \(p_2\le0.04\). Is it a 0.05 test?

Solutions. (1) Solve \(1-(1-c)^2=0.05\), giving \(c=1-\sqrt{0.95}=0.0253206\). (2) For a coordinatewise decreasing rejection region, the rejection probability is no greater than 0.05; it need not equal 0.05. (3) The interim data influence which stage-two design is used. Conditional validity ensures that, under the null, the selected design and its new data still yield a valid stage-two p-value. (4) Independence gives \(1-(1-0.02)(1-0.04)=1-(0.98)(0.96)=0.0592\). The error is 5.92%, so this rule is not calibrated at 0.05.