Combination Tests for Changing Stagewise Hypotheses

Why Combine Evidence Across Stages?

In a multi-arm multi-stage trial, an interim analysis may lead investigators to stop some arms, continue others, or change which hypotheses will be tested at the end. A test designed only for the original set of hypotheses may not automatically retain its error control after such a change. A combination test offers a way to use evidence from separate stages while allowing the later-stage testing plan to be adapted. Its validity, however, depends on how the stagewise p-values are obtained and how the adaptive decision is accounted for.

The key distinction is between combining evidence for a specified null hypothesis and controlling error across a data-selected family of hypotheses. A weighted combination can have a valid null distribution for an individual hypothesis without, by itself, controlling the probability of any false rejection across several selected arms.

Definition: For a one-sided test with evidence in the upper tail, let $p_1$ and $p_2$ be stagewise p-values, and set $Z_k=\Phi^{-1}(1-p_k)$ for stage $k$, where $\Phi$ is the standard normal cumulative distribution function. A two-stage inverse-normal combination statistic is $$ Z_{\mathrm{comb}}=w_1Z_1+w_2Z_2, \qquad w_1^2+w_2^2=1, \qquad w_1,w_2\geq 0. $$ The weights are specified before the stagewise results are used to test the final hypothesis. For a one-sided level-$\alpha$ test, reject when $Z_{\mathrm{comb}}\geq \Phi^{-1}(1-\alpha)$.

The p-values must be oriented consistently: smaller p-values mean stronger evidence against the null in both stages. A two-sided p-value or a p-value oriented toward the opposite treatment effect cannot be inserted into this formula unchanged. The weights often reflect planned information across stages, but their essential requirements here are that they are nonnegative, prespecified for the test, and have squared sum one.

Why the Combination Has a Standard Normal Null Distribution

First consider a fixed null hypothesis $H$. Suppose that under every parameter value in $H$, $p_1$ and $p_2$ are independent Uniform(0,1) random variables. Then $Z_k=\Phi^{-1}(1-p_k)$ has a standard normal distribution, and the two transformed values are independent. A weighted sum of independent standard normal variables is normal with mean zero and variance $w_1^2+w_2^2$, which equals one. Thus $Z_{\mathrm{comb}}$ is standard normal under $H$, and the stated critical value gives a level-$\alpha$ test.

In an adaptive trial, stage 2 is often conducted after observing stage 1. A useful sufficient condition is that, under the null being tested, the stage-2 p-value is Uniform(0,1) conditional on all stage-1 data and the adaptation. This makes $Z_2$ conditionally standard normal and independent of the stage-1 information. If $p_1$ is also super-uniform under the final null (that is, $P(p_1\leq u)\leq u$ for all $u$ between zero and one), the combination test is valid or conservative. Conditional validity at stage 2 is especially important when the chosen arms or the later-stage hypotheses depend on interim data.

Condition for changing hypotheses: For a final null $H$, the stagewise evidence must still be valid for that null: under every parameter value in $H$, the first-stage p-value must be super-uniform, and the second-stage p-value must be super-uniform conditional on the interim information and the adaptation. Independent stage data and a correctly calibrated stage-2 test are a standard way to secure the latter condition. Merely changing the hypothesis label after selection does not establish either condition.

This condition matters when the final test concerns a different hypothesis from the one emphasized at stage 1. For example, a final hypothesis about a selected arm can use that arm's earlier p-value if the earlier test was valid under the selected arm's null, and the final null is contained in the null under which that p-value was calibrated. If the final null does not imply the stage-1 null, the stage-1 p-value may not be valid for the final test. The hypothesis relationship must be checked rather than assumed.

Worked Examples

Worked Example: Combining Two Stagewise P-Values

Suppose a hypothetical one-sided test uses independent stagewise p-values $p_1=0.10$ and $p_2=0.02$, with prespecified weights $w_1=0.6$ and $w_2=0.8$. The squared weights sum to one: $0.6^2+0.8^2=1$. Assume both p-values are valid under the null, and that stage 2 is independent of stage 1 under that null.

Transform the p-values to upper-tail normal scores: $$ Z_1=\Phi^{-1}(0.90)\approx1.2816, \qquad Z_2=\Phi^{-1}(0.98)\approx2.0537. $$ Then $$ Z_{\mathrm{comb}} =0.6(1.2816)+0.8(2.0537) \approx0.7690+1.6430 =2.4120. $$ The one-sided combination p-value is $1-\Phi(2.4120)\approx0.00794$. At level $0.05$, the critical value is $\Phi^{-1}(0.95)\approx1.6449$, so this test rejects the null. The result is calibrated for this hypothesis under the stated validity and independence conditions; it does not alone establish familywise control if this is one of several hypotheses chosen after looking at the data.

Worked Example: Combining Evidence for an Interim-Selected Arm

Imagine a hypothetical trial with three experimental arms. At the interim analysis, the design selects arm B to continue. For arm B, suppose the stage-1 one-sided p-value is $0.08$ and the new, independent stage-2 data give a p-value of $0.02$. Use prespecified weights $w_1=0.5$ and $w_2=\sqrt{0.75}\approx0.8660$; their squared sum is one. Assume that the stagewise tests are valid for the arm B null, including conditional validity of the stage-2 test after selection.

The corresponding normal scores are $Z_1=\Phi^{-1}(0.92)\approx1.4051$ and $Z_2=\Phi^{-1}(0.98)\approx2.0537$. Therefore, $$ Z_{\mathrm{comb},B} =0.5(1.4051)+0.8660(2.0537) \approx0.7026+1.7785 =2.4811, \qquad p_{\mathrm{comb},B}=1-\Phi(2.4811)\approx0.00655. $$ For this arm-specific combination test, the evidence exceeds the one-sided $0.05$ critical value. This calculation does not make the selected-arm p-value a familywise-adjusted p-value. Selection among three arms can change the probability of at least one false rejection, so a familywise procedure must also account for the selection and the set of hypotheses being tested.

Conditional Error and Adaptive Continuation

The conditional-error principle gives a precise way to assess whether an adaptive continuation preserves an earlier design's type I error. At interim data $x$, define $A_H(x)$ as the probability, under null $H$, that the original design would reject $H$ using its planned future data, conditional on those interim data. A replacement procedure preserves conditional error if its conditional probability of rejecting $H$ is no greater than $A_H(x)$ for every relevant interim outcome $x$. The earlier material on conditional error provides the setting for this comparison; here, the combination statistic supplies a convenient form for a replacement test.

For a fixed combination critical value $c$, once the interim data are known, $Z_1$ is fixed. The combination test rejects when $$ Z_2\geq\frac{c-w_1Z_1}{w_2}. $$ If the stage-2 p-value is conditionally Uniform(0,1), this gives a conditional rejection probability of $$ 1-\Phi\left(\frac{c-w_1Z_1}{w_2}\right). $$ That probability can be larger than the original conditional error $A_H(x)$. Thus an unconditional combination test calibrated at level $\alpha$ is not necessarily a conditional-error-preserving replacement at every interim outcome.

Worked Example: Checking a Conditional-Error Budget

Suppose, hypothetically, that at a particular interim outcome the original design's conditional error for a null hypothesis is $A_H(x)=0.04$. The observed stage-1 p-value is $0.08$, so $Z_1\approx1.4051$. Let $w_1=0.5$, $w_2=0.8660$, and consider first the ordinary fixed $0.05$ combination cutoff $c=\Phi^{-1}(0.95)\approx1.6449$.

The stage-2 normal-score threshold is $$ \frac{1.6449-0.5(1.4051)}{0.8660} \approx\frac{0.9424}{0.8660} \approx1.088. $$ The corresponding conditional rejection probability is $1-\Phi(1.088)\approx0.138$, which is greater than $0.04$. This fixed cutoff does not preserve the stated conditional-error budget at this interim outcome.

To use the full budget of $0.04$, require $p_2\leq0.04$, or equivalently $Z_2\geq\Phi^{-1}(0.96)\approx1.7507$. Expressed as a combination-statistic boundary, this is $$ Z_{\mathrm{comb}}\geq 0.5(1.4051)+0.8660(1.7507) \approx0.7026+1.5161 =2.2187. $$ Conditional on these interim data, the probability of crossing this boundary is $0.04$. A lower conditional probability would also preserve the budget. In a real adaptive design, the boundary must be derived from the original $A_H(x)$ for each relevant interim outcome and null hypothesis, not chosen from this illustrative calculation.

What Selection Does—and Does Not—Guarantee

When an arm is selected using interim results, its stage-1 p-value is generally not Uniform(0,1) conditional on being selected. For example, selecting the arm with the smallest interim p-value preferentially selects unusually small values, even if all arm nulls are true. This is why it is unsafe to argue that a standard combination test automatically controls familywise error merely because its stagewise p-values are valid when considered arm by arm.

The conditional-error approach addresses this by evaluating the replacement procedure under each relevant null and interim history. In a multi-arm design, the error budget may need to be respected for intersection hypotheses as well as for individual arms, in keeping with the closed-testing framework. The selection rule, the hypotheses that remain testable, and the error control required must all be included in that assessment. A prespecified weight is useful for defining the combination, but it is not a substitute for this multiplicity accounting.

Other approaches include retaining and recalibrating a prespecified sequential boundary, or allocating error across arms and stages. Those methods may be more natural when the testing family and continuation plan remain fixed. A combination test is attractive when later evidence can be summarized by valid stagewise p-values and the continuation plan may change. In either case, the error guarantee must match the actual adaptation and target family of hypotheses.

Diagnostics and Common Pitfalls

Stage-2 validity after adaptation: Check that the later-stage test is valid conditional on the interim data and selection. Reusing interim observations in a nominally independent stage-2 analysis can invalidate this condition unless the analysis accounts for that dependence.
Hypothesis mismatch: Verify that the final null implies the null conditions under which the stagewise p-values were calibrated. A stagewise p-value for one estimand or contrast cannot automatically test a different final hypothesis.
Selection mistaken for multiplicity control: A valid combination test for each fixed arm does not necessarily control the chance of any false rejection among selected arms. Check the relevant familywise or intersection-hypothesis error budgets.
Weights chosen after seeing results: Selecting weights to maximize the observed combination statistic changes the test. Use weights prespecified for the test, or justify any adaptive choice through an appropriate calibration.
Inconsistent p-value direction: Ensure that both p-values measure evidence in the same direction. The transformation $Z_k=\Phi^{-1}(1-p_k)$ assumes small p-values correspond to evidence against the same one-sided null.
Assuming unconditional calibration implies conditional preservation: A level-$\alpha$ combination test can have conditional rejection probability above the original conditional error at particular interim outcomes. Compare the replacement's conditional error with $A_H(x)$, not only its overall nominal level.

Key takeaway: A weighted combination of stagewise p-values is calibrated when the stagewise evidence is valid for the final null and the required independence or conditional-validity conditions hold. After arm selection or hypothesis changes, preserve the appropriate conditional error and account for multiplicity separately; prespecified weights alone do not guarantee familywise error control.

Check Your Understanding

Answer each question, then compare with the brief solutions below.

With weights $0.6$ and $0.8$, why is the weighted sum of independent standard normal stage scores standard normal?
Why might the stage-1 p-value for an arm fail to be uniform conditional on that arm being selected?
At an interim outcome, the original conditional error is $0.03$. What condition must a replacement combination test meet to preserve it?
True or false: If each arm's combination test is individually valid, selecting an arm at interim automatically gives familywise error control.
Name one validity condition to check when the final hypothesis changes after interim selection.

Solutions. (1) The weighted sum has mean zero and variance $0.6^2+0.8^2=1$, and a linear combination of independent normal variables is normal. (2) Selection depends on interim data; for example, choosing the smallest p-value favors unusually small stage-1 values. (3) Conditional on that interim outcome, its rejection probability under the null must be no greater than $0.03$. (4) False; arm selection and testing multiple hypotheses require familywise or intersection-hypothesis error accounting. (5) For instance, verify that under every parameter value in the final null, the stage-2 p-value is super-uniform conditional on the interim data and adaptation.