Equivalence Between Conditional Error and Combination Tests

Why Conditional Error Connects Two Testing Approaches

A combination test and a conditional-error procedure can look different: one combines stagewise evidence, while the other limits the chance of a future rejection after seeing interim data. The connection is precise. For a given first-stage result, calculate the probability that the original combination test would reject over the as-yet-unobserved second-stage data. That probability is its conditional error. A redesigned second stage retains the original Type I error control if its conditional rejection probability does not exceed this allowance, for every interim result that can occur.

This is a conditional statement, not a claim that each interim result has probability at most $\alpha$ of leading to rejection under the null. Some interim results can have conditional error above $\alpha$; others have much less. What matters is that the conditional errors average to the original overall rejection probability.

Definition: Let $X_1$ denote all first-stage information used by the testing procedure, and let $R$ be the event of rejecting $H_0$. The conditional error function of a test is $$ A(X_1)=P_{H_0}(R\mid X_1), $$ where the probability is over the future data, conditional on the observed first-stage information. For a composite null hypothesis, the relevant conditional probability and control must be specified for each null parameter value, or uniformly over the null as required by the design.

When the Equivalence Holds

Consider a two-stage test with a prespecified rejection rule. Under $H_0$, suppose the future stage-two data have a known conditional distribution given $X_1$. After stage one, an adaptation may choose the stage-two sample size or another permitted feature of the test. Write $B(X_1)$ for the probability of rejection under that adapted procedure, conditional on $X_1$, under $H_0$.

If, for every possible interim history and every null parameter value covered by the design, $B(X_1)\leq A(X_1)$, the adapted test has Type I error no larger than the original test. This comparison requires the original and adapted procedures to use compatible conditional null models. In particular, the stage-two test must remain valid conditional on the observed first-stage data and on the adaptation decision. Independence of stagewise data under the null is one common way to obtain this property.

P_{H_0}(\text{reject after adaptation}) =E_{H_0}\{B(X_1)\} \leq E_{H_0}\{A(X_1)\} =P_{H_0}(\text{reject under the original test}).

The equalities are the law of total probability applied to the first-stage information. Thus, if the original procedure is level $\alpha$, the adapted procedure is also level $\alpha$ or more conservative. If $B(X_1)=A(X_1)$ at every interim history, the overall Type I error is preserved exactly, assuming the same first-stage distribution under the null.

This equivalence concerns conditional rejection probabilities, not necessarily identical rejection rules. Two tests can spend the same conditional error at a given interim result but reject for different stage-two outcomes, and so have different power under alternatives. To reproduce the original rule itself, the adapted test must match its conditional rejection event, not merely its probability.

Deriving the Conditional Error for an Inverse-Normal Test

For a one-sided test, suppose $Z_1$ and $Z_2$ are independent standard normal stagewise statistics under $H_0$. The original inverse-normal combination statistic is $C=w_1Z_1+w_2Z_2$, where $w_1,w_2>0$ and $w_1^2+w_2^2=1$. The procedure rejects when $C>c$, with $c$ chosen to give the intended overall level under this null model.

\begin{aligned} A(z_1) &=P_{H_0}(w_1z_1+w_2Z_2>c\mid Z_1=z_1)\\ &=P_{H_0}\left(Z_2>\frac{c-w_1z_1}{w_2}\right)\\ &=1-\Phi\left(\frac{c-w_1z_1}{w_2}\right), \end{aligned}

The second line follows by rearranging the rejection inequality; the last uses $Z_2\sim N(0,1)$ under the null. Therefore an adapted second-stage test that still has a standard normal statistic $Z_2^{\mathrm{new}}$, conditionally on stage one and the adaptation, can preserve the original conditional error by rejecting when $Z_2^{\mathrm{new}}>\Phi^{-1}\{1-A(z_1)\}$. This remains valid if information changes, provided the new stage-two statistic is conditionally standard normal under $H_0$. A new weight or sample size alone does not establish that condition; the conditional null distribution must be justified.

Worked Example: Computing and Reusing an Inverse-Normal Conditional Error

Suppose the original test has $w_1=0.6$, $w_2=0.8$, and $c=1.96$. These weights satisfy $0.6^2+0.8^2=1$. The observed first-stage statistic is $z_1=1.20$. Assume the independent standard-normal stagewise null model stated above.

The original test would reject if $0.6(1.20)+0.8Z_2>1.96$. Solving for $Z_2$ gives $$ Z_2>\frac{1.96-(0.6)(1.20)}{0.8} =\frac{1.24}{0.8}=1.55. $$

Thus the conditional error is $A(1.20)=P(Z_2>1.55)=1-\Phi(1.55)\approx0.0606$. The conditional error is about 6.06%, even if the overall test was designed at a lower level: it is an interim-conditional allowance, not the unconditional Type I error.

Suppose the adapted second-stage statistic remains standard normal under $H_0$, conditional on the first-stage result and the adaptation. The adapted procedure can preserve this conditional error by rejecting if $Z_2^{\mathrm{new}}>1.55$. For example, if $Z_2^{\mathrm{new}}=1.70$, the adapted test rejects. The original combination rule would also reject for that same second-stage value, since $0.6(1.20)+0.8(1.70)=2.08>1.96$. In this example the threshold recovers the original conditional rejection rule as well as its conditional error.

A Combination Test More Generally

The calculation is not restricted to normal statistics. Let $p_1$ and $p_2$ be one-sided stagewise p-values, and suppose that, under $H_0$, $p_2$ is uniform on $(0,1)$ conditional on the first-stage information. If the original combination function rejects when $g(p_1,p_2)$ falls in a specified rejection region, then its conditional error at $p_1$ is the probability, over this conditional null distribution of $p_2$, that the combination rule rejects. For a lower-tail rule this can be expressed as the length of the set of stage-two p-values that lead to rejection.

Worked Example: Conditional Error Under a Fisher Combination Test

Suppose the original one-sided Fisher test rejects when $-2\{\log(p_1)+\log(p_2)\}>9.488$, the 5% upper critical value of a chi-square distribution with 4 degrees of freedom. Under the null, assume $p_2$ is conditionally uniform given the first-stage information. The observed first-stage p-value is $p_1=0.10$.

Rearranging the rejection rule, $$ -2\{\log(0.10)+\log(p_2)\}>9.488 \quad\Longleftrightarrow\quad p_2<\frac{\exp(-9.488/2)}{0.10}\approx0.0870. $$

Since a uniform p-value is below 0.0870 with probability 0.0870, the conditional error at $p_1=0.10$ is approximately 8.70%. An adapted stage-two test can preserve this conditional error by rejecting when its conditionally valid p-value is below 0.0870. This number is not the overall Type I error; it is the conditional probability of rejection given this particular first-stage result.

The calculation presumes that the second-stage p-value is valid conditional on the first-stage information and the chosen adaptation. If that conditional uniformity fails—for example, because the second-stage data or p-value depend on the first-stage information in an unaccounted-for way—the 0.0870 threshold need not deliver the stated conditional error.

Equality, Conservative Control, and Practical Checks

A combination test always induces a conditional error when its conditional null distribution is defined. The reverse construction is not unique: many second-stage rejection rules can have the same conditional rejection probability. Exact equality $B(X_1)=A(X_1)$ preserves the original overall rejection probability; the pointwise inequality $B(X_1)\leq A(X_1)$ gives conservative control. A rule that exceeds $A(X_1)$ for some histories is not automatically invalid, but the simple pointwise guarantee no longer applies. Any such design needs a separate justification of its overall error rate.

Before using this connection, check the elements on which it depends:

Conditioning information: include all interim information used to adapt, not merely a statistic if the adaptation also depends on other observed data.
Conditional null law: establish the distribution of the future test statistic or p-value given that information and the selected design.
Null coverage: verify control at every relevant null parameter value, particularly for composite null hypotheses.
Calibration: confirm that the original combination rule has the claimed level under its stated model.
Implementation: calculate the conditional probability from the actual rejection boundary, including any early stopping or randomization in that rule.

A diagnostic warning is a mismatch between the assumed and actual conditional distribution: a threshold computed from a standard normal tail or a uniform p-value cannot be trusted if that reference distribution is no longer valid after adaptation. Also check that the adaptation uses only information available at the decision time and that the computed conditional-error function is based on the same rejection rule the protocol intends to preserve.

Approach	What is specified directly?	How the connection is used	Key limitation
Combination test	A rule combining stagewise evidence	Condition on stage one and calculate its probability of rejecting	Requires a valid conditional null distribution for future evidence
Conditional-error procedure	An allowable future rejection probability at each interim history	Choose a future rule whose conditional rejection probability is no larger than that allowance	The allowance alone does not uniquely determine the future rejection rule or its power

The combination-test representation is useful when a prespecified stagewise rule is the natural starting point; the conditional-error representation is useful when specifying or evaluating an adapted future test. Neither representation removes the need to justify the conditional null model. Nor does matching conditional error guarantee identical power: power also depends on how the future rejection region is arranged and on the distributions under alternatives.

Common Mistake

Wrong reasoning: “The conditional error at the observed interim result is 8.70%, which is greater than 5%, so the combination test violates a 5% Type I error limit.” This confuses a conditional probability at one particular first-stage outcome with the unconditional Type I error. Conditional errors can vary across interim outcomes.

Correction: Under the null, average the conditional errors over the first-stage distribution. For an original level-$\alpha$ test, $E_{H_0}\{A(X_1)\}=\alpha$ (or is at most $\alpha$ for a conservative test). An adapted procedure inherits this control when its conditional rejection probability is no larger than $A(X_1)$ at each relevant history.

Key takeaway: A combination test’s conditional error is the probability that its original rejection rule would reject, given the interim data. An adapted test preserves Type I error control when its conditional null rejection probability is bounded by that error for every relevant interim history and null parameter value.

Check Your Understanding

Answer each question, then compare with the brief solutions below.

An original combination test has conditional error $A(x)=0.04$ at a particular interim history. An adapted second-stage test has conditional rejection probability $0.03$ there. What does this comparison establish, and what more must be checked for an overall guarantee?
For an inverse-normal test with $w_1=0.5$, $w_2=\sqrt{0.75}$, $c=1.96$, and $z_1=0$, calculate the stage-two threshold and conditional error.
Why does matching a conditional rejection probability not necessarily make two tests identical?
A Fisher combination rule gives a conditional stage-two p-value cutoff of 0.02 at one interim result. Is its conditional error necessarily the test’s overall Type I error? Explain.

Solutions. (1) At that history the adapted procedure is no less conservative than the original, since $0.03\leq0.04$. For an overall guarantee, the bound and the conditional null model must hold at every relevant interim history and null parameter value, and the original test must itself have controlled Type I error. (2) The threshold is $\{1.96-(0.5)(0)\}/\sqrt{0.75}=2.263$ (approximately); the conditional error is $1-\Phi(2.263)\approx0.0118$. (3) They may reject on different sets of stage-two outcomes that have the same probability under the conditional null, so their power can differ. (4) No. The cutoff gives a conditional rejection probability of 0.02 if the stage-two p-value is conditionally uniform; overall Type I error averages conditional errors over the first-stage distribution.