Why an Average Can Hide a Problem
An adaptive trial may use interim data to change how testing continues. A proposed continuation can have a small average probability of rejection under the null and still spend more error than the original test allowed in a particular interim state. The distinction matters whenever the justification for the adaptation is that it preserves the original test’s conditional error state by state.
This does not mean that every test with a statewise excess necessarily has an inflated overall type I error. If its rejection probability is uniformly bounded by the significance level under every null configuration, it can still control marginal type I error. The narrower point is that an average bound alone does not establish the stronger, state-by-state conditional error guarantee.
The original test’s overall type I error is the average of its conditional errors:
That identity is the law of total expectation. It also explains the central issue: an expectation averages across interim states, weighting each by its probability. A low average does not imply that each value being averaged is low.
The Statewise Condition and What It Guarantees
Suppose the proposed continuation uses only information available at the interim analysis, and its conditional rejection probability is \(q(s)\). Under the same null distribution used to define \(A(s)\), the statewise conditional error condition is
“Almost every” allows exceptions on states having probability zero under the specified null distribution. With a composite null, a guarantee intended to hold for every null parameter must be established under each relevant null distribution, not merely under one convenient weighting of interim states.
The condition gives a direct marginal guarantee. Taking expectations of both sides yields
Thus, if the original test has type I error at most \(\alpha\), a continuation satisfying the conditional inequality also has type I error at most \(\alpha\), under the null distribution for which the inequality holds. The converse does not follow: an inequality between averages does not establish the statewise inequality.
Worked Example: A Small Overall Error With a Statewise Excess
Consider a hypothetical test with significance level \(\alpha=0.05\). Under a specified null, the interim result is in state L with probability 0.90 and state H with probability 0.10. The original test has conditional errors \(A(\mathrm{L})=0.01\) and \(A(\mathrm{H})=0.20\). A proposed continuation has \(q(\mathrm{L})=0.04\) and \(q(\mathrm{H})=0.01\).
The original test’s marginal rejection probability is
The proposed continuation’s marginal rejection probability is
The proposed average, 0.037, is below \(\alpha=0.05\), so this calculation is consistent with marginal control at the stated null. But in state L, \(q(\mathrm{L})=0.04\) exceeds \(A(\mathrm{L})=0.01\). The continuation therefore does not preserve the original conditional error in every state. The small overall probability is not evidence that the statewise requirement holds.
Why Even a Smaller Average May Mislead
The concern is not limited to comparing a proposed test’s average with \(\alpha\). Even a proposed average smaller than the original test’s average can coexist with a statewise excess. A different state can contribute enough unused conditional error to compensate in the average, while concealing the excess elsewhere.
Worked Example: Compensation Across Common and Rare States
In a hypothetical trial, suppose a common interim state C has probability 0.99 and a rare state R has probability 0.01 under the null. The original conditional errors are \(A(\mathrm{C})=0.005\) and \(A(\mathrm{R})=0.30\). The proposed continuation has \(q(\mathrm{C})=0.007\) and \(q(\mathrm{R})=0.05\).
The original average conditional error is
The proposed average is
The proposed average, 0.00743, is smaller than the original average, 0.00795. Nevertheless, in the common state C, the proposed conditional rejection probability 0.007 exceeds the original conditional error 0.005. In the rare state R, the proposed continuation uses less than the original conditional error. Averaging those states hides the excess in C.
The numbers also show why checking only the rare state can be misleading. Its original conditional error is large, but its small probability gives it little influence on the average. Statewise comparison asks a different question: whether the proposed continuation respects the available conditional error at each state.
A Short Justification of the Distinction
For a finite set of interim states \(s\), with probabilities \(w_s=P_0(\mathcal{F}=s)\), the marginal rejection probability of the continuation is
The original test’s marginal rejection probability is \(\sum_s w_s A(s)\). If \(q(s)\leq A(s)\) for every state with positive probability, then each difference \(w_s\{A(s)-q(s)\}\) is nonnegative, so
But the reverse implication fails: a sum of weighted differences can be nonnegative even when one difference is negative, because other states can offset it. For example, the preceding calculation gives \(E_0(A)-E_0(q)=0.00052>0\), even though \(A(\mathrm{C})-q(\mathrm{C})=-0.002\). This is why a marginal comparison cannot prove the statewise condition.
State Probabilities and Composite Nulls
An average depends on the probabilities assigned to interim states. If those probabilities differ across null parameter values, the same conditional rejection probabilities can produce different marginal error rates. A bound calculated using one set of state probabilities therefore does not automatically establish a bound for another null configuration. The earlier material on nuisance parameters in error bounds addresses the broader challenge of ensuring that error control holds across a composite null.
Worked Example: The Same Continuation Under Two Null Configurations
Suppose a hypothetical null model has two possible configurations, \(\theta_1\) and \(\theta_2\). There are two interim states, L and H. At either configuration, the original conditional errors are \(A(\mathrm{L})=0.005\) and \(A(\mathrm{H})=0.08\); the proposed continuation has \(q(\mathrm{L})=0.015\) and \(q(\mathrm{H})=0.04\). The probability of state H is 0.10 under \(\theta_1\) and 0.40 under \(\theta_2\).
Under \(\theta_1\), the original and proposed marginal errors are
Under \(\theta_2\), they are
Both proposed marginal errors are below \(\alpha=0.05\) for these two configurations, but the statewise inequality fails at L because \(0.015>0.005\). The proposed average is greater than the original average under \(\theta_1\) and smaller under \(\theta_2\), illustrating how the weighting changes the comparison. This example does not show that the proposed test lacks marginal control at these two configurations; it shows that those average calculations do not establish conditional error preservation.
Assumptions, Diagnostics, and Alternatives
A conditional error argument is meaningful only when the conditioning information and the null distribution are specified correctly. In particular, the proposed continuation’s \(q(s)\) must describe its rejection probability conditional on the actual interim information used to adapt the trial. The original \(A(s)\) must be computed under the same relevant null model and conditioning scheme. If adaptation uses information omitted from the conditioning set, the stated conditional probabilities may not represent the actual procedure.
A practical diagnostic is to put \(A(s)\) and \(q(s)\) side by side over the relevant interim states and inspect the difference \(q(s)-A(s)\). For a continuous interim statistic, the analogous check concerns the conditional functions over the states that can occur; an average evaluated at a few points is not a statewise check. Where the null is composite, repeat or otherwise justify the comparison across the null configurations required by the error-control claim. Simulation can help find problematic states, but a finite simulation that fails to observe an excess does not prove that no excess exists.
There are two distinct goals, and the appropriate check depends on which one is being claimed:
| Goal | Relevant check | What the check establishes |
|---|---|---|
| Marginal type I error control | Bound the overall rejection probability under every null configuration required by the claim. | The probability of a false rejection, averaged over interim states, is controlled. |
| Preservation of the original conditional error | Show \(q(s)\leq A(s)\) state by state, under the relevant null distributions. | The proposed continuation does not exceed the original test’s conditional error at an interim state. |
An average calculation is useful for the first goal if its bound is valid uniformly over the null configurations of interest. It is not a substitute for the second goal. Conversely, statewise domination is a sufficient route to a marginal bound by averaging, but it may be more restrictive than necessary if the only required claim is marginal control.
Common Mistake and Limitations
The statewise requirement can be conservative: it may rule out a continuation that has a valid marginal type I error under the null model of interest. It also depends on correctly identifying the interim information and null distributions relevant to the guarantee. The useful practice is to state the intended guarantee precisely, then supply evidence for that guarantee rather than using an average bound to stand in for a different one.
Check Your Understanding
Answer each question, then compare with the solutions.
- In one sentence, distinguish a marginal error bound from a statewise conditional error bound.
- Two interim states have probabilities 0.8 and 0.2, with original conditional errors 0.01 and 0.10. A proposed continuation has conditional errors 0.02 and 0.01. Calculate both averages and identify any statewise excess.
- True or false: if the proposed average rejection probability is below the original average, then the proposed conditional rejection probability is no larger in every state. Explain.
- Why can an average error calculation under one null configuration be insufficient for a composite-null guarantee?
Solutions: (1) A marginal bound controls rejection probability averaged over interim states, whereas a statewise bound compares conditional rejection probabilities separately at each state. (2) The original average is \(0.8(0.01)+0.2(0.10)=0.028\). The proposed average is \(0.8(0.02)+0.2(0.01)=0.018\). The proposed probability exceeds the original conditional error in the first state because \(0.02>0.01\), despite the smaller average. (3) False. A weighted sum can be smaller even if one term is larger, because another state can offset it. (4) The null configuration can change the probabilities of interim states, and therefore change the average; control must be justified for every null configuration included in the claim.