Conditionally Valid Tests

Why an Average Error Rate May Not Be Enough

In an adaptive trial, a test used after interim review can depend on what was observed at that review. Its null rejection probability may therefore differ across interim outcomes. An overall Type I error rate summarizes those outcomes by averaging over them; it does not say how much error the test spends in any particular interim state.

That distinction matters when a design must remain controlled after the interim data are known. A test can have an acceptable average rejection probability yet reject too often in some states. Conditional validity makes the statewise requirement explicit. The earlier material on conditioning on interim data and the conditional error bound provides the setting: interim information is represented by a sigma-field, and the remaining null rejection probability is evaluated conditional on that information.

The conditional probability is a random quantity: it is a function of the interim information. “Almost surely” means that the inequality must hold except possibly on interim events with probability zero under the null distribution in question. It is not a requirement on impossible data configurations. When a claim covers more than one null distribution, the inequality must hold under each distribution included in that claim.

A bound \(b\) may be a constant, such as \(\alpha\), or may vary with the interim state, as a conditional error allowance can. The key requirement is not that the same amount of error be used in every state; it is that the conditional rejection probability not exceed the applicable bound in each state, apart from null-probability exceptions.

Conditional Control Implies Average Control

The connection between conditional and unconditional validity follows directly from the law of total probability. Because \(\phi\) is an indicator, its expectation is the probability of rejection. Taking expectations of the conditional rejection probability gives

Thus, if \(P_\theta(\phi=1\mid\mathcal F_I)\leq\alpha\) almost surely, averaging both sides yields \(P_\theta(\phi=1)\leq\alpha\). More generally, if the state-dependent allowance is \(A\), then

The reverse implication does not hold. An average can be within its target even when one or more states exceed that target, provided other states have sufficiently low rejection probabilities. This is why an unconditional Type I error calculation alone cannot establish conditional validity.

Statewise Bounds and Conditional Error Allowances

In adaptive design, the relevant statewise bound need not be the trial-wide significance level. It may instead be the conditional error allowance from the original design: the null rejection probability the original procedure had remaining after the interim information was observed. A redesigned continuation preserves that allowance when its conditional rejection probability is no greater than the allowance in every relevant state.

This comparison is stricter than checking whether the new design’s unconditional error is below the original unconditional level. The statewise comparison prevents one interim outcome from receiving more conditional error than the original procedure allowed, even if other outcomes receive less. The conditional error preservation result developed earlier relies on precisely this type of bound.

Checking Conditional Calibration

A direct way to assess conditional validity is to identify the interim information being conditioned on and calculate the null rejection probability within each state. If the test uses a statistic with a state-dependent null distribution, calibration must account for that distribution. For example, a common unstandardized cutoff may correspond to different tail probabilities in different states.

The next example illustrates state-specific standardization. Its conclusion depends on the stated conditional null distributions: if those distributions are wrong, the calculated tail probabilities do not establish conditional validity.

Diagnostics, Alternatives, and Limitations

A practical diagnostic is to tabulate the conditional null rejection probability and the applicable bound for every interim state the test can produce. For a more detailed interim record, the corresponding calculation is a function of that record rather than just a small table. If the test depends on a data-dependent adaptation, the calculation must reflect the adaptation rule and the conditional distribution of the remaining data under the null.

Warning signs include a single pooled error calculation with no statewise check, a cutoff justified using a marginal rather than conditional distribution, or a state with a conditional rejection probability above its allowance. If some states are rare, a pooled average can be especially uninformative about their conditional error. A state of probability zero under the specified null does not affect the almost-sure bound for that null; whether it matters under another null distribution requires checking that distribution separately.

Unconditional level control is often a useful design objective and may be all that a particular claim requires. Conditional validity makes a stronger, state-resolved claim. Conditional error preservation is more specific still: it compares a redesigned test with the allowance inherited from an original design. These approaches are related but not interchangeable.

Conditional guarantees also depend on the probability model used to calculate them. If the conditional null distribution is misspecified, or if the interim information and adaptation are not represented correctly, a numerical statewise check may not provide the claimed guarantee. In applications, the conditioning information, null model, rejection rule, and allowance should be specified clearly enough that the conditional probabilities can be reproduced.