Why Interim-Specific Error Rates Must Be Averaged
An adaptive trial can use interim data to choose how it proceeds: for example, it may select a continuation path based on an interim statistic. Under a null hypothesis, the probability of rejecting that null can differ across the possible interim outcomes. A single interim outcome therefore does not determine the trial’s type I error rate. The unconditional rate averages the conditional rejection probabilities over the distribution of interim data that would occur under the null.
The key is to specify the event whose probability is being controlled. For one null hypothesis, it is the event that the procedure rejects that hypothesis when it is true. For a family of hypotheses, it may be the event of rejecting at least one true null. In either case, the probability must be calculated under the relevant null configuration and across all possible interim outcomes and subsequent trial data.
The conditional rejection probability is a function of the interim data: different values of \(Y\) can produce different values of \(r_\theta(Y)\). The unconditional type I error is the expectation of that function under the null distribution of \(Y\):
This is the law of total probability written using conditional probabilities. It does not require interim and later data to be independent. The conditional probability must, however, use the correct distribution of later data given the interim data under \(\theta\), and it must reflect the procedure actually used after each interim outcome.
Deriving the Averaging Identity
For a finite set of possible interim outcomes \(y_1,\ldots,y_k\), the events \(Y=y_j\) are mutually exclusive and together exhaust the possible interim outcomes. Decomposing \(R\) across these events gives
The second equality follows from the definition of conditional probability. The sum is precisely the expectation of \(r_\theta(Y)\). Thus, each conditional rejection probability is weighted by the probability of the corresponding interim outcome under the null; the unweighted average is generally not the unconditional error rate.
For continuous interim data, the same decomposition is expressed as an integral. Suppose \(Y\) has null density \(f_Y(y;\theta)\). If \(r_\theta(y)=P_\theta(R\mid Y=y)\), then
This follows by integrating the conditional probability over the null distribution of \(Y\). For example, if \(W\) denotes later data and the procedure’s rejection rule is represented by an indicator \(I_R(y,w)\), then \(r_\theta(y)\) is the conditional expectation of \(I_R(y,W)\) given \(Y=y\). Integrating that conditional expectation over \(Y\) recovers the probability of rejection over the complete trial.
The assumptions needed for this identity are modest: the full procedure defines a rejection event, the interim data are part of the trial’s data, and the probabilities are evaluated under the same specified null distribution. No particular test statistic, normal approximation, or independence condition is needed for the identity itself. If the procedure has multiple possible paths, those paths must all be represented in \(R\) and in the conditional probabilities.
Worked Example: Two Interim Outcomes
Suppose an interim review has two possible outcomes under a hypothetical null model: favorable, with probability \(0.25\), and not favorable, with probability \(0.75\). Given favorable interim data, the probability of eventual rejection of the true null is \(0.04\). Given not favorable data, it is \(0.008\). Find the unconditional type I error probability.
Let \(Y=F\) denote a favorable interim outcome and \(Y=N\) a not-favorable outcome. The two conditional rejection probabilities and their weights are specified under the same null model, so the law of total probability gives
The unconditional type I error probability is \(0.016\), or \(1.6\%\). It is not the simple average \((0.04+0.008)/2=0.024\), because the two interim outcomes are not equally likely under the null.
Continuous Interim Data
When interim data are summarized by a continuous statistic, the averaging still uses the null distribution of that statistic. The conditional rejection probability may vary with the observed value, while the integral accounts for how often values in different regions occur under the null.
Worked Example: Integrating Over a Normal Interim Statistic
In a hypothetical trial, suppose the interim statistic \(Z\) has distribution \(Z\sim N(0,1)\) under the null. Assume the specified adaptive procedure has conditional rejection probability \(0.030\) when \(Z>1\), and \(0.012\) when \(Z\leq 1\). These are given as the procedure’s conditional probabilities under the null; calculate its unconditional error rate.
Under \(N(0,1)\), \(P_\theta(Z>1)=0.1587\), so \(P_\theta(Z\leq 1)=1-0.1587=0.8413\). Averaging the two conditional probabilities with these null probabilities gives
The unconditional type I error probability is approximately \(1.486\%\). The calculation uses the null distribution of \(Z\), not the distribution of \(Z\) in a population where the alternative is true. The assumed conditional probabilities already summarize what can happen after each region of interim data, including the later data relevant to rejection.
Applying the Identity to a Familywise Error Event
In a multiple-testing problem, the rejection event must match the error criterion. Under a configuration in which several hypotheses are true, the familywise error event is that at least one of those true null hypotheses is rejected. Conditional probabilities for individual rejections cannot simply be added unless their overlap is handled; the event to average is the union of the relevant rejection events.
Worked Example: Averaging a Familywise Error Probability
Suppose \(H_1\) and \(H_2\) are both true under a hypothetical null configuration. Interim data fall into strata \(A\), \(B\), and \(C\), with probabilities \(0.20\), \(0.50\), and \(0.30\), respectively. For each stratum, the conditional probabilities of rejecting \(H_1\), rejecting \(H_2\), and rejecting both are given below. Calculate the unconditional probability of rejecting at least one true null.
| Interim stratum | Probability of stratum | \(P_\theta(\text{reject }H_1\mid\text{stratum})\) | \(P_\theta(\text{reject }H_2\mid\text{stratum})\) | \(P_\theta(\text{reject both}\mid\text{stratum})\) |
|---|---|---|---|---|
| A | 0.20 | 0.02 | 0.03 | 0.01 |
| B | 0.50 | 0.05 | 0.04 | 0.02 |
| C | 0.30 | 0.08 | 0.06 | 0.03 |
Within each stratum, the conditional probability of at least one rejection is the probability of the union. For events \(R_1\) and \(R_2\), \(P_\theta(R_1\cup R_2\mid Y)=P_\theta(R_1\mid Y)+P_\theta(R_2\mid Y)-P_\theta(R_1\cap R_2\mid Y)\). Thus, the conditional familywise probabilities are \(0.02+0.03-0.01=0.04\) in stratum \(A\), \(0.05+0.04-0.02=0.07\) in \(B\), and \(0.08+0.06-0.03=0.11\) in \(C\). The unconditional value is
The probability of at least one false rejection in this configuration is \(0.076\), or \(7.6\%\). Adding the two marginal rejection probabilities within each stratum without subtracting their overlap would count cases in which both hypotheses are rejected twice.
Checks, Alternatives, and Common Errors
A practical calculation should make explicit what is being conditioned on and which null distribution supplies the weights. The conditional probabilities must correspond to the actual decision rule for every interim outcome. If the procedure stops in one region, for example, that stopping path contributes its own probability of rejection (often zero for a later rejection) rather than disappearing from the calculation.
Useful checks include confirming that the interim-outcome probabilities sum to one, that each conditional rejection probability lies between zero and one, and that all rejection probabilities refer to the same event and null configuration. For a continuous \(Y\), check that the density used in the integral is normalized and is the density under the null. In simulation, estimate the unconditional error by applying the full procedure to repeated data generated under the null; averaging estimated conditional probabilities over the simulated interim outcomes should target the same quantity.
Direct simulation of the complete trial is an alternative way to estimate the unconditional error rate when analytic conditional probabilities are difficult to obtain. Stratifying or integrating by interim data can clarify which parts of the sample space contribute to the result, but it still requires correct conditional calculations. Neither approach repairs a misspecified null model or an incompletely described adaptation rule. This averaging identity also does not, by itself, assert that a particular conditional rejection probability is acceptable; it states how the unconditional probability relates to those conditional probabilities.
- Wrong: “The two conditional error probabilities are \(0.04\) and \(0.008\), so the unconditional rate is their average, \(0.024\).” Correction: Weight each value by the probability of its interim outcome under the null. If those probabilities are \(0.25\) and \(0.75\), the rate is \(0.016\).
- Wrong: “The interim statistic and later data must be independent for the identity to hold.” Correction: Independence is not required. The conditional probability must use the correct distribution of later data given the interim data.
- Wrong: “A conditional rejection calculation for one pathway gives the trial’s overall type I error.” Correction: Include all interim outcomes and every path of the procedure in the event and its probability calculation.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- Under a null hypothesis, interim outcomes \(A\) and \(B\) have probabilities \(0.60\) and \(0.40\). The conditional rejection probabilities are \(0.01\) and \(0.05\). What is the unconditional type I error probability?
- If \(P_\theta(R\mid Y=y)=0.02\) for every possible interim outcome \(y\), what is \(P_\theta(R)\)? State why.
- A continuous interim statistic \(Y\) has null density \(f_Y(y;\theta)\). Write the integral for the unconditional error probability in terms of \(r_\theta(y)=P_\theta(R\mid Y=y)\).
- For two true null hypotheses, the conditional rejection probabilities in one stratum are \(0.04\) and \(0.03\), and the conditional probability of rejecting both is \(0.01\). What is the conditional familywise error probability in that stratum?
- Why is it generally incorrect to take the unweighted average of conditional rejection probabilities across interim strata?
Solutions. (1) \(P_\theta(R)=(0.01)(0.60)+(0.05)(0.40)=0.006+0.020=0.026\), or \(2.6\%\). (2) \(P_\theta(R)=E_\theta[0.02]=0.02\), since the conditional probability is constant across interim outcomes. (3) \(P_\theta(R)=\int r_\theta(y)f_Y(y;\theta)\,dy\). (4) The union probability is \(0.04+0.03-0.01=0.06\), or \(6\%\). (5) Interim strata may have different null probabilities; the correct average weights each conditional rejection probability by its stratum probability under the null.