Why Audit a Conditional Error Function?
A conditional error function describes how much type I error remains available after each possible interim history. It is tied to a particular original test: the function at a history is the probability, under the null, that the original test would reject given that history. When a proposed function is used to justify a changed continuation rule, small errors in its definition or calculation can undermine the error-control argument.
An audit therefore asks three distinct questions. Does the function take values in the range of probabilities? Does it agree with the original rejection rule at each relevant interim history? And, when averaged over histories using their null probabilities, does it recover the original unconditional type I error? Passing only one or two of these checks is not enough to establish that a proposed function is the original test’s conditional error function.
This definition presumes that the original rule and the null distribution are specified, that the rejection event can be evaluated from the interim and future data, and that the conditional distribution of future data given the interim history is correctly specified. A function defined as a conditional probability must satisfy \(0\leq A_0(h)\leq 1\) for histories with positive probability. For continuous histories, conditional probabilities are defined up to sets of histories having probability zero; values assigned on such sets do not affect unconditional error.
For a composite null, the conditional distribution may depend on the null parameter. In that case, an audit must state which null distribution is being used, or check the relevant error-control claim across the null parameter values required by the design. A calculation under one null value does not automatically establish control over a composite null.
The Three Core Checks
First, check probability bounds: every proposed value must lie between zero and one. Second, check rule consistency: calculate the probability of the original rejection event under the correct conditional distribution and compare it with the proposed value at the same history. Third, check averaging: weight each conditional error by the null probability of its history. For a continuous history, the corresponding average is an integral.
This identity is the law of total probability applied to the rejection event. It says that the average of the original conditional error function must equal the original unconditional type I error. If the original test has size \(\alpha_0\), meaning \(P_0(\phi_0=1)=\alpha_0\) under the null being considered, then \(E_0[A_0(H)]=\alpha_0\). If its type I error is below a nominal target, the average equals its actual type I error, not necessarily the nominal target.
Worked Example: Deriving the Conditional Error for a Normal Test
Suppose, hypothetically, the original one-sided test observes interim statistic \(Z_1\) and future statistic \(Z_2\). Under the null, \(Z_1\) and \(Z_2\) are independent standard normal random variables. The original test rejects when \(T=\sqrt{0.5}Z_1+\sqrt{0.5}Z_2>1.96\). Find and evaluate its conditional error function.
The assumptions specify the null distribution and the conditional distribution needed for the calculation: given \(Z_1=z\), the future statistic \(Z_2\) remains standard normal. The original rejection rule becomes \(\sqrt{0.5}z+\sqrt{0.5}Z_2>1.96\). Solving for \(Z_2\) gives the equivalent event \(Z_2>(1.96-\sqrt{0.5}z)/\sqrt{0.5}\). Therefore,
where \(\Phi\) is the standard normal cumulative distribution function. At \(z=0\), the threshold is \(1.96/\sqrt{0.5}=2.772\), so \(A_0(0)=1-\Phi(2.772)\approx 0.0028\). At \(z=1.5\), the threshold is \((1.96-1.0607)/0.7071\approx 1.272\), giving \(A_0(1.5)\approx 0.1017\). Both values are between zero and one, and both come directly from the original rejection event.
For the averaging check, \(T\) is standard normal under the null: it is a weighted sum of independent standard normal variables with mean zero and variance \(0.5+0.5=1\). Hence \(P_0(T>1.96)=1-\Phi(1.96)\approx 0.0250\). By the law of total probability, averaging \(A_0(Z_1)\) over the standard normal distribution of \(Z_1\) also gives \(0.0250\). The large difference between \(A_0(0)\) and \(A_0(1.5)\) is expected: the conditional probability changes with the interim result even though its average is the original test’s type I error.
Averaging Discrete Histories Correctly
When interim histories fall into a finite set of categories, the averaging check is a sum. The weights are the probabilities of those histories under the specified null distribution; they are not generally equal. If a proposed function is claimed to be the original conditional error function, compare it with the conditional rejection probability separately in every history category before relying on the weighted average.
Worked Example: A Correct Average Does Not Establish Consistency
In a hypothetical test, the interim history is \(h_1\) with null probability \(0.4\) and \(h_2\) with probability \(0.6\). Given \(h_1\), the original test rejects with probability \(0.10\); given \(h_2\), it rejects with probability \(0.30\). A proposed function assigns \(0.05\) to \(h_1\) and \(1/3\) to \(h_2\). Audit the proposal.
The original conditional errors are \(A_0(h_1)=0.10\) and \(A_0(h_2)=0.30\). Their weighted average gives the original type I error: \[ 0.4(0.10)+0.6(0.30)=0.04+0.18=0.22. \] The proposed values are within the probability bounds. Their weighted average is also \(0.4(0.05)+0.6(1/3)=0.02+0.20=0.22\), since \(0.6(1/3)=0.20\).
Despite passing the bounds and average checks, the proposal is not the original conditional error function: it disagrees at both histories. In particular, \(1/3>0.30\) at \(h_2\). The matching average is not proof of consistency; a lower value at one history can offset a higher value at another. If a changed test has conditional rejection probabilities no greater than the original conditional errors at every history, averaging those inequalities guarantees its unconditional type I error is no greater than \(0.22\). This proposal does not supply that pointwise guarantee.
Check the Rejection Event, Not Just the Formula
A frequent source of error is calculating the probability of a different event from the one in the original rule. If the rule rejects only when a future count reaches a specified threshold, replacing that event by a broader one changes the conditional error—even if the resulting calculation is internally correct. Auditing requires tracing the event in the rule through the conditional probability calculation.
Worked Example: Detecting an Incorrect Rejection Event
Consider a hypothetical test with two possible interim histories, \(h_1\) and \(h_2\), having null probabilities \(0.3\) and \(0.7\). Given \(h_1\), the future count \(Y\) has a binomial distribution with 2 trials and success probability \(0.2\); given \(h_2\), it has 2 trials and success probability \(0.4\). The original rule rejects only if \(Y=2\). A proposed calculation instead uses the event \(Y\geq 1\).
For a binomial count with 2 trials and success probability \(p\), the probability of the original rejection event is \(P(Y=2)=p^2\). Thus \(A_0(h_1)=0.2^2=0.04\) and \(A_0(h_2)=0.4^2=0.16\). The original unconditional error is \(0.3(0.04)+0.7(0.16)=0.012+0.112=0.124\).
The broader event has probability \(P(Y\geq1)=1-P(Y=0)=1-(1-p)^2\). It gives \(1-0.8^2=0.36\) at \(h_1\) and \(1-0.6^2=0.64\) at \(h_2\). These are bounded probabilities, but they are not consistent with the original rule. Their weighted average is \(0.3(0.36)+0.7(0.64)=0.108+0.448=0.556\), not \(0.124\). Both the event check and the averaging check flag the mistake.
A Practical Audit and Its Failure Modes
A reliable audit separates the logical checks from the numerical work. For a finite set of histories, each conditional probability can often be calculated directly. For a continuous history, it may require a conditional distribution calculation or numerical integration. In either case, the inputs must match the original design: the same null, the same interim information, the same future-data distribution given that information, and the same rejection boundary.
Write down the null distribution, the interim history \(H\), and the original rejection indicator \(\phi_0\). Record the original unconditional type I error or its nominal target, as appropriate.
For each history, calculate \(P_0(\phi_0=1\mid H=h)\) using the original rule and the correct conditional distribution of future data.
Confirm that each proposed value lies in \([0,1]\), then compare it with the conditional probability just calculated. Do not use a matching average to excuse a history-specific discrepancy.
Sum over discrete histories or integrate over continuous histories, weighting by their null probabilities. Compare the result with the original unconditional type I error.
For numerical integration, useful diagnostics include refining the integration grid, checking that tail regions are represented, and comparing with a known analytic result when one is available. For simulation, report Monte Carlo uncertainty: a simulated average near the target does not establish exact equality. A discrepancy can also arise from using the distribution of histories under the wrong null, using equal weights for unequal-probability histories, or conditioning only on an adaptation choice when the full interim history contains additional information.
The conditional error approach differs from calibrating a new test solely by its unconditional rejection probability. Unconditional calibration can target an overall type I error, but it does not by itself show that the new rule respects the original error available at each history. It also differs from conditional power: conditional power concerns the probability of rejection under an alternative, whereas conditional error is calculated under the null using the original rejection rule. Alpha-spending functions allocate type I error across stages; a conditional error function instead describes the original test’s remaining conditional rejection probability given the observed history.
Common Mistake and Limitations
Wrong reasoning: “The proposed values are all between zero and one and their weighted average equals the original alpha, so they are the original conditional error function.” Correction: Those checks do not show that the values equal the conditional rejection probabilities of the original rule. Recalculate the original rejection event at each history and compare point by point. If the purpose is to justify a new rule by conditional error control, verify that its conditional rejection probability is no greater than the original conditional error at each relevant history; an unconditional average alone does not establish this guarantee.
An audit is only as reliable as its conditioning information and null model. If the conditional distribution is misspecified, the calculated function may be wrong even when the algebra is correct. For composite null hypotheses, error control may require checking more than one null parameter value. And for continuous histories, values at probability-zero histories do not affect the unconditional average, although a proposed rule must still be operationally defined for histories that can actually occur.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- If the original conditional errors are \(0.08\) and \(0.20\) at histories with null probabilities \(0.25\) and \(0.75\), what is the original unconditional type I error?
- A proposed function is within \([0,1]\) at every history and has the correct weighted average. What important check remains?
- Why must a continuous-history average use the null distribution of \(H\), rather than an unweighted average over selected grid points?
- Under what condition does a new test’s pointwise conditional error bound imply an unconditional type I error bound?
Solutions. (1) \(0.25(0.08)+0.75(0.20)=0.02+0.15=0.17\). (2) Check that the proposed value equals the conditional rejection probability of the original rule at each relevant history; bounds and an average do not establish that equality. (3) The law of total probability weights histories according to how often they occur under the null, so equal grid-point weights generally give the wrong average. (4) If, under the specified null, the new test’s conditional rejection probability is no greater than the original conditional error at almost every history, then averaging both sides over the null distribution of \(H\) gives the unconditional bound.