Conditional Error Given Interim Data

Why Condition on the Interim Data?

A design can have a well-controlled unconditional Type I error rate and still assign very different null rejection probabilities to different interim outcomes. Conditional error makes this data-specific quantity explicit: after observing interim data, it asks how likely the original design would be to reject eventually if the null hypothesis were true and the original plan were followed.

The conditioning is on observed interim information, not on the truth of a hypothesis. The resulting value is a probability over the unobserved future data under the null model. It is therefore different from conditional power, which evaluates rejection probability under an alternative effect assumption.

Definition: Let $A$ be the event that a specified original design rejects $H_0$, and let $D_1$ denote the data available at an interim analysis. The conditional error at the observed interim data is $$ \operatorname{CE}(D_1)=P_0(A\mid D_1), $$ where $P_0$ denotes probability under the null model. If the conditional distribution of future data depends on $D_1$ only through the interim test statistic $Z_1$, then the conditional error can be written as a function $\operatorname{CE}(Z_1)$.

The event $A$ must be defined by the original design, including its rejection boundaries and any stopping decisions. If the original design has already rejected at the interim look, eventual rejection has occurred, so the conditional error is 1. If it has stopped without rejection and no further testing is allowed, the conditional error is 0. The nontrivial calculation is usually for interim data in the design’s continuation region.

Canonical One-Interim, One-Final Model

A useful model for deriving a closed-form conditional error uses one interim analysis and one final analysis of a normally distributed treatment effect. Assume the null hypothesis is at its boundary, the variance is known, and the test statistic has the canonical independent-increments representation. Let $I_1$ be the interim information and $I_2$ the planned total information, with $0<I_1<I_2$. Define the information fraction $t=I_1/I_2$, so $0<t<1$.

Under the null boundary, $Z_1\sim N(0,1)$. The final statistic $Z_2$ can be written as

Z_2=\sqrt{t}\,Z_1+\sqrt{1-t}\,Z_{\Delta}, \qquad Z_{\Delta}\sim N(0,1),

where $Z_{\Delta}$ is independent of the interim data. This representation applies, for example, to a normal-theory test with fixed information and independent increments. It does not automatically apply to every test, variance estimator, or adaptive data-generating process; those settings require their own conditional distribution.

Suppose the original design rejects only at the final look when $Z_2>c_{\alpha}$, where $c_{\alpha}$ is the one-sided upper critical value satisfying $P(Z>c_{\alpha})=\alpha$ for $Z\sim N(0,1)$. Given $Z_1=z_1$, the conditional mean of $Z_2$ is $\sqrt{t}\,z_1$, and its conditional standard deviation is $\sqrt{1-t}$. Standardizing the rejection event gives the conditional-error function.

Result: Under the stated canonical model, for a one-sided final rejection rule $Z_2>c_{\alpha}$, $$ \operatorname{CE}(z_1) =P_0(Z_2>c_{\alpha}\mid Z_1=z_1) =1-\Phi\!\left(\frac{c_{\alpha}-\sqrt{t}\,z_1}{\sqrt{1-t}}\right), $$ where $\Phi$ is the standard normal cumulative distribution function. The formula is for interim outcomes that continue to the final analysis; any original early-stopping rule must also be applied.

Derivation and Interpretation

Conditioning on $Z_1=z_1$ fixes the first term in the canonical representation. Only the future increment remains random:

\begin{aligned} P_0(Z_2>c_{\alpha}\mid Z_1=z_1) &=P_0\!\left(\sqrt{t}\,z_1+\sqrt{1-t}\,Z_{\Delta}>c_{\alpha}\right)\\ &=P_0\!\left(Z_{\Delta}> \frac{c_{\alpha}-\sqrt{t}\,z_1}{\sqrt{1-t}}\right)\\ &=1-\Phi\!\left(\frac{c_{\alpha}-\sqrt{t}\,z_1}{\sqrt{1-t}}\right). \end{aligned}

This derivation shows why the conditional error is a function of the observed interim statistic: the observation shifts the conditional mean of the final statistic, while the remaining information determines its conditional standard deviation. With $t$ and the critical value fixed, a larger $z_1$ gives a larger conditional error. That monotonicity follows because increasing $z_1$ lowers the standardized distance to the rejection boundary.

A conditional error is not itself the overall Type I error rate. For a design whose unconditional null rejection probability is $\alpha$, averaging its conditional error over the null distribution of interim data recovers that unconditional probability:

E_0\!\left[P_0(A\mid D_1)\right]=P_0(A)=\alpha.

This is a useful calibration check: a small conditional error for one observed interim result is compatible with an overall Type I error of $\alpha$, because other interim outcomes can have larger conditional errors.

Worked Example: One-Sided Conditional Error at an Interim Look

Suppose a hypothetical normally distributed trial has information fraction $t=0.50$. Its original design makes a one-sided final rejection when $Z_2>1.96$. At the interim analysis, the observed statistic is $z_1=1.20$. Assume no early stopping rule and the canonical null model above. Calculate the conditional error.

The conditional mean of the final statistic is $\sqrt{0.50}(1.20)=0.8485$, and its conditional standard deviation is $\sqrt{1-0.50}=0.7071$. Therefore,

\begin{aligned} \operatorname{CE}(1.20) &=1-\Phi\!\left(\frac{1.96-\sqrt{0.50}(1.20)}{\sqrt{0.50}}\right)\\ &=1-\Phi\!\left(\frac{1.96-0.8485}{0.7071}\right)\\ &=1-\Phi(1.572)\\ &\approx 0.0580. \end{aligned}

Under the null model and conditional on this interim statistic, the original design has about a 5.8% probability of crossing its final one-sided boundary. This conditional probability exceeds the nominal 2.5% one-sided level because the observed interim statistic is favorable to rejection. It does not mean the null has a 5.8% probability of being true.

Two-Sided Tests and Stopping Boundaries

For a two-sided final rule that rejects when $|Z_2|>c$, both tails must be included. Under the same canonical assumptions, the conditional error at a continuing interim value $z_1$ is

\operatorname{CE}(z_1) = \Phi\!\left(\frac{-c-\sqrt{t}\,z_1}{\sqrt{1-t}}\right) + 1-\Phi\!\left(\frac{c-\sqrt{t}\,z_1}{\sqrt{1-t}}\right).

The first term is the conditional probability of crossing the lower boundary; the second is the conditional probability of crossing the upper boundary. Omitting either tail would calculate a one-sided rejection probability rather than the specified two-sided conditional error.

In a design with several looks, the same definition applies, but the event $A$ includes the complete original stopping and rejection rule. Given interim data, conditional error is the probability—over the future increments—that this rule eventually rejects. If the design has several future looks, this generally requires evaluating the joint probability of crossing a rejection boundary at one or more future looks while respecting any earlier stopping boundaries. A single final-look normal tail is not enough unless the design truly has only one remaining analysis.

Worked Example: Two-Sided Conditional Error

Consider a hypothetical canonical design with one interim look at $t=0.50$ and a final two-sided rule $|Z_2|>1.96$. Suppose $z_1=1.00$, and assume the original design has no interim stopping boundary. The conditional final distribution has mean $\sqrt{0.50}(1.00)=0.7071$ and standard deviation $0.7071$.

The lower-tail and upper-tail probabilities are calculated separately:

\begin{aligned} P_0(Z_2<-1.96\mid Z_1=1.00) &=\Phi\!\left(\frac{-1.96-0.7071}{0.7071}\right) =\Phi(-3.772)\approx 0.000081,\\ P_0(Z_2>1.96\mid Z_1=1.00) &=1-\Phi\!\left(\frac{1.96-0.7071}{0.7071}\right) =1-\Phi(1.772)\approx 0.0382. \end{aligned}

Adding the disjoint tail events gives $\operatorname{CE}(1.00)\approx0.0383$. Most of this conditional probability comes from crossing the upper boundary, but the lower-tail probability is part of the two-sided rejection event and must not be discarded.

Worked Example: A Less Favorable Interim Statistic

Suppose a hypothetical one-sided design has information fraction $t=0.60$, final critical value $c_{\alpha}=1.96$, and interim statistic $z_1=-0.50$. Assume the known-variance canonical model and no early stopping. The conditional mean is $\sqrt{0.60}(-0.50)=-0.3873$; the conditional standard deviation is $\sqrt{0.40}=0.6325$.

\begin{aligned} \operatorname{CE}(-0.50) &=1-\Phi\!\left(\frac{1.96-\sqrt{0.60}(-0.50)}{\sqrt{0.40}}\right)\\ &=1-\Phi\!\left(\frac{2.3473}{0.6325}\right)\\ &=1-\Phi(3.711)\\ &\approx 0.00010. \end{aligned}

Conditional on this interim result, the original design has about a 0.010% null probability of eventual rejection. This is a small probability, not a statement that rejection is impossible. It also does not describe power under a non-null effect; that would require specifying an alternative model.

Assumptions, Diagnostics, and Common Pitfalls

The definition $P_0(A\mid D_1)$ is general, but a numerical formula is only as reliable as the conditional model and the event it represents. Before using the closed form, check that the test statistic follows the stated canonical representation under the null, that $t$ reflects the actual information fraction, and that future increments are independent of the interim data as assumed. Also verify that the critical value and sidedness match the original rejection rule.

If the interim statistic is not enough to determine the conditional distribution of future data, conditioning on $Z_1=z_1$ alone may be insufficient. For example, a nuisance-parameter estimate, an adaptive variance estimate, or a data-dependent information amount may affect the future distribution. In such a setting the conditional error may depend on more of $D_1$, and a formula based only on $z_1$ can be misleading. Diagnostics include comparing the assumed information with the planned and observed information, checking the original boundary and stopping logic, and validating numerical probabilities against simulation under the specified null model.

Conditional error is useful alongside, not instead of, other design summaries. Unconditional Type I error averages over all possible interim outcomes; conditional error describes the original design at a particular observed outcome. Conditional power instead asks about rejection under an alternative effect assumption. Neither an unconditional error calculation nor a conditional-power target supplies the conditional error for a particular interim result.

Common mistake: “The trial’s unconditional Type I error is 2.5%, so the conditional error after every interim result must also be 2.5%.” This is incorrect: unconditional error is an average over interim data, whereas conditional error varies with those data. Calculate $P_0(A\mid D_1)$ using the original design’s rejection event and the appropriate conditional null distribution.

A second common error is to treat the old design’s conditional error as if it were already the Type I error of any revised design. It is a probability defined for the original rejection event and null model. Calculating it does not, by itself, establish the error rate of a modified procedure; that requires a separate justification of the modified decision rule.

Check Your Understanding

Use the one-sided canonical formula where appropriate. Assume a known-variance null model, no early stopping unless a question states otherwise, and report probabilities to about three significant figures.

For $t=0.40$, $z_1=0.80$, and $c_{\alpha}=1.96$, calculate the conditional error.
For $t=0.75$, $z_1=0$, and one-sided critical value $c_{\alpha}=1.645$, calculate the conditional error.
For $t=0.50$, $z_1=1.50$, and $c_{\alpha}=1.96$, calculate the conditional error. Is it larger or smaller than the value in Worked Example 1, and why?
An original design has an interim futility rule that stops the trial without rejection for a particular observed interim result. What is the conditional error for that result under the original rule?
Explain why an unconditional null rejection probability of $\alpha$ does not imply conditional error $\alpha$ at each possible interim value.

Solutions. (1) Here $\sqrt{t}z_1=\sqrt{0.40}(0.80)=0.5060$ and $\sqrt{1-t}=0.7746$. Thus $\operatorname{CE}=1-\Phi((1.96-0.5060)/0.7746)=1-\Phi(1.877)\approx0.0303$. (2) The conditional mean is zero and the standard deviation is $\sqrt{0.25}=0.50$. Hence $\operatorname{CE}=1-\Phi(1.645/0.50)=1-\Phi(3.290)\approx0.00050$. (3) The conditional mean is $\sqrt{0.50}(1.50)=1.0607$, and the standard deviation is $0.7071$. Therefore $\operatorname{CE}=1-\Phi((1.96-1.0607)/0.7071)=1-\Phi(1.272)\approx0.102$. It is larger than 0.0580 in Worked Example 1 because the interim statistic is more favorable to the one-sided rejection boundary, with the same information fraction and critical value. (4) The conditional error is 0: the original rule has stopped without rejection, so eventual rejection under that rule cannot occur. (5) The unconditional probability is the average of the conditional probabilities over the null distribution of interim data. That average can equal $\alpha$ even when individual conditional errors differ from $\alpha$.