Worst-Case Conditional Error Calibration

Why Calibrate Against the Worst Case?

A continuation boundary is often calibrated using the conditional error remaining after an interim analysis. That calibration can depend on nuisance parameters: for example, a variance or a parameter governing the conditional distribution of the future test statistic. If the nuisance value is unknown, calibrating at one estimate may leave some plausible values with a conditional rejection probability above the allowed budget. Worst-case calibration addresses this by requiring the rule to meet the budget for every nuisance value in a specified set.

The introductory material on conditional error functions motivates preserving the conditional error of the original design. Here the focus is narrower: how to replace a nuisance-dependent conditional rejection probability with a uniform bound, how to find the boundary that achieves that bound, and what it means for the bound to be sharp.

Definition: Let $D=d$ denote the interim data, let $\eta$ be a nuisance parameter in a specified set $\mathcal{H}_0$ under the null hypothesis, and let $R_b$ be the event that the adapted continuation rule rejects when its boundary is $b$. The worst-case conditional rejection probability is $$ C_{\sup}(d;b)=\sup_{\eta\in\mathcal{H}_0} P_{\eta}(R_b\mid D=d). $$ It is a conditional error bound uniform over the stated null nuisance-parameter set. The set $\mathcal{H}_0$ and the conditional model must be specified; changing either can change the bound.

For a one-sided upper-tail rule, suppose the trial continues and rejects when a future statistic $Y$ is at least $b$. Let $e(d)$ be the conditional error budget at the observed interim data. The calibration requirement is $C_{\sup}(d;b)\le e(d)$. If the rule also has a stopping branch, this condition applies to the continuation branch; the full rule must account for the stopping branch in the same conditional error calculation.

Normal-Model Calibration

Suppose, under the null hypothesis and for each fixed nuisance value, the conditional distribution is $$ Y\mid D=d,\eta \sim N\bigl(m(d,\eta),s^2(d,\eta)\bigr), $$ where $s(d,\eta)>0$. Assume that this conditional model is valid for the observed interim data and for every $\eta$ in $\mathcal{H}_0$. For a continuous normal distribution, the conditional rejection probability is

C_{\eta}(d;b) =P_{\eta}(Y\ge b\mid D=d) =1-\Phi\!\left(\frac{b-m(d,\eta)}{s(d,\eta)}\right),

where $\Phi$ is the standard normal cumulative distribution function. The worst case is the largest of these probabilities, not necessarily the value at the largest conditional mean: if the standard deviation also varies, a more dispersed distribution can have a larger upper-tail probability even when its mean is smaller.

Result: For a fixed $d$ and a budget $0<e(d)<1$, the smallest boundary that satisfies the normal-model worst-case constraint is $$ b_{\min}(d)=\sup_{\eta\in\mathcal{H}_0} \left\{m(d,\eta)+s(d,\eta)\Phi^{-1}\bigl(1-e(d)\bigr)\right\}, $$ provided the displayed supremum is finite. If it is attained, this boundary gives worst-case conditional rejection probability exactly $e(d)$. If the supremum is not attained, the bound can be approached arbitrarily closely but need not be attained at any nuisance value.

This expression follows by solving the constraint separately for each nuisance value. Because $\Phi$ is increasing, $C_\eta(d;b)\le e(d)$ holds exactly when $$ \frac{b-m(d,\eta)}{s(d,\eta)}\ge \Phi^{-1}\bigl(1-e(d)\bigr). $$ Since $s(d,\eta)>0$, this is equivalent to $b\ge m(d,\eta)+s(d,\eta)\Phi^{-1}(1-e(d))$. The boundary must satisfy this inequality for every $\eta$, so it must be at least the supremum of the right-hand side. Conversely, any boundary at least that supremum satisfies every one of the inequalities. This proves both the calibration formula and its minimality.

When $s(d,\eta)$ is constant and $m(d,\eta)$ is maximized at a value $\eta^\ast$, the worst-case tail occurs at that largest mean. When the standard deviation varies, the relevant quantity for the calibrated boundary is $m+s\Phi^{-1}(1-e)$, not the mean alone. For a non-normal conditional distribution, the same principle applies, but the boundary must be obtained from the relevant conditional quantiles or by directly evaluating the supremum of tail probabilities.

Worked Example: A Common Conditional Variance

Suppose the observed interim statistic is $d=1.2$, the null nuisance set is $\mathcal{H}_0=[-0.2,0.2]$, and the hypothetical conditional model is $$ Y\mid D=1.2,\eta \sim N\bigl(0.6+0.5\eta,\;0.75\bigr). $$ The conditional standard deviation is $\sqrt{0.75}\approx0.8660$. Consider first the boundary $b=1.96$. The conditional mean increases with $\eta$, so the largest mean, and therefore the largest upper-tail probability under this common variance, occurs at $\eta=0.2$. There the mean is $0.6+0.5(0.2)=0.7$, and $$ C_{\sup}(1.2;1.96) =1-\Phi\!\left(\frac{1.96-0.7}{\sqrt{0.75}}\right) =1-\Phi(1.4549) \approx 0.0728. $$ Thus this boundary has worst-case conditional rejection probability about 0.0728.

At the other endpoint, $\eta=-0.2$, the mean is $0.6+0.5(-0.2)=0.5$, giving $$ 1-\Phi\!\left(\frac{1.96-0.5}{\sqrt{0.75}}\right) =1-\Phi(1.6859) \approx 0.0459. $$ The endpoint comparison is consistent with the monotonicity of the mean and the constant variance.

Now calibrate to a conditional error budget $e(1.2)=0.08$. The boundary formula gives $$ b_{\min}=0.7+\sqrt{0.75}\,\Phi^{-1}(0.92) \approx 0.7+0.8660(1.4051) \approx 1.917. $$ At $\eta=0.2$, this boundary gives conditional rejection probability 0.08; for every smaller $\eta$, the mean is smaller and the conditional variance is unchanged, so the probability is no larger. The boundary 1.96 is therefore conservative for this budget, while approximately 1.917 is the smallest boundary that meets it.

When the Worst Case Is Not the Largest Mean

If nuisance values affect both the mean and the spread, ordering them by their means can give the wrong answer. The worst-case probability at a proposed boundary must be found by comparing the full conditional tail probabilities. Equivalently, for a normal model and a fixed budget, compare the boundary requirements $m+s\Phi^{-1}(1-e)$.

Worked Example: A Larger Variance Governs Calibration

Consider two possible null nuisance values, $\eta_A$ and $\eta_B$, with the following hypothetical conditional distributions at the observed interim data: $$ Y\mid D=d,\eta_A\sim N(1.0,0.2^2), \qquad Y\mid D=d,\eta_B\sim N(0.8,0.8^2). $$ At $b=1.5$, the mean is higher under $\eta_A$, but the conditional tail probabilities are $$ C_{\eta_A}(d;1.5)=1-\Phi\!\left(\frac{1.5-1.0}{0.2}\right) =1-\Phi(2.5)\approx0.0062, $$ and $$ C_{\eta_B}(d;1.5)=1-\Phi\!\left(\frac{1.5-0.8}{0.8}\right) =1-\Phi(0.875)\approx0.1908. $$ Thus $\eta_B$, not the value with the larger mean, is the worst case at this boundary.

For a budget of 0.10, $\Phi^{-1}(0.90)\approx1.2816$. The boundary requirements are $$ b_A=1.0+0.2(1.2816)\approx1.256, \qquad b_B=0.8+0.8(1.2816)\approx1.825. $$ The calibrated boundary for these two nuisance values is their maximum, approximately $1.825$. It is governed by $\eta_B$, whose larger standard deviation makes its upper tail harder to control.

Sharpness, Error Control, and Diagnostics

A worst-case bound is sharp when it is attained by at least one nuisance value in the specified set. For the normal calibration formula, if the supremum defining $b_{\min}(d)$ is attained at $\eta^\ast$, then at $b=b_{\min}(d)$ the conditional rejection probability at $\eta^\ast$ is exactly $e(d)$. No smaller boundary can satisfy the uniform requirement: reducing $b$ raises the upper-tail probability at $\eta^\ast$ above the budget. If the supremum is not attained, there may be no single least-favourable nuisance value, even though values in the set can make the conditional probability arbitrarily close to the supremum.

Sharpness is relative to the model and nuisance set used. A bound can be unnecessarily conservative if the set includes nuisance values ruled out by justified design information, or if the conditional model is too broad. Conversely, narrowing the set without a defensible basis can invalidate the claimed error guarantee. The nuisance set should be specified before applying the rule, and all plausible values under the null that the guarantee is meant to cover should be included.

Conditional control also connects to unconditional type I error control. Suppose that for each null nuisance value $\eta$, the rule satisfies $P_\eta(R\mid D)\le e(D)$ for every relevant interim outcome, and the original conditional error function obeys $E_\eta[e(D)]\le\alpha$. The law of total probability then gives $$ P_\eta(R)=E_\eta\!\left[P_\eta(R\mid D)\right] \le E_\eta[e(D)] \le\alpha. $$ The guarantee depends on conditioning on the interim information used by the rule and on using a conditional error budget valid under the same null model. A pointwise worst-case bound does not repair a misspecified conditional distribution or an invalid budget.

Useful checks include verifying that $s(d,\eta)>0$, that the nuisance set is the intended null set, and that the computed supremum covers the whole set rather than only a convenient selection of values. With varying variance, examine the standardized threshold $(b-m)/s$ or the boundary requirement $m+s\Phi^{-1}(1-e)$; checking only the conditional mean is inadequate. In numerical work, compare the calibrated boundary by substituting it back into the conditional tail calculation at the maximizing value and at other plausible values. A coarse grid can help locate a maximum, but it does not prove that a maximum between grid points has not been missed.

The principal alternatives answer different questions. A plug-in calibration substitutes one nuisance estimate and controls the conditional probability only at that value; it does not generally provide a uniform guarantee. Averaging conditional probabilities over a distribution on nuisance values yields an average, not a worst-case bound, unless the model explicitly treats the nuisance value as random and that averaging is the intended criterion. Worst-case calibration is robust to uncertainty within the stated set, at the cost of potentially using a more stringent boundary. Its limitations are that it can lose efficiency when the set is broad and that its protection is only as credible as the conditional model and the set itself.

Worked Example: A Sharp Boundary at the Budget

Return to the model $Y\mid D=1.2,\eta\sim N(0.6+0.5\eta,0.75)$, with $\eta\in[-0.2,0.2]$, and set the budget to 0.05. The maximum mean is 0.7 at $\eta^\ast=0.2$. Since $\Phi^{-1}(0.95)\approx1.6449$, the calibrated boundary is $$ b_{\min}=0.7+\sqrt{0.75}(1.6449) \approx0.7+1.4245 \approx2.1245. $$ At $\eta^\ast=0.2$, the standardized threshold is approximately $1.6449$, so the conditional rejection probability is 0.05. For every $\eta<0.2$, the conditional mean is smaller and the variance is the same, so the tail probability is below 0.05. The worst-case bound is sharp because it is attained at $\eta^\ast$. Any boundary strictly below $0.7+\sqrt{0.75}\,\Phi^{-1}(0.95)$ gives a strictly larger tail probability at that nuisance value, so it cannot meet the uniform 0.05 requirement.

Common Mistakes

Using the estimated nuisance value as if it were the worst case. A plug-in value may be useful for another purpose, but it does not establish a bound over the whole nuisance set. Maximize over the stated set.
Choosing the largest mean when variance also changes. Upper-tail probabilities depend on both mean and standard deviation. Compare the tail probabilities or, for normal calibration, the full boundary requirements.
Calling every conservative bound sharp. A bound is sharp only if its supremum is attained; otherwise it may be approached but not attained.
Changing the nuisance set after calibration without recalculating. The supremum, and thus the boundary, can change when the set changes.
Interpreting conditional control as an automatic guarantee under any model. It yields the intended unconditional control only when the conditional model and budget are valid for each null nuisance value.

Key takeaway: Calibrate a continuation rule by bounding its conditional rejection probability over every nuisance value in the prespecified null set. In a normal model, the smallest valid upper-tail boundary is the supremum of $m+s\Phi^{-1}(1-e)$; it is sharp when that supremum is attained.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

For an upper-tail rule, what quantity is maximized in worst-case conditional error calibration?
In a normal conditional model with common standard deviation, when is the largest conditional mean also the worst case?
Why can the nuisance value with the largest mean fail to have the largest upper-tail probability when variances differ?
What makes a calibrated boundary sharp, and what happens if the nuisance supremum is not attained?

Solutions. (1) Maximize $P_\eta(Y\ge b\mid D=d)$ over the entire specified null nuisance set. (2) With a common positive standard deviation, a larger mean shifts the normal distribution to the right, increasing its upper-tail probability at any fixed boundary; thus a nuisance value attaining the largest mean is worst. (3) The tail depends on the standardized threshold $(b-m)/s$, so a smaller mean combined with a sufficiently larger standard deviation can produce a larger tail, as in the second worked example. (4) The boundary is sharp when some nuisance value attains the supremum and its conditional rejection probability equals the budget. If the supremum is not attained, probabilities may approach the bound without any nuisance value attaining it.