Least-Favorable Configurations in Error Proofs

Why a Worst-Case Null Configuration Matters

A type I error guarantee is a statement about all parameter values covered by the null hypothesis, not just one convenient value. In a composite null, the probability of rejection can vary across the null parameter space. A proof may become much shorter if it is possible to identify a configuration at which that probability is largest. But the reduction is valid only when the proposed configuration really bounds the error everywhere else in the null.

This issue arises in adaptive trials because the final rejection rule can depend on interim data, selected treatments, or a data-dependent analysis path. Those features do not automatically invalidate a least-favorable-configuration argument. They do mean that the argument must apply to the full rejection event under the complete design, including its adaptation rule.

Definition: Let $\theta$ denote the parameter and let $\Theta_0$ be the parameter space specified by a null hypothesis $H_0$. For a test with rejection event $R$, its type I error at $\theta\in\Theta_0$ is $P_\theta(R)$. A least-favorable null configuration is a value $\theta^\star\in\Theta_0$ that maximizes this probability: $$ P_{\theta^\star}(R)=\sup_{\theta\in\Theta_0}P_\theta(R). $$ If the supremum is not attained, a least-favorable configuration can instead describe a sequence of null parameter values whose rejection probabilities approach the supremum. A test has level $\alpha$ if $\sup_{\theta\in\Theta_0}P_\theta(R)\leq\alpha$.

Finding a least-favorable configuration is not the same as choosing a representative null value. The required claim is an inequality: the rejection probability at the proposed worst case must be at least as large as it is at every other null value. If the proof establishes that inequality, evaluating the probability at the worst case is sufficient. If it does not, a calculation at one point alone cannot establish uniform control.

When a Boundary Reduction Is Justified

For a one-sided null such as $H_0:\theta\leq 0$, the boundary value $\theta=0$ is often least favorable for a test that rejects for unusually large values of a statistic. The reason is that larger $\theta$ shifts the statistic toward the rejection region. The following coupling argument makes the logic explicit.

Suppose $X_\theta=\theta+\varepsilon$, where the distribution of $\varepsilon$ does not depend on $\theta$, and the test rejects when $X_\theta>c$. For any $\theta\leq 0$, use the same value of $\varepsilon$ to construct $X_\theta$ and $X_0$. Then $X_\theta\leq X_0$, so $X_\theta>c$ implies $X_0>c$. Taking probabilities gives the required bound:

P_\theta(X_\theta>c)\leq P_0(X_0>c), \qquad \theta\leq 0.

The conclusion depends on the stated model and on the rejection event being increasing in $X_\theta$. It is not a general fact that the null boundary is worst. A two-sided rejection region, a parameter-dependent variance, or an adaptation that changes the distribution can require a different argument.

Worked Example: A One-Sided Gaussian Test

Suppose a hypothetical trial estimates a treatment effect $\theta$ in units of points on a prespecified symptom scale. Assume the test statistic has distribution $X\sim N(\theta,1)$, where the variance is known to be 1, and reject $H_0:\theta\leq 0$ when $X>1.645$. Find the largest type I error over the null.

Model and assumptions. The normal model and known variance are assumed to hold for every $\theta\leq0$. The critical value is fixed, and rejection is the increasing event $X>1.645$. Write $X_\theta=\theta+\varepsilon$, with $\varepsilon\sim N(0,1)$. The coupling argument shows that the rejection probability cannot decrease as $\theta$ increases, so the maximum over $\theta\leq0$ occurs at $\theta=0$.

Calculation at the least-favorable value. At $\theta=0$, $X\sim N(0,1)$, so $$ P_0(X>1.645)=1-\Phi(1.645)\approx 0.0500. $$

Thus the test has type I error approximately 0.0500, or 5.00%, uniformly over $\theta\leq0$. The boundary calculation is sufficient here because the model and rejection event establish the inequality for every null value, not merely because zero is the boundary.

Least-Favorable Configurations for Multiple Nulls

With multiple hypotheses, the relevant error event depends on which hypotheses are true. For a parameter vector $\theta$, let $I_0(\theta)$ be the set of true null hypotheses. The familywise error rate at $\theta$ is the probability of rejecting at least one hypothesis in $I_0(\theta)$. Strong control requires this probability to be at most $\alpha$ for every parameter configuration, including configurations where some hypotheses are false.

A least-favorable configuration for the global null, where all hypotheses are true, does not by itself prove strong control. For each possible true-null set, the proof must either bound its error directly or show that the global-null bound also covers it. The monotonicity argument must concern the error event for the true hypotheses, not simply the chance of making any rejection.

Worked Example: Two Adaptive Claims and a Global-Null Worst Case

Consider a hypothetical design that may select whichever of two treatment claims has the larger observed score. Let $Z_j=\theta_j+\varepsilon_j$, for $j=1,2$, where the errors are independent standard normal variables and the variances are known. The fixed rule rejects claim $j$ if $Z_j>1.960$. Each claim has null $H_{0j}:\theta_j\leq0$. First consider the global null $\theta_1\leq0,\theta_2\leq0$.

Why the boundary is worst. Couple each score at $(\theta_1,\theta_2)$ to the score at $(0,0)$ using the same pair of errors. If both effects are nonpositive, each $Z_j$ is no larger than its corresponding score at $(0,0)$. Therefore the event that at least one claim is rejected at $(\theta_1,\theta_2)$ is contained in the event that at least one is rejected at $(0,0)$. The global-null rejection probability is maximized at the joint boundary.

Calculation. At $(0,0)$, each score exceeds 1.960 with probability 0.0250. Independence gives $$ P_{(0,0)}(Z_1>1.960\text{ or }Z_2>1.960) =1-(1-0.0250)^2 =1-0.950625 =0.049375. $$

The global-null familywise error is therefore 0.0494, or 4.94%. If only the first null is true, the familywise error is $P(Z_1>1.960)\leq0.0250$; rejecting the second, false null does not count as a type I error. The calculation thus also bounds the one-true-null configuration in this example. A selection rule that reports the larger score does not change the event “at least one score exceeds the cutoff.” If selection instead changed the cutoff or the distribution, this containment argument would have to be re-established for that complete rule.

Nuisance Parameters Must Be Included

A null hypothesis may restrict a parameter of interest while leaving nuisance parameters unrestricted or only partly restricted. A reduction to the boundary of the parameter of interest is then not enough unless the error bound also covers the nuisance parameters. The least-favorable configuration can involve a boundary value for the effect and a different boundary or interior value for a nuisance parameter.

Worked Example: The Effect Boundary Is Not the Whole Worst Case

Suppose a hypothetical test observes $X\sim N(\theta,\sigma^2)$, rejects when $X>1.645$, and tests $H_0:\theta\leq0$. Assume the nuisance standard deviation is unknown but restricted by $1\leq\sigma\leq2$, in the measurement units of $X$. Find the largest null rejection probability.

Model and assumptions. The normal model is assumed for every allowed pair $(\theta,\sigma)$. For fixed $\sigma$, the upper-tail probability increases with $\theta$, so its largest value over $\theta\leq0$ occurs at $\theta=0$. At that value, $$ P_{0,\sigma}(X>1.645)=1-\Phi\left(\frac{1.645}{\sigma}\right). $$

For $1\leq\sigma\leq2$, the ratio $1.645/\sigma$ decreases as $\sigma$ increases. Since $\Phi$ is increasing, the upper-tail probability increases with $\sigma$. The least-favorable configuration is therefore $(\theta,\sigma)=(0,2)$, and $$ P_{0,2}(X>1.645)=1-\Phi(0.8225)\approx 0.2054. $$

The worst-case type I error is approximately 20.5%, not 5.00%. Checking the effect boundary while silently fixing $\sigma=1$ would miss the nuisance value that maximizes rejection. This example also shows why a cutoff calibrated under one variance cannot be presumed to control error over a larger nuisance-parameter space.

Adaptive Rules, Diagnostics, and Limitations

For an adaptive trial, define $R$ using the entire prespecified procedure: interim data, any treatment or population selection, changes in sample size or allocation, and the final rejection decision. The least-favorable-configuration question is then whether $P_\theta(R)$ is bounded by its value at a proposed configuration for every $\theta$ in the relevant null space. A monotonicity argument based only on the final test statistic may be inadequate if the adaptation changes the sampling distribution or the critical value.

Useful checks include verifying that the parameter space includes every nuisance value allowed under the null, that the same adaptation rule is used in each probability being compared, and that the ordering of outcomes really implies an ordering of rejection events. For a simulated error assessment, examine the error rate across a grid of null and nuisance configurations, including boundaries and plausible interior points. Such a grid can reveal a suspicious reduction, but it is not a mathematical proof of a uniform bound unless the parameter space and numerical error are handled appropriately.

Common approaches to finding a worst case include direct maximization over the null space, an analytic monotonicity or coupling argument, and a valid probability bound that holds for every null value. These approaches are not interchangeable without justification. For example, a boundary argument is efficient when stochastic ordering is established; direct maximization is more appropriate when the error probability is nonmonotone or has several nuisance parameters. A conservative uniform bound may avoid locating an exact maximum, at the cost of being less sharp.

Common mistake: “The null is $\theta\leq0$, so it is enough to calculate error at $\theta=0$.” This conclusion does not follow from the form of the null alone. It follows only if the rejection probability is maximized there, as shown under the model and rejection rule. With nuisance parameters, an adaptation-dependent distribution, or a nonmonotone rejection event, the maximum may occur elsewhere. The correction is to state the full null parameter space and prove the required bound over that space.

A least-favorable configuration can also be nonunique, and the supremum may be approached without being attained. Moreover, the configuration that is worst for one true-null subset need not be worst for another. The method simplifies an error proof; it does not remove the need to define the error event and the parameter space precisely.

Check Your Understanding

Answer each question, then compare with the brief solutions below.

In the Gaussian model $X\sim N(\theta,1)$ with rejection when $X>c$, $c>0$, and null $\theta\leq0$, why is $\theta=0$ least favorable?
For the two independent tests in the worked example, what is the probability of rejecting at least one claim when both effects equal zero?
In the nuisance-parameter example, why does evaluating the error at $(\theta,\sigma)=(0,1)$ fail to establish a 5% uniform bound?
Why does control at the global null not automatically establish strong familywise error control?

Solutions. (1) Using the same error $\varepsilon$ for each $\theta$, $X_\theta=\theta+\varepsilon\leq X_0=\varepsilon$ whenever $\theta\leq0$. Hence $\{X_\theta>c\}$ is contained in $\{X_0>c\}$, so its probability is no greater. (2) The probability is $1-(1-0.0250)^2=0.049375$, or 4.94%. (3) The allowed nuisance range includes $\sigma=2$, where the rejection probability at $\theta=0$ is approximately 0.2054; the value at $\sigma=1$ does not bound it. (4) Under partial nulls, the error event includes rejections only among the true hypotheses, and its probability must be bounded for every possible true-null subset.