Composite Nulls And Nuisance Parameters

Why the Null Parameter Matters

Under a simple null hypothesis, a conditional error function describes the remaining probability of rejection after conditioning on interim information. With a composite null, the null hypothesis includes multiple parameter values. If the distribution of future data depends on a nuisance parameter, the conditional error can differ across those values—even when the observed interim data are held fixed. A conditional error calculated at one convenient parameter value is therefore not automatically a valid bound for the whole null.

The key requirement is uniformity: an adapted test must control its type I error for every parameter value in the null hypothesis. This tutorial derives a sufficient conditional condition for that control, explains when it may be conservative, and distinguishes it from substituting a nuisance estimate or checking only a boundary value.

Definition: Let the parameter be $\theta$, with null parameter space $\Theta_0$, and let $\mathcal{F}$ denote the information available at an interim analysis. If $\phi$ is the original test’s rejection indicator, its parameter-specific conditional error is $$ a_\theta(\mathcal{F})=E_\theta(\phi\mid\mathcal{F}) =P_\theta(\text{original test rejects}\mid\mathcal{F}),\qquad \theta\in\Theta_0. $$ It is the conditional probability of rejection under the specified null parameter value, given the interim information. It need not be the same function of the observed interim data for every $\theta$.

The original test is assumed to have uniform level $\alpha$, meaning $E_\theta(\phi)\leq\alpha$ for every $\theta\in\Theta_0$. An adapted procedure may use the interim information to change how the study continues, but it must account for the conditional distribution of its future data under each null parameter value.

A Uniform Preservation Condition

Write $q_\theta(\mathcal{F})$ for the adapted procedure’s conditional probability of ultimately rejecting, under parameter $\theta$, given the interim information. This conditional probability includes any future observations and any randomization used by the procedure. A sufficient preservation condition is that, for every $\theta\in\Theta_0$,

q_\theta(\mathcal{F})\leq a_\theta(\mathcal{F}) \quad P_\theta\text{-almost surely}.

The condition is required separately under every null parameter value. The almost-sure qualification matters: conditional probabilities are defined up to events of probability zero, and the condition need not constrain an interim outcome that has probability zero under that particular parameter value. The operational adaptation rule, however, must be specified so that the required inequality holds under each null model where the interim information can occur.

Theorem: If the original test has uniform level $\alpha$ on $\Theta_0$ and the adapted test satisfies $q_\theta(\mathcal{F})\leq a_\theta(\mathcal{F})$ almost surely for every $\theta\in\Theta_0$, then the adapted test also has type I error at most $\alpha$ for every $\theta\in\Theta_0$.

The argument is the conditional expectation identity. It does not require the conditional error to be constant across null parameter values. It does require the inequality to hold for the parameter value generating the data.

P_\theta(\text{adapted test rejects}) =E_\theta\{q_\theta(\mathcal{F})\} \leq E_\theta\{a_\theta(\mathcal{F})\} =E_\theta(\phi) \leq\alpha,\qquad \theta\in\Theta_0.

The third expression equals the original rejection probability by the defining property of conditional expectation. The final inequality is the original test’s uniform level guarantee. Applying this argument for each null value establishes uniform type I error control for the adapted test.

Worked Examples With an Unknown Variance

Worked Example: Computing Conditional Error Across Nuisance Values

Suppose two independent stage statistics satisfy $X\sim N(0,\sigma^2)$ and $Y\sim N(0,\sigma^2)$ under $H_0$, where the nuisance parameter $\sigma$ lies in $[0.8,1.2]$. The original test rejects when $X+Y>c$. Find a cutoff giving uniform one-sided level $0.05$, then calculate its conditional error at $X=1$ for the two endpoint variance values.

Under the null, $X+Y\sim N(0,2\sigma^2)$. The probability of exceeding a fixed positive cutoff increases with $\sigma$, so the largest null rejection probability occurs at $\sigma=1.2$. Choose the cutoff to give probability $0.05$ there:

c=z_{0.95}\sqrt{2}(1.2) =1.6448536\sqrt{2}(1.2) \approx 2.79141.

At $\sigma=1.2$, the standardized cutoff is $c/(\sqrt{2}\,1.2)=1.6448536$, giving a rejection probability of $0.05$. For smaller $\sigma$, the standardized cutoff is larger, so the rejection probability is below $0.05$. Thus the original test has uniform level $0.05$ over the stated nuisance range.

Given $X=x$, independence gives $Y\sim N(0,\sigma^2)$. Therefore the conditional error is $$ a_\sigma(x)=P_\sigma(Y>c-x\mid X=x) =1-\Phi\left(\frac{c-x}{\sigma}\right). $$ At $x=1$, $c-x\approx1.79141$. For $\sigma=0.8$, the standardized threshold is $1.79141/0.8\approx2.23926$, so $a_{0.8}(1)\approx0.01257$. For $\sigma=1.2$, it is $1.79141/1.2\approx1.49284$, so $a_{1.2}(1)\approx0.0677$. The same interim value thus has substantially different conditional errors under different null nuisance values.

These are conditional probabilities, not unconditional type I error rates. The fact that the conditional error at $x=1$ exceeds $0.05$ for $\sigma=1.2$ does not contradict the original test’s level: other interim values have different conditional errors, and their average under that null parameter is $0.05$.

Worked Example: A Common Conservative Bound

Continue with the two-stage model and suppose the adapted procedure, at interim value $x=1$, has conditional rejection probability $0.01$, the same under every $\sigma$ in the null range. Does this satisfy the conditional preservation condition at $x=1$?

Here $c-x\approx1.79141>0$. For a positive threshold, $1-\Phi((c-x)/\sigma)$ increases as $\sigma$ increases: increasing $\sigma$ makes the positive standardized threshold smaller. Consequently the smallest conditional error in $[0.8,1.2]$ occurs at $\sigma=0.8$. The calculation above gives $$ 0.01\leq a_{0.8}(1)\approx0.01257 \leq a_\sigma(1),\qquad 0.8\leq\sigma\leq1.2. $$

Thus a conditional rejection probability of $0.01$ is below the original conditional error for every null variance at this interim value. More generally, where a common conditional bound is needed, define the conservative envelope $$ a_*(x)=\inf_{\sigma\in[0.8,1.2]}a_\sigma(x). $$ A rule with conditional rejection probability no greater than $a_*(x)$ satisfies the pointwise requirement for every variance, provided the condition is met at all interim outcomes relevant to each null model. The infimum, not the supremum, is the direction that gives a bound below every parameter-specific conditional error.

This example checks the inequality at one interim value to illustrate its calculation; a complete design must check the rule over its full interim information space. A bound that works at $x=1$ alone does not establish preservation elsewhere.

Worked Example: Why a Plug-In Variance Can Fail

At $X=1$, suppose an analyst estimates $\sigma$ as $0.8$, calculates the original conditional error as $0.01257$, and then allows the adapted procedure to reject with conditional probability $0.02$. Is this justified uniformly over the null variance range?

No. At the estimated value, the proposed probability already exceeds the calculated conditional error: $$ 0.02> a_{0.8}(1)\approx0.01257. $$ Therefore even the inequality at the plug-in value fails. More generally, replacing the unknown $\sigma$ by an estimate and checking only the resulting conditional error cannot establish the required inequalities for all $\sigma\in[0.8,1.2]$. In this example, the conditional error at $\sigma=1.2$ is larger, about $0.0677$, but that does not repair the failure at $\sigma=0.8$.

The conclusion is specifically that conditional preservation has not been demonstrated; the calculation alone does not determine the adapted procedure’s unconditional type I error. That error would require averaging its conditional rejection probability under the joint distribution at each null parameter value. Uniform conditional preservation is a sufficient route to control it, not a claim that every failure of the condition must inflate the unconditional error.

Composite Mean Nulls and the Boundary Trap

Worked Example: A Boundary Value Does Not Bound Every Conditional Error

Let $X$ and $Y$ be independent with $X\sim N(\mu,1)$ and $Y\sim N(\mu,1)$, and test the composite null $H_0:\mu\leq0$. The original test rejects if $X+Y>c$, where $c=z_{0.95}\sqrt{2}\approx2.32617$. Verify its level, then examine the conditional error at $X=0$.

Under a null value $\mu$, $X+Y\sim N(2\mu,2)$. Its rejection probability is $$ P_\mu(X+Y>c) =1-\Phi\left(\frac{c-2\mu}{\sqrt{2}}\right). $$ This probability increases as $\mu$ increases, so its maximum over $\mu\leq0$ occurs at the boundary $\mu=0$, where it is $0.05$. The original test therefore has level $0.05$ for the composite null.

Conditionally on $X=0$, $Y\sim N(\mu,1)$, and the parameter-specific conditional error is $$ a_\mu(0)=1-\Phi(c-\mu). $$ At the boundary $\mu=0$, this is $1-\Phi(2.32617)\approx0.01000$. At $\mu=-1$, it is $1-\Phi(3.32617)\approx0.00044$, and it approaches zero as $\mu$ decreases without bound. Thus the boundary conditional error is not a lower bound for all conditional errors; the common lower envelope over this null is zero at $X=0$.

The unconditional rejection probability is maximized at the boundary in this example, but that does not make the boundary conditional error a uniform conditional bound. The two statements concern different probabilities: one averages over $X$, while the other conditions on its observed value.

Diagnostics, Alternatives, and Limitations

For each proposed adaptation, identify the full null parameter space first, then write the parameter-specific conditional error under that null. The following checks help expose a nonuniform argument:

Check the nuisance dependence. Determine whether the conditional distribution of future data depends on nuisance parameters. If it does, retain those parameters in the conditional error until a valid uniform argument removes them.
Check the inequality in the right direction. To preserve conditional error, the adapted conditional rejection probability must be no greater than the original conditional error under each relevant null value. An upper envelope of the original errors generally does not provide that bound.
Check all relevant interim values. A numerical comparison at one observed value is illustrative, not a proof over the full interim information space. Analytic bounds, optimization, or carefully validated computation may be needed.
Check the conditional model. Independence, the distribution and variance of future outcomes, and any changes to the sampling plan must match the model used to calculate the conditional probability.

A common robust construction is to use a single bound no larger than the parameter-specific conditional error for every null value. In a model with a common representation on the relevant interim outcomes, the pointwise infimum over $\theta\in\Theta_0$ provides such a bound. It can be very conservative or even zero, as the composite mean example shows. Conditional probabilities are only defined up to parameter-specific probability-zero sets, so an infimum of arbitrary versions should not be used without attention to their common support; the operative requirement is the inequality under each null model.

The main alternative is to evaluate the adapted procedure’s unconditional rejection probability directly under each $\theta\in\Theta_0$, and then verify that the supremum over the null space is at most $\alpha$. This may be less conservative because it does not require pointwise conditional preservation, but it requires correctly modeling the full adapted procedure and controlling the maximum over nuisance values. A parameter-specific conditional calculation can also be appropriate when the nuisance value is known; it is not a uniform solution when that value is unknown.

The conditional method is sufficient, not necessary: an adaptation might satisfy uniform type I error control without satisfying the pointwise conditional inequality. Conversely, using a plug-in nuisance estimate, checking only one favourable null value, or invoking a least-favourable boundary for an unconditional test does not by itself establish conditional preservation. These distinctions matter because nuisance uncertainty can affect the future-data distribution even after interim information has been observed.

Key takeaway: Under a composite null, conditional error is generally a family $a_\theta(\mathcal{F})$, not a single number. Uniform preservation requires the adapted conditional rejection probability to be bounded by the appropriate original conditional error for every null parameter value; averaging then carries the original uniform level guarantee to the adapted test.

Check Your Understanding

Answer each question, then compare with the solutions below.

State the conditional preservation inequality and explain why it must hold for every null parameter value.
In the unknown-variance example, why is the conditional error at $X=1$ smaller for $\sigma=0.8$ than for $\sigma=1.2$?
Why does a least-favourable boundary for an unconditional test not necessarily give a lower bound on conditional error?
What does failure of the conditional preservation inequality establish, and what does it not establish by itself?

Solutions. (1) The requirement is $q_\theta(\mathcal{F})\leq a_\theta(\mathcal{F})$, $P_\theta$-almost surely, for every $\theta\in\Theta_0$. The parameter affects the conditional distribution, and the type I error guarantee must hold under every null value. (2) At $X=1$, the remaining threshold $c-X$ is positive. Increasing $\sigma$ reduces its standardized value, increasing the normal upper-tail probability. (3) Unconditional rejection probability averages conditional errors over the interim-data distribution; conditioning on a particular interim value changes the comparison, as the example with $\mu\leq0$ demonstrates. (4) It shows that this sufficient conditional preservation argument has not been established. It does not, without further calculation, show that unconditional type I error exceeds $\alpha$.