Why the nuisance parameter matters
A null hypothesis may leave parts of the data-generating model unspecified. For example, a null of no treatment difference can still allow many baseline event rates, variances, or response distributions. Such an unspecified model feature is a nuisance parameter. A test that controls Type I error for one convenient value of that parameter need not control error for the other values allowed by the null.
This issue persists in an adaptive trial. Interim data can determine which test, endpoint, or analysis is used next. The error guarantee must cover the resulting procedure under every parameter value in the null model, not just under a parameter estimate or a selected reference value. The central operation is therefore a supremum: evaluate the rejection probability across the null parameter space and bound its largest value.
The word “supremum” is important. It means the least upper bound across all allowed values; the largest probability may or may not be attained at a particular value. If the null also includes a parameter of interest taking multiple values, the same definition takes the supremum over the entire null parameter space. In the examples below, the null is indexed only by a nuisance parameter to keep the calculations focused.
A direct calculation over the null model
Consider a null statistic \(X\) with \(X\sim N(\theta,1)\), where the nuisance parameter satisfies \(0\leq\theta\leq1\). Suppose the rule rejects for large values of \(X\). The rejection probability at a cutoff \(c\) is \(P_\theta(X>c)\). Because this probability increases with \(\theta\), calibrating the cutoff at \(\theta=0\) would not protect the Type I error rate at \(\theta=1\).
Worked Example: Calibrating at the Largest Null Tail Probability
Suppose the desired Type I error level is \(\alpha=0.05\), and the test rejects when \(X>c\). Find a cutoff that controls error for every \(\theta\in[0,1]\).
Under the stated model, \(X-\theta\sim N(0,1)\), so $$ P_\theta(X>c)=1-\Phi(c-\theta), $$ where \(\Phi\) is the standard normal cumulative distribution function. This probability increases as \(\theta\) increases: increasing \(\theta\) decreases \(c-\theta\), which increases its upper-tail probability. Thus its supremum over the null interval occurs at \(\theta=1\).
To make the largest rejection probability equal to 0.05, choose \(c\) so that $$ P_{\theta=1}(X>c)=0.05, \qquad c=1+z_{0.95}=1+1.6449=2.6449. $$ At \(\theta=1\), the rejection probability is \(1-\Phi(1.6449)=0.0500\). At \(\theta=0\), it is \(1-\Phi(2.6449)\approx0.0041\). Since the probability increases throughout \([0,1]\), the test has size 0.05 and controls Type I error for every allowed nuisance value.
This calculation relies on the specified normal model, unit variance, known parameter range, and upper-tail rejection rule. If the variance or the range of \(\theta\) were different, the calculation would need to use that model and parameter space instead.
The example illustrates an important distinction: a conservative test can have substantially less than \(\alpha\) error at some nuisance values while still having size \(\alpha\). The guarantee concerns the largest null rejection probability, not an average across null parameter values.
Conditional error bounds after adaptation
Let \(S\) denote interim data and let \(A(S)\) be a measurable adaptation rule that selects a final analysis or test. Measurability means, in this setting, that the selected action and resulting rejection event are well-defined functions of the observed data and any explicitly randomized decisions. For each nuisance value \(\lambda\), define the conditional rejection probability $$ e_\lambda(s,a)=P_{0,\lambda}(R\mid S=s,\ A=a). $$ Conditional probabilities are understood under the specified null model. If conditioning on \(S=s\) leaves a conditional distribution that depends on \(\lambda\), that dependence must be retained.
A sufficient route to a uniform guarantee is to bound the conditional error uniformly over nuisance values and actions. For example, if \(e_\lambda(s,a)\leq c(s)\) for every allowed \(\lambda\), every relevant interim value \(s\), and every action \(a\), and if \(E_{0,\lambda}[c(S)]\leq\alpha\) for every \(\lambda\), then the adapted procedure controls Type I error. The conditional bound must apply to the action actually chosen; it cannot be established only for a fixed action and then assumed to hold for an unexamined selection rule.
The justification uses the law of total probability, not an assumption that the interim data are irrelevant. For each fixed \(\lambda\), take expectations of the conditional probability:
Because this inequality holds separately for every \(\lambda\), its left side remains at most \(\alpha\) after taking the supremum over \(\lambda\). If the adaptation is described explicitly as \(A(S)\), the conditional probability in this argument is the one for the selected action, \(e_\lambda(S,A(S))\). A useful, stronger check is to verify a common bound for every action the rule could select.
Worked Example: Selecting a Test Using Interim Data
Let interim data satisfy \(S\sim N(\theta,1)\), and let an independent final statistic \(W\sim N(\theta,1)\) under the null, with \(0\leq\theta\leq1\). An adaptation selects test 1 if \(S\geq0\), and test 2 otherwise. Test 1 rejects if \(W>2.6449\); test 2 rejects if \(W>3.6449\). Check the Type I error guarantee for this data-dependent choice.
Independence of \(S\) and \(W\) under each null value means that, conditional on \(S=s\), \(W\) still has distribution \(N(\theta,1)\). Thus the conditional rejection probabilities are $$ e_\theta(s,1)=1-\Phi(2.6449-\theta), \qquad e_\theta(s,2)=1-\Phi(3.6449-\theta). $$ For \(0\leq\theta\leq1\), each probability increases with \(\theta\). At the largest allowed value, the two bounds are $$ e_1(s,1)=1-\Phi(1.6449)=0.0500, \qquad e_1(s,2)=1-\Phi(2.6449)\approx0.0041. $$ Therefore, for every interim value \(s\) and every allowed \(\theta\), the selected test has conditional rejection probability at most 0.05. Taking expectations over \(S\) gives \(P_{0,\theta}(R)\leq0.05\) for every \(\theta\in[0,1]\), so the supremum is at most 0.05. This conclusion depends on the stated independence and normal models; without them, these conditional probabilities would have to be recalculated.
Why averaging over nuisance values is not enough
A frequentist Type I error guarantee is a bound at each parameter value in the null, summarized by the supremum. It is not a bound averaged over a distribution assigned to the nuisance parameter. Such an average can be useful for a different inferential goal, but it does not establish the stated uniform guarantee.
Worked Example: A Safe Average Can Hide an Unsafe Null Value
Under a null model, let \(X\) and \(Y\) be independent Bernoulli observations with the same nuisance event probability \(p\in[0,1]\). Consider the rule that rejects only when \(X=1\) and \(Y=0\). Calculate its size and compare it with an average over \(p\).
The probability of rejection at a fixed \(p\) is $$ P_p(X=1,Y=0)=p(1-p)=\frac14-\left(p-\frac12\right)^2. $$ The maximum over \(0\leq p\leq1\) is \(1/4=0.25\), attained at \(p=0.5\). The rule therefore does not control Type I error at level 0.20.
If instead one averages this rejection probability using a uniform distribution over \(p\in[0,1]\), the result is $$ \int_0^1p(1-p)\,dp =\left[\frac{p^2}{2}-\frac{p^3}{3}\right]_0^1 =\frac12-\frac13 =\frac16\approx0.1667. $$ That average is below 0.20, but the rejection probability at \(p=0.5\) is 0.25. Hence the average does not establish a uniform 0.20 Type I error guarantee. The calculation assumes independent Bernoulli observations with a common \(p\) under the null.
The same issue arises when a nuisance value is replaced by an estimate and the test is calibrated as if that estimate were the true value. Unless the procedure has an additional justification, estimation does not show that the rejection probability is bounded for all nuisance values. The supremum calculation is a direct way to state and check the required protection.
Diagnostics, alternatives, and limitations
When auditing an error bound, first write down the null parameter space and the distribution of the data under each value in it. Then identify the rejection event for the complete procedure, including adaptation and randomization. Check whether the conditional distribution used after an interim observation is correct, and whether the conditional bound holds for every allowed nuisance value and every action that can be selected. Finally, verify that the expectation of the bound is no greater than \(\alpha\) for every null value.
Warning signs include calibrating only at a single convenient nuisance value, substituting an interim estimate without a uniform argument, reporting an average rejection probability as though it were a worst-case bound, and using a conditional distribution that ignores dependence between interim and final data. Any of these can make a nominal error guarantee fail. If a calculation uses a numerical optimization over a continuous nuisance space, the optimization must cover the whole permitted space; checking a few grid points alone does not prove that no larger value lies between them.
A supremum-based bound is robust but can be conservative: the nuisance value that maximizes error may be incompatible with other features of the data or may force a procedure to use a cutoff that is unnecessarily strict in typical cases. Alternatives include deriving a sharper bound from the model, using a test whose null distribution does not depend on the nuisance parameter, or conditioning on a statistic that removes the nuisance parameter when such conditioning is justified. Each alternative requires its own assumptions and proof of error control. A plug-in calculation or a parameter average is not a substitute for that proof.
The guarantee also applies only to the model and parameter space that were specified. If the null model is misspecified, the nuisance range is too narrow, or the adaptation uses information omitted from the analysis, the supremum may not protect the actual procedure. Conversely, a bound that is uniform over a larger parameter space remains valid for a smaller one, though it may be less powerful.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A test has rejection probability \(0.03+0.04\lambda\) under the null, with \(0\leq\lambda\leq1\). What is its size, and does it control Type I error at \(\alpha=0.05\)?
- Why does a uniform conditional rejection bound of 0.04 imply an unconditional bound of 0.04, even if the final test is selected using interim data?
- Suppose the rejection probability is \(p(1-p)\) for \(p\in[0,1]\). Is an average rejection probability of \(1/6\) enough to claim Type I error control at 0.20?
- In the adaptive normal example, what assumption ensures that conditioning on \(S\) does not change the distribution of \(W\)?
Solutions. 1. The probability increases with \(\lambda\), so its supremum is \(0.03+0.04(1)=0.07\). It does not control at 0.05. 2. For each null parameter value, the law of total probability gives \(P(R)=E[P(R\mid S)]\leq E[0.04]=0.04\); taking the supremum preserves the bound. 3. No. Its supremum is \(1/4=0.25\), so the average \(1/6\) conceals a null value exceeding 0.20. 4. The model assumes \(S\) and \(W\) are independent under each allowed null value; therefore the conditional distribution of \(W\) given \(S\) is still \(N(\theta,1)\).