Composite Null Hypotheses in Adaptive Tests

Why Composite Nulls Require Uniform Control

An adaptive test may change its second-stage sample size or rejection rule after interim data are observed. Error control must therefore account both for that adaptation and for the full null hypothesis. If the null contains many parameter values, a procedure that has rejection probability at most $\alpha$ at one selected value can still exceed $\alpha$ elsewhere in the null.

The key requirement is uniformity: the test must control its type I error for every parameter value in the composite null. For adaptive procedures, this can be established by verifying stagewise p-values across that null or by bounding the conditional chance of rejection after each possible interim result.

Definition: Let $H_0:\theta\in\Theta_0$, and let $\varphi$ be a test rule that equals 1 when it rejects $H_0$ and 0 otherwise. The test has level $\alpha$ over the composite null if $$ \sup_{\theta\in\Theta_0} P_\theta(\varphi=1)\leq\alpha. $$ The supremum is the largest rejection probability among all parameter values allowed by $H_0$. Controlling error at just one value of $\theta$ is not sufficient unless it has also been shown that this value maximizes the rejection probability.

For example, in a one-sided test of $H_0:\mu\leq 0$, the boundary value $\mu=0$ often gives the largest rejection probability for an upper-tail statistic. But that conclusion depends on the model and rejection rule. It should be justified, not assumed. With nuisance parameters, non-monotone rejection regions, or adaptation, the worst-case null value may not be obvious.

Stagewise Validity Across the Null

A stagewise p-value is valid over a composite null if, for every $\theta\in\Theta_0$ and every $u\in[0,1]$, its probability of being at most $u$ is no greater than $u$:

P_\theta(p\leq u)\leq u \qquad \text{for every }\theta\in\Theta_0.

A continuous p-value is exactly uniform at a particular null value when $P_\theta(p\leq u)=u$ for all $u$; at other null values it may be conservative. For a discrete test, validity usually means the inequality rather than equality. In an adaptive design, a second-stage p-value may also need to satisfy this inequality conditional on the interim data and the chosen second-stage design.

The parameter-specific tail probability provides a standard construction. If $T$ is a statistic where larger values count against $H_0$, then for a fixed null value $\theta$, $P_\theta(T\geq t_{\mathrm{obs}})$ is a valid p-value under that value, with suitable treatment of discreteness. For a composite null, taking the supremum of these tail probabilities across null values gives a conservative p-value:

p_{\mathrm{sup}}(t_{\mathrm{obs}}) = \sup_{\theta\in\Theta_0}P_\theta(T\geq t_{\mathrm{obs}}).

This construction can be conservative because it uses the largest null tail probability. It also requires the relevant null distributions to be specified or bounded well enough to compute that supremum.

Worked Example: A One-Sided Normal Test Over a Composite Null

Suppose independent measurements have a normal distribution with known standard deviation $\sigma=10$ units and mean $\mu$. Test $H_0:\mu\leq 0$ against $H_a:\mu>0$ using the sample mean from $n=25$ observations. Under the model, $\bar{x}\sim N(\mu,\sigma^2/n)$. A proposed upper-tail p-value is $p=1-\Phi(\sqrt{n}\bar{x}/\sigma)$, where $\Phi$ is the standard normal cumulative distribution function.

Check validity. For any $\mu\leq0$, the standardized statistic $Z=\sqrt{n}\bar{x}/\sigma$ has distribution $N(\sqrt{n}\mu/\sigma,1)$, whose mean is no greater than zero. For any $u\in(0,1)$, let $z_{1-u}$ be the standard normal quantile with upper-tail probability $u$. Then

P_\mu(p\leq u) =P_\mu(Z\geq z_{1-u}) \leq P_0(Z\geq z_{1-u}) =u.

The inequality follows because shifting the normal mean downward cannot increase the probability of exceeding a fixed upper-tail threshold. Thus the p-value is valid for every $\mu\leq0$, not only at $\mu=0$.

Calculate the observed p-value. If the hypothetical data give $\bar{x}=1.2$ units, then $Z=\sqrt{25}(1.2)/10=0.600$, so $p=1-\Phi(0.600)\approx0.2743$. At significance level $0.05$, this result does not reject $H_0$. The calculation assumes independent normal measurements and a known standard deviation of 10 units; changing those assumptions would require checking the test and its p-value validity again.

Combination Tests: What Must Hold Across the Null

Combination tests use stagewise evidence to form a final test statistic. For a combination rule to control a composite null, it is not enough for the stagewise p-values to be valid at one null value. Their validity must hold at every $\theta\in\Theta_0$. In addition, the dependence conditions used to calibrate the combination must hold under each such value.

A particularly useful sufficient condition is that $p_1$ is valid under every null value, and $p_2$ is valid conditional on the first-stage data and the design chosen from those data. If a fixed-weight inverse-normal combination is used, it takes the form

Z_{\mathrm{comb}} =w_1\Phi^{-1}(1-p_1)+w_2\Phi^{-1}(1-p_2), \qquad w_1^2+w_2^2=1.

When both p-values are independent uniform variables under a null value, this statistic has a standard normal distribution. When p-values are only superuniform, the standard-normal calibration is still conservative for this upper-tail combination if the stagewise p-values are independent under that null value: larger p-values yield smaller transformed evidence, so replacing independent uniform p-values by independent superuniform p-values cannot increase the probability of crossing the upper-tail rejection threshold. This reasoning must apply at every null value.

In an adaptive design, the second-stage p-value should be calculated from the second-stage data using a test valid for the selected design, and its validity should hold conditional on the interim information under each null value. Fixed weights and a correctly calibrated combination rule are also part of the argument. Data-dependent weights or a second-stage calculation whose conditional null distribution is wrong cannot be justified merely by noting that the stagewise p-values looked valid before adaptation.

Worked Example: Checking a Fixed-Weight Combination

Suppose a hypothetical two-stage upper-tail test uses fixed weights $w_1=w_2=1/\sqrt{2}$. Assume, for every $\theta\in\Theta_0$, that the first-stage p-value is valid and that the second-stage p-value is valid conditional on the first-stage data and the selected sample size. Also assume the stagewise evidence satisfies the independence condition required for this combination calibration. The observed p-values are $p_1=0.0800$ and $p_2=0.0300$.

Transform the p-values: $\Phi^{-1}(1-0.0800)\approx1.405$ and $\Phi^{-1}(1-0.0300)\approx1.881$. Therefore,

Z_{\mathrm{comb}} =\frac{1.405+1.881}{\sqrt{2}} \approx2.324.

The upper-tail combined p-value is $1-\Phi(2.324)\approx0.0101$. At level $0.025$, the combination test rejects $H_0$. This observed-data calculation alone does not establish error control: the justification is the stated validity and dependence conditions holding for every $\theta\in\Theta_0$. If, for example, the second-stage p-value were valid only at the boundary but not elsewhere in the null, this calculation would not establish uniform type I error control.

Conditional Rejection Bounds

A conditional-error argument provides another route. Let $X_1$ denote the interim data, and consider an original design with rejection rule $\varphi_0$. At a fixed null value $\theta$, its conditional error after observing $X_1=x$ is

C_\theta(x)=P_\theta(\varphi_0=1\mid X_1=x).

If the original design has level $\alpha$, then averaging its conditional error under any null value gives its rejection probability, which is at most $\alpha$. A modified procedure preserves that bound at a given $\theta$ if, for every possible interim result, its conditional rejection probability is no greater than $C_\theta(x)$. A composite null requires this argument to work for every null value, not merely for the value used to calculate a convenient conditional bound.

P_\theta(\text{modified rejection}) =E_\theta\{P_\theta(\text{modified rejection}\mid X_1)\} \leq E_\theta\{C_\theta(X_1)\} =P_\theta(\varphi_0=1) \leq\alpha.

In practice, a conditional bound used by the design must be implementable without knowing the true null parameter. One sufficient construction is a common bound $B(x)$ such that the modified conditional rejection probability is at most $B(x)$ for every null value and $E_\theta\{B(X_1)\}\leq\alpha$ for every null value. A pointwise conditional-error calculation at one $\theta$ does not automatically supply such a common bound.

Worked Example: A Uniform Bound From an Interim P-Value

Consider a hypothetical design with level $\alpha=0.0500$. Suppose its interim p-value $p_1$ is valid under every null value, and the second-stage p-value is conditionally valid given the interim data and selected design. Define a second-stage rejection cap by $B(p_1)=0.100$ when $p_1\leq0.500$, and $B(p_1)=0$ otherwise. Reject only when $p_2\leq B(p_1)$.

For any null value $\theta$, conditional validity of $p_2$ gives a conditional rejection probability no greater than $B(p_1)$. Also, validity of $p_1$ gives $P_\theta(p_1\leq0.500)\leq0.500$. Thus

P_\theta(\text{reject}) \leq E_\theta\{B(p_1)\} =0.100P_\theta(p_1\leq0.500) \leq0.100(0.500) =0.0500.

For hypothetical observed values $p_1=0.320$ and $p_2=0.0700$, the cap is $0.100$, so the rule rejects. If instead $p_1=0.700$, the cap is zero and the rule cannot reject. The design is valid because its bound holds under every null value; the particular interim result only determines which conditional cap applies.

Diagnostics, Alternatives, and Limitations

When checking a proposed adaptive test, start by identifying the complete null parameter space and the source of randomness in each stage. Then verify that the stagewise p-values, conditional distributions, and design-selection rule satisfy their stated conditions throughout that space. If the model contains nuisance parameters, calculate or bound rejection probabilities over those values rather than silently substituting one estimate.

Boundary-only checking: A boundary value is sufficient only when a valid argument shows that it is least favorable for the actual adaptive rejection rule.
Unconditional stage-two calibration: A p-value valid before adaptation is not necessarily valid conditional on the interim data and the selected stage-two design.
Unjustified independence: A combination calibration that assumes independence must use stagewise evidence whose independence is supported under each null value.
Misused conditional error: A bound computed at one null parameter does not by itself control rejection probabilities at other null parameters.

Combination tests are useful when the stagewise evidence and their joint calibration can be justified across the null. Conditional-error bounds are useful when the design is to be modified while constraining its conditional chance of rejection. Both approaches can be conservative, especially when worst-case null values differ across interim outcomes. Neither removes the need for a correct probability model or a valid conditional calculation. If nuisance parameters are poorly specified, or if the conditional distribution changes in an unaccounted-for way, the claimed uniform error control may not hold.

Common mistake: “The test has size $0.05$ at the null boundary, so it has size $0.05$ for the composite null.” This conclusion is justified only if the boundary is proved to be least favorable for the complete adaptive test. The correction is to verify $\sup_{\theta\in\Theta_0}P_\theta(\text{reject})\leq0.05$, using uniform stagewise validity or a conditional bound whose expectation is controlled for every null value.

Check Your Understanding

Answer the questions, then compare your reasoning with the solutions below.

State the definition of level-$\alpha$ control for $H_0:\theta\in\Theta_0$.
A stage-one p-value is valid only at one boundary value. What additional fact would be needed before using it to claim control over the full composite null?
For a conditionally valid second-stage p-value, why does a rejection rule $p_2\leq B(X_1)$ have rejection probability at most $E_\theta\{B(X_1)\}$?
If $p_1$ is valid over the null and $B(p_1)=0.08$ for $p_1\leq0.25$, and zero otherwise, what upper bound follows for $E_\theta\{B(p_1)\}$?

Solutions. (1) The supremum of the rejection probability over $\theta\in\Theta_0$ must be no greater than $\alpha$. (2) One must show that the boundary is least favorable for the adaptive procedure, or establish validity at all null values by another argument. (3) Conditional validity gives $P_\theta(p_2\leq B(X_1)\mid X_1)\leq B(X_1)$; taking expectations gives the stated bound. (4) Validity implies $P_\theta(p_1\leq0.25)\leq0.25$, so $E_\theta\{B(p_1)\}=0.08P_\theta(p_1\leq0.25)\leq0.08(0.25)=0.0200$.