Composite Nulls and Valid Stagewise P-Values

Why a Composite Null Changes the Validity Question

A stagewise p-value is an input to a combination test, so its calibration matters before the stage results are combined. The key issue is that a clinical-trial null hypothesis often contains many possible parameter values, not just one boundary case. For example, the null of no positive treatment effect includes both zero effect and effects favoring control. The distribution of a stage statistic may also depend on nuisance parameters, such as a baseline response rate or an unknown outcome variance.

It is not enough for a p-value to be calibrated at one convenient null configuration. For a composite null, its null rejection probability must be controlled at every parameter value allowed by that null. This is the sense of validity needed when assessing a stagewise p-value for use in a combination test. It is a property of the stagewise procedure, not a statement about whether the observed result supports the treatment.

Definition: Let $H_0$ be a composite null with parameter space $\Theta_0$. A stagewise p-value $P_i$ is valid for $H_0$ if, for every $\theta\in\Theta_0$ and every $u\in[0,1]$, $$ P_\theta(P_i\leq u)\leq u. $$ Here $P_\theta$ denotes probability when the true parameter is $\theta$. This property is also called super-uniformity under the null. It allows the p-value distribution to be more conservative than uniform, but not to put too much probability near zero.

Write the model parameter as $\theta=(\psi,\lambda)$, where $\psi$ is the treatment-effect parameter and $\lambda$ collects nuisance parameters. A one-sided null might be $H_0:\psi\leq 0$, while $\lambda$ includes an unknown variance and perhaps other features of the outcome distribution. Validity then requires the inequality in the definition for all $\psi\leq0$ and all allowed values of $\lambda$. A calculation only at $\psi=0$, or only at one selected $\lambda$, needs an additional argument before it establishes this requirement.

A Stagewise Example With an Unknown Variance

Consider a fixed stage with $n$ independent patient-level changes in a continuous clinical score. Assume the changes are identically distributed as $X_j\sim N(\psi,\sigma^2)$, with unknown $\sigma>0$; positive $\psi$ represents improvement. These assumptions specify the model before the test is used. The one-sided null is $H_0:\psi\leq0$. The sample mean is $\bar{x}$, the sample standard deviation is $s$, and the Student statistic is $T=\sqrt{n}\bar{x}/s$.

Result: Under the stated independent normal model, the p-value $P_i=1-F_{t_{n-1}}(T)$, where $F_{t_{n-1}}$ is the central Student $t$ cumulative distribution function, is valid for $H_0:\psi\leq0$ for every $\sigma>0$. At $\psi=0$, $T$ has a central $t$ distribution with $n-1$ degrees of freedom. For $\psi<0$, its distribution is a noncentral $t$ distribution with a nonpositive noncentrality parameter, so its upper tail is no larger than at the boundary.

Worked Example: A Valid Stagewise t P-Value

Suppose a hypothetical stage has $n=16$ independent patient changes, with sample mean $\bar{x}=0.75$ score points and sample standard deviation $s=1.00$ score point. Assume the changes follow the normal model above. Test whether the mean change is positive.

The standard error is $SE=s/\sqrt{n}=1.00/4=0.250$ score points. Thus $$ T=\frac{\bar{x}-0}{SE} =\frac{0.75\text{ points}}{0.250\text{ points}} =3.00,\qquad df=n-1=15. $$ The one-sided p-value is $$ P_i=P(T_{15}\geq 3.00)\approx 0.004486. $$

The units cancel in the statistic, as they should. Under the specified model, this p-value is calibrated at $\psi=0$ for every unknown $\sigma>0$; its null distribution does not require plugging in a value of $\sigma$. For $\psi<0$, the upper-tail rejection probability is no larger than at the boundary. The result is evidence against $H_0$, but it does not establish that the effect is clinically important or that every assumption is satisfied.

Justifying Validity Across the Null

The unknown variance in the example does not invalidate the p-value because studentization removes it from the boundary null distribution. The remaining issue is the rest of the composite null: why is calibration at $\psi=0$ enough when the null also allows $\psi<0$? In this model, the statistic has a direct representation that answers the question.

T=\frac{Z+\delta}{\sqrt{V/df}}, \qquad Z\sim N(0,1),\quad V\sim\chi^2_{df},\quad \delta=\frac{\sqrt{n}\psi}{\sigma},

where $Z$ and $V$ are independent. This representation gives a short justification. Use the same $Z$ and $V$ to compare the statistic at a null value $\psi\leq0$ with the statistic at $\psi=0$. Because $\sigma>0$, $\delta\leq0$, and therefore $$ \frac{Z+\delta}{\sqrt{V/df}}\leq\frac{Z}{\sqrt{V/df}}. $$ For every cutoff $c$, it follows that $P_\psi(T\geq c)\leq P_0(T\geq c)$. The central $t$ upper-tail p-value is calibrated at the boundary, so its probability of being at most $u$ is at most $u$ there; the inequality shows that moving into the null cannot increase that rejection probability. The nuisance scale $\sigma$ disappears from the boundary distribution and only sets the nonpositive shift $\delta$ elsewhere in the null.

This argument depends on the model and the direction of the test. It does not license using a boundary calibration for every statistic or every composite null. A different model may have a null distribution that changes non-monotonically with a nuisance parameter, or the boundary may not be the least favorable null value. In those settings, uniform validity needs another justification.

Why Plugging In a Nuisance Estimate Can Fail

A common shortcut is to estimate a nuisance parameter and then calculate a p-value as if that estimate were the known true value. That calculation generally ignores the sampling variation in the estimate. For example, replacing the Student distribution in the preceding example with a standard normal distribution does not preserve the finite-sample null calibration when the variance is estimated from the same observations.

Worked Example: A Normal-Tail Shortcut Is Anti-Conservative

Use the same hypothetical stage data, for which $T=3.00$ and $df=15$. If someone reports $1-\Phi(3.00)$, treating the estimated standard deviation as though it were known, the result is approximately $0.001350$. The correctly calibrated Student p-value under the stated model is $0.004486$, which is larger because the $t_{15}$ distribution has heavier tails than the standard normal distribution.

The defect is not only that these two p-values differ for this observation. Under the boundary null, the shortcut would declare significance at level $0.05$ whenever $T>\Phi^{-1}(0.95)\approx1.645$. But $T$ has a $t_{15}$, not a standard normal, distribution, and $P(T_{15}>1.645)>0.05$. Thus the proposed p-value is not super-uniform under the boundary null and cannot be called valid for the composite null on the basis of this calculation.

This diagnosis assumes the normal model and unknown variance specified for the example. If the variance were known independently and the normal model held, the normal calibration could instead be appropriate. The lesson is to match the reference distribution to the sampling procedure, including nuisance estimation, rather than to choose it because it is convenient.

When a Least-Favorable Nuisance Calibration Is Useful

Sometimes a statistic has a known null distribution for each nuisance value, but no single pivot removes the nuisance parameter. One general construction is to use the largest null tail probability over all allowed nuisance settings. This may be conservative, but it protects validity without pretending that the nuisance value is known.

Result: Suppose a statistic $T$ is oriented so that large values are evidence against $H_0$, and its tail probability can be evaluated under every $\theta\in\Theta_0$. Define $$ P_{\mathrm{sup}}(t)=\sup_{\theta\in\Theta_0}P_\theta(T\geq t). $$ Then $P_{\mathrm{sup}}(T)$ is super-uniform under every $\theta\in\Theta_0$. The supremum includes the treatment-effect values permitted by the null as well as its nuisance-parameter values.

To justify the result, fix any true null value $\theta_0$. By definition, $P_{\mathrm{sup}}(T)\geq P_{\theta_0}(T'\geq T)$, where $T'$ has the distribution of the statistic under $\theta_0$. Therefore the event $P_{\mathrm{sup}}(T)\leq u$ is contained in the event that the ordinary right-tail p-value calculated under $\theta_0$ is at most $u$. The latter has probability at most $u$, including in discrete settings where ties can make the p-value conservative. Hence $$ P_{\theta_0}\{P_{\mathrm{sup}}(T)\leq u\}\leq u. $$ Since $\theta_0$ was arbitrary in $\Theta_0$, the construction is valid across the composite null.

Worked Example: Maximizing Over Two Null Configurations

For illustration, suppose a hypothetical stage uses an integer statistic $T$, with larger values favoring treatment. The null model has two possible nuisance configurations, A and B, and the null distributions are fully specified as follows.

Statistic value	Probability under A	Probability under B
$T=0$	0.80	0.50
$T=1$	0.15	0.30
$T=2$	0.05	0.20

Suppose the observed value is $T=2$. Under A, $P_A(T\geq2)=0.05$; under B, $P_B(T\geq2)=0.20$. Therefore $$ P_{\mathrm{sup}}(2)=\max(0.05,0.20)=0.20. $$ Reporting 0.05 would calibrate only at configuration A and would understate the tail probability at B. The supremum p-value is valid for both configurations, though it is conservative under A. For comparison, the supremum upper-tail p-values at $T=1$ and $T=0$ are $0.50$ and $1.00$, respectively. Across the three possible statistic values, neither configuration assigns probability greater than $u$ to a supremum p-value at most $u$.

Choosing and Checking a Stagewise Calibration

There are several legitimate routes to stagewise validity, but they make different demands. An exact pivot, such as the Student statistic under the normal model, gives a reference distribution that does not depend on the nuisance scale at the null boundary. A least-favorable calibration protects against nuisance values by taking a supremum, often at the cost of power. A valid approximation may be appropriate when its error is controlled for the sample size and parameter range in question. A plug-in calculation is not automatically any of these: estimating a nuisance parameter does not by itself prove that the resulting p-value is super-uniform.

Approach	What supports validity?	Typical limitation
Exact pivot	A justified reference distribution that removes or accounts for nuisance parameters	Requires the specified model and may be unavailable
Least-favorable tail	The largest null tail probability over the full allowed parameter set	Can be conservative and computationally demanding
Approximate calibration	A stated approximation with adequate accuracy over the relevant null range	Finite-sample or extreme-nuisance errors may affect size
Unadjusted plug-in	No guarantee from substitution alone	Can be anti-conservative because nuisance estimation is random

Useful diagnostics follow directly from the definition. Check the assumed sampling model, the direction of the alternative, and whether the reference distribution includes uncertainty from nuisance estimation. Then ask whether calibration holds at the null boundary and whether the boundary is least favorable. If nuisance values remain, examine the null rejection probability across their allowed range, analytically or by a carefully designed simulation. A simulation is a diagnostic rather than a proof unless the parameter range, design, and numerical error are adequately addressed.

In an adaptive trial, each stagewise p-value must be valid for the null configurations relevant to that stage. Calibration at one nuisance setting does not establish validity across a composite null. Nor does validity alone mean that every possible combination rule is calibrated: the combination procedure has its own conditions. The earlier material on stagewise error rates and final test calibration provides the broader context; here the essential check is whether each stagewise input retains its null guarantee uniformly.

Common Mistake and Practical Limits

Wrong reasoning: “The test is valid when the treatment effect is zero, so the stagewise p-value is valid for the whole null.”
Correction: Boundary validity suffices only when a result such as monotonicity establishes that no other null value has a larger rejection probability. Otherwise, check the full null parameter space or use a justified worst-case calibration.
Wrong reasoning: “The nuisance estimate is precise, so I can treat it as known when computing the p-value.”
Correction: Precision alone is not a finite-sample validity argument. Use a reference distribution that accounts for estimation, or justify the approximation for the intended operating conditions.

Worst-case methods also have limits. The supremum may be difficult to compute, may be attained at an unexpected nuisance value, or may be so conservative that the test has little power. A model-based exact pivot can be more efficient but may fail if its distributional assumptions are poor. These are reasons to state the assumptions and assess sensitivity, not reasons to relax the validity requirement silently.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

State the definition of a valid p-value for a composite null $H_0$ with parameter space $\Theta_0$.
In the normal mean example, why is the Student p-value valid for all $\sigma>0$ at the boundary $\psi=0$, and what additional argument covers $\psi<0$?
For $T=3.00$ with $df=15$, why is $1-\Phi(3.00)$ not the appropriate p-value when the variance is estimated from the same normal sample?
In the two-configuration example, what is the supremum p-value for an observed $T=1$, and why?
Does a valid stagewise p-value by itself establish that an arbitrary combination test has the desired error rate? Explain.

Solutions. (1) For every $\theta\in\Theta_0$ and $u\in[0,1]$, validity requires $P_\theta(P_i\leq u)\leq u$. (2) At $\psi=0$, the statistic has a central $t_{n-1}$ distribution independent of $\sigma$; for $\psi<0$, the noncentrality is nonpositive, so the upper-tail probability is no greater than at the boundary. (3) The statistic follows $t_{15}$, not a standard normal, under the stated model; treating the estimated standard deviation as known uses the wrong reference distribution and can inflate the rejection probability. (4) The tails at $T\geq1$ are $0.15+0.05=0.20$ under A and $0.30+0.20=0.50$ under B, so the supremum is $0.50$. (5) No. Stagewise validity is a necessary calibration property of the inputs, but the combination rule must also meet its own conditions for the intended error control.