Valid Stagewise P-Values Under the Null

Why Stagewise P-Value Validity Matters

In a multi-stage trial, a stagewise p-value summarizes the evidence from a specified stage against a null hypothesis. Before such a value can serve as a calibrated measure of evidence, its behavior under the null must be controlled. In particular, a small p-value should not occur more often than its numerical threshold would suggest when the null is true. This property is called validity under the null.

A valid stagewise p-value is not necessarily uniformly distributed. It may be conservative, especially when the data are discrete or when a test uses an approximation. Validity is the essential inequality; exact uniformity is a stronger special case. This distinction matters when a trial uses stagewise evidence in a later analysis: a p-value that is miscalibrated at an individual stage cannot be made valid merely by labelling it a p-value.

Definition: Let $P$ be a p-value for a null hypothesis $H_0$. It is valid under $H_0$ if, for every $u$ with $0\leq u\leq 1$, $$ \sup_{\theta\in\Theta_0} P_\theta(P\leq u)\leq u, $$ where $\Theta_0$ is the set of parameter values allowed by $H_0$. Equivalently, under every parameter value in the null, the probability that the p-value is at most $u$ is no greater than $u$. A p-value with this property is also called super-uniform under the null.

For a simple null, $\Theta_0$ contains one parameter value, so the supremum is just the probability under that value. For a composite null, validity must hold throughout the null parameter set, not only at one convenient value. If $P$ has a continuous uniform distribution under every null value, equality holds for all $u$. If $P$ is conservative, the inequality can be strict.

One-Sided Tests and Tail Calibration

A one-sided p-value is tied to a prespecified direction of evidence. Suppose larger values of a statistic $T$ favor the alternative, and the null distribution of $T$ is known or correctly bounded for every null parameter value. A standard conservative p-value is the largest null tail probability at least as extreme as the observed statistic:

P(t_{\mathrm{obs}})=\sup_{\theta\in\Theta_0}P_\theta(T\geq t_{\mathrm{obs}}).

The supremum protects against choosing a null parameter value that understates the probability of the observed evidence. When the null distribution is stochastically largest at a boundary value, the supremum is attained there. For a discrete statistic, including equality in the tail ($\geq$) is important: it yields a conservative p-value rather than one that is too small.

Justification: Fix any $\theta\in\Theta_0$. The supremum in the p-value definition is at least $P_\theta(T\geq T_{\mathrm{obs}})$. Therefore the event $P\leq u$ is contained in the event that the null upper-tail probability evaluated at $T_{\mathrm{obs}}$ is at most $u$. For a continuous null distribution, that tail probability is uniform. For a discrete distribution, its attainable tail probabilities are conservative, so the latter event has probability at most $u$. Thus $P_\theta(P\leq u)\leq u$ for every null value $\theta$, which establishes validity.

This reasoning depends on the statistic, tail, and null distribution being specified and calibrated correctly. It does not justify changing the direction after seeing the data. Nor does a one-sided p-value answer a two-sided question: evidence in the opposite direction is not counted as evidence against the null by that one-sided test.

Worked Example: A One-Sided Normal Stage Test

Suppose a hypothetical trial stage estimates a treatment effect $\hat{\theta}$, measured in points on a clinical outcome where larger values are beneficial. Assume $\hat{\theta}\sim N(\theta, SE^2)$, with known $SE=2$ points, and test $H_0:\theta\leq 0$ against $H_a:\theta>0$. These normal-model and known-standard-error assumptions are stipulated for this example.

The statistic is $Z=\hat{\theta}/SE$. Under the boundary null $\theta=0$, $Z\sim N(0,1)$. Under any $\theta<0$, its mean is negative and its upper tail is no larger than at the boundary. Thus the boundary gives the largest null upper-tail probability, and the valid one-sided p-value is $P=1-\Phi(Z_{\mathrm{obs}})$.

Suppose the hypothetical observed estimate is $\hat{\theta}=3.0$ points. Then $$ Z_{\mathrm{obs}}=\frac{3.0\ \text{points}}{2\ \text{points}}=1.50,\qquad P=1-\Phi(1.50)=1-0.9332=0.0668. $$

The stagewise p-value is about 0.0668. At a one-sided significance level of $\alpha=0.05$, this stage alone does not cross the rejection threshold. For the continuous boundary-null distribution, $P(P\leq 0.05)=0.05$, since the event is $Z\geq\Phi^{-1}(0.95)$. This exact probability statement uses the quantile $\Phi^{-1}(0.95)$, not a rounded approximation.

Discrete Data: Validity Without Uniformity

In a discrete model, the test statistic can take only separated values. Consequently, the p-value also takes only separated values and may be conservative. This is not a defect: super-uniformity, rather than exact uniformity, is the required calibration condition.

Worked Example: An Exact Binomial Stagewise P-Value

Consider a hypothetical stage with $n=10$ independent binary outcomes, each having success probability $p$. Assume the sample size is fixed, outcomes have a common probability, and the null is $H_0:p\leq 0.5$, with $H_a:p>0.5$. Let $X$ be the number of successes. Large values of $X$ favor the alternative.

For an observed count $x$, use $P=\sup_{p\leq 0.5}P_p(X\geq x)$. The binomial upper tail increases with $p$, so its largest value over the null occurs at $p=0.5$. Suppose the hypothetical stage observes $x=8$. Then $$ P=P_{0.5}(X\geq 8) =\sum_{k=8}^{10}{10\choose k}(0.5)^{10} =\frac{45+10+1}{1024} =\frac{56}{1024} =0.05469. $$

The p-value is about 0.0547, so this stage does not reject at a one-sided level of 0.05. The p-value is not continuously uniform: it can take only a finite set of values. Nevertheless it is valid, because for each $p\leq0.5$, the chance that the reported upper-tail p-value is at most $u$ is no greater than $u$. The binomial model and the composite-null boundary are both accounted for in the calculation.

For a discrete test, a randomized p-value can sometimes attain exact uniformity by randomizing within the probability mass at the observed statistic. Such randomization is not usually needed for validity and may be undesirable in a clinical interpretation. The ordinary inclusive-tail p-value is transparent and conservative.

Direction and Two-Sided Evidence

A frequent source of invalidity is to calculate a one-sided tail probability and then describe it as a two-sided p-value. Consider $Z\sim N(0,1)$ under a two-sided null, where evidence against the null can occur in either direction. The quantity $\min\{\Phi(Z),1-\Phi(Z)\}$ is the smaller one-sided tail, not a valid two-sided p-value at the same threshold.

Worked Example: Why the Smaller Tail Is Not a Two-Sided P-Value

Let $Z\sim N(0,1)$ under the null and report $P_{\min}=\min\{\Phi(Z),1-\Phi(Z)\}$. At a threshold of 0.05, this reported value is at most 0.05 when $Z\geq q$ or $Z\leq -q$, where $q=\Phi^{-1}(0.95)$. The two events are disjoint, and each has probability 0.05. Therefore $$ P(P_{\min}\leq0.05) =P(Z\geq q)+P(Z\leq-q) =0.05+0.05 =0.10. $$

Thus $P_{\min}$ is not valid as a two-sided p-value: values at most 0.05 occur with probability 0.10 under this null. A standard valid two-sided p-value for this symmetric continuous model is $P_2=\min\{1,2P_{\min}\}$. For a hypothetical observation $Z=2.10$, the calculation is $$ P_2=2\{1-\Phi(2.10)\}=2(0.01786)=0.03573. $$

If the cutoff is rounded to 1.645 instead of using $q=\Phi^{-1}(0.95)$, the corresponding probability is $2\{1-\Phi(1.645)\}\approx0.09997$, not exactly 0.10. The exact calculation uses the quantile; a rounded cutoff gives an approximation.

The doubling rule above relies on the stated symmetric continuous null model. In other models, two-sided p-values must be defined using a valid two-sided test procedure; simply doubling a tail is not automatically correct. For example, discreteness and asymmetry can make a doubled tail conservative or otherwise inappropriate unless the construction is justified.

Validity in a Multi-Stage Trial

For each stage, the null hypothesis and the stagewise analysis must be specified clearly enough to identify the relevant null distribution. This includes the outcome, the statistic, the direction of evidence, the analysis population, and any nuisance parameters that affect calibration. A stagewise p-value is valid only relative to that stated test and its assumptions.

An important distinction is between marginal validity and conditional validity. Marginal validity controls $P(P\leq u)$ under the null across repetitions of the stage and trial design. Conditional validity controls $P(P\leq u\mid \mathcal{H})$ given specified prior information $\mathcal{H}$, such as an earlier-stage history. One does not automatically imply the other. If a design or analysis requires validity conditional on prior information, the stagewise calculation must be calibrated for that conditional distribution.

This issue is especially relevant when an adaptation changes the stage-two sample, population, endpoint, or test statistic. A p-value computed as though the original fixed design still applied may have the wrong null distribution. The remedy is not to assume that the word “p-value” guarantees calibration: derive or simulate the null behavior of the actual prespecified stagewise procedure, including the adaptation when required.

Specify the stagewise question.
State $H_0$, $H_a$, and which direction of the statistic represents evidence against $H_0$.

State the sampling model and assumptions.
Identify the distribution, design features, nuisance parameters, and any conditions needed for the reference distribution.

Calibrate the p-value over the null.
Use a null tail probability or another justified test construction that controls the probability of small p-values for every null value.

Check the actual stage procedure.
If adaptation affects the data or analysis, confirm that the calibration applies to that procedure and to the marginal or conditional validity required.

Diagnostics, Alternatives, and Common Mistakes

Useful checks include verifying the tail direction, confirming that the null distribution covers the full null parameter set, and checking small-tail probabilities by calculation or simulation. A simulation check should generate data under null values, apply the complete stagewise procedure, and estimate $P(P\leq u)$ at several thresholds. Estimates materially above $u$ suggest invalid calibration; simulation uncertainty should also be considered.

Approach	What it offers	What to check
Exact null distribution	Finite-sample calibration under the stated model	Model assumptions and nuisance-parameter handling
Asymptotic reference distribution	Convenient calibration for sufficiently informative samples	Whether the approximation is adequate at the stage sample size
Randomized discrete p-value	Can attain exact uniformity under a discrete null	Whether randomization is scientifically and operationally appropriate
Simulation calibration	Can assess a complex prespecified procedure	Coverage of null scenarios, simulation error, and faithful implementation

Common mistake: “The smaller one-sided tail is 0.05, so the two-sided p-value is 0.05.” The reasoning ignores that equally extreme outcomes in the opposite direction also count under a two-sided test. In the symmetric continuous normal example, the probability of a smaller tail at most 0.05 is 0.10. The correction is to use a valid two-sided construction, such as twice the smaller tail in that model, or to calibrate the chosen two-sided statistic directly.

A second pitfall is treating a conservative p-value as invalid because it is not uniform. Validity requires $P(P\leq u)\leq u$, not equality. Conversely, observing that a procedure works at one threshold does not establish validity for all $u$ or for all parameter values in a composite null.

Key takeaway: A stagewise p-value is valid when, under every parameter value in its null hypothesis, its probability of falling at or below any threshold $u$ is no greater than $u$. Correct tail direction, null-distribution calibration, and assumptions that match the actual stage procedure are essential.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

State the super-uniformity condition for a p-value under a composite null.
In a discrete upper-tail test, why is the inclusive tail $P(T\geq t_{\mathrm{obs}})$ generally preferred to a strict tail?
For $Z=1.00$ in a one-sided standard normal test where large values favor the alternative, calculate the p-value to three significant figures.
What is the probability under the standard normal null that $\min\{\Phi(Z),1-\Phi(Z)\}\leq0.01$? Is this smaller-tail quantity a valid two-sided p-value at that threshold?
Explain why marginal validity does not by itself establish conditional validity given the first-stage history.

Solutions. (1) For every $0\leq u\leq1$, $\sup_{\theta\in\Theta_0}P_\theta(P\leq u)\leq u$. (2) Including equality retains the probability mass at the observed statistic, preventing the discrete tail from understating the chance of equally or more extreme outcomes. (3) $P=1-\Phi(1.00)=1-0.8413=0.1587$, approximately 0.159. (4) The event consists of two disjoint tails, each of probability 0.01, so its probability is 0.02. It is not valid as a two-sided p-value at threshold 0.01. (5) Marginal validity averages over possible histories; validity conditional on a particular history requires the small-p-value bound to hold within that history, which does not follow from the average bound alone.