Superuniformity as the Validity Criterion for a P-Value

Why a P-Value Needs a Distributional Validity Criterion

A reported p-value is a number between zero and one, but its numerical range alone does not make it valid. Its meaning depends on how the procedure producing it behaves when the null hypothesis is true. This matters in a group sequential trial because the observed result includes not only the data at the stopping look, but also the fact that the trial stopped there under its stopping rule.

The relevant calibration question is: if the null hypothesis were true and the entire prespecified trial—including its stopping rule—were repeated, how often would the reported p-value be as small as a chosen cutoff? A valid p-value must not produce small values too frequently under the null. The precise requirement is called superuniformity.

Definition: Let $P\in[0,1]$ be a p-value computed from the outcome of a trial, including its stopping decision. Under a simple null hypothesis, $P$ is superuniform if $$ P(P\leq u\mid H_0)\leq u \quad\text{for every }u\in[0,1]. $$ For a composite null with parameter space $\Theta_0$, validity requires $$ \sup_{\theta\in\Theta_0}P_{\theta}(P\leq u)\leq u \quad\text{for every }u\in[0,1]. $$

The probability in this definition is calculated under the null sampling model. In this course, that model must include the planned interim looks and stopping rule: the p-value is a function of the stopped-trial outcome, not merely of a statistic considered as if sampling had continued to a fixed sample size. For a composite null, the inequality must hold at every null parameter value, or equivalently at the worst-case null probability shown above.

The definition is unconditional over the trial’s possible stopping outcomes. It does not require the same inequality after conditioning on each stopping look. Such conditional requirements are stronger and are not part of the basic validity criterion taught here.

Why Superuniformity Gives Type I Error Control

A test that rejects when $P\leq\alpha$ has type I error probability equal to the probability that its p-value falls at or below $\alpha$, when the null is true. Substituting $u=\alpha$ into the superuniformity condition gives the promised control directly.

P_{\theta}(\text{reject }H_0) = P_{\theta}(P\leq\alpha) \leq\alpha, \qquad \theta\in\Theta_0.

This argument works for every nominal level $\alpha\in[0,1]$, not just for a particular chosen level such as 0.05. The converse is also immediate: if the rule “reject when $P\leq\alpha$” controls type I error at every nominal level $\alpha$, then setting $\alpha=u$ gives $P_{\theta}(P\leq u)\leq u$ for every $u$. Thus superuniformity is exactly the distributional criterion for a p-value whose threshold rule controls rejection probability at every level.

Theorem: A p-value $P$ is valid under a simple null if and only if the test that rejects $H_0$ when $P\leq\alpha$ has null rejection probability at most $\alpha$ for every $\alpha\in[0,1]$. For a composite null, the same statement holds uniformly over all $\theta\in\Theta_0$.

For this result, the p-value must be computed according to a specified procedure, and its null distribution must correspond to the actual trial design and null model. In a sequential setting, that includes the stopping mechanism. A correctly calculated tail probability for a different sampling plan does not establish the theorem’s premise.

Checking the Condition for Discrete P-Values

A p-value need not have a continuous uniform distribution to be valid. If it takes only a finite set of values, its cumulative distribution function is a staircase. It is enough to check the superuniformity inequality at the support points: between consecutive support points, the cumulative probability stays fixed while the cutoff $u$ increases, so the inequality cannot first fail between them.

Worked Example: A Valid Discrete P-Value

Suppose a hypothetical null model gives the following distribution for a p-value $P$:

Value of $P$	Probability	Cumulative probability $P(P\leq u)$ at that value
0.02	0.01	0.01
0.10	0.08	0.09
0.35	0.26	0.35
0.70	0.30	0.65
1.00	0.35	1.00

The probabilities sum to $0.01+0.08+0.26+0.30+0.35=1.00$. At the five support points, the required comparisons are $0.01\leq0.02$, $0.09\leq0.10$, $0.35=0.35$, $0.65\leq0.70$, and $1.00=1.00$. The condition therefore holds at every support point.

For instance, if $0.35\leq u<0.70$, then $P(P\leq u)=0.35$, which is at most $u$. The same reasoning applies between each pair of support points; below 0.02 the cumulative probability is zero. Thus the p-value is superuniform for every $u\in[0,1]$, not only at the listed values.

At nominal level $\alpha=0.10$, the rejection probability is $P(P\leq0.10)=0.09$, which is no greater than 0.10. At $\alpha=0.35$, it is exactly 0.35. The p-value is valid, but at some levels it is conservative: the test rejects less often than the nominal maximum.

A continuous uniform p-value is the special case in which $P(P\leq u)=u$ for all $u$. Superuniformity allows the cumulative probability to be strictly below the diagonal. That slack is the mathematical expression of conservatism; it is not a validity defect.

Worked Example: An Invalid Assignment and the Cutoff That Fails

Consider a different hypothetical null distribution with $P=0.15$ having probability 0.15, $P=0.40$ having probability 0.25, and $P=0.80$ having probability 0.60. The masses sum to one. Its cumulative probabilities at the support points are

P(P\leq0.15)=0.15,\qquad P(P\leq0.40)=0.15+0.25=0.40,\qquad P(P\leq0.80)=0.15+0.25+0.60=1.00.

At $u=0.40$, the cumulative probability is 0.40, so the superuniformity condition holds with equality. At $u=0.80$, the cumulative probability is 1.00, which is greater than 0.80; that cutoff violates the condition. Consequently, at nominal level 0.80, the rule $P\leq0.80$ rejects with null probability 1.00, exceeding its nominal level. This assignment is not a valid p-value.

The failure at one cutoff is enough to invalidate the p-value. Validity is a statement about all cutoffs, not about most of them or about one conventional significance level. In particular, passing the check at $u=0.40$ does not repair the failure at $u=0.80$.

Applying the Criterion to a Stopped-Trial Procedure

For a group sequential procedure, suppose a specified calculation assigns one p-value to each possible stopped-trial outcome. To assess its validity under the null, consider the distribution of that assigned value over repetitions of the complete design. This distribution must account for the data observed at each look and for the rule that determines whether another look occurs.

Worked Example: Rejection Probabilities Across Nominal Levels

Suppose a hypothetical stopped-trial procedure has this null distribution for its reported p-value: values 0.04, 0.15, 0.33, 0.60, and 1.00 occur with probabilities 0.03, 0.12, 0.25, 0.30, and 0.30, respectively. These probabilities sum to 1.00. The cumulative probabilities at the support points are 0.03, 0.15, 0.40, 0.70, and 1.00.

Compare each cumulative probability with its cutoff: $0.03\leq0.04$, $0.15=0.15$, $0.40\leq0.33$ is false, $0.70\leq0.60$ is false, and $1.00=1.00$. The checks at 0.33 and 0.60 fail, so this distribution is not superuniform. For example, at $\alpha=0.33$, the null rejection probability is 0.40, which exceeds 0.33. Despite being below the nominal level at the first two support points, it cannot be called valid.

This calculation illustrates the required check, not a claim about a particular trial design. In an actual design, the probabilities must come from the null distribution of the p-value produced by the full stopping procedure. If the null is composite, the comparisons must hold for every null parameter value or for the supremum over those values.

Assumptions, Diagnostics, and Scope

Superuniformity is a property of the p-value procedure under a specified null model, not a label attached to a formula in isolation. The probability calculation must use the intended sampling model, including the number and timing of analyses and the stopping rules. The p-value assignment must be applied consistently to each possible stopped outcome. In a composite-null problem, calibration at one convenient null parameter is insufficient unless it is known to be the worst case or validity has otherwise been established over the whole null space.

A useful diagnostic is to compare the null cumulative distribution of the reported p-value with the diagonal $u\mapsto u$. Any point where the cumulative probability exceeds the cutoff is direct evidence of invalidity. Simulation can help reveal such a problem, but a finite simulation is a diagnostic rather than a proof: an estimated exceedance rate can fluctuate, especially in the tail. Numerical calculations also need adequate accuracy near the relevant cutoffs, since small integration or simulation errors can obscure a small excess.

This criterion is unconditional across the stopped-trial outcomes. It does not say that, among trials stopping at a particular look, the p-values must themselves be superuniform. Nor does it say that the probability the null is true equals the reported p-value. A p-value is a statistic whose repeated-sampling behavior under the null is controlled; it is not a posterior probability about the hypothesis.

The main alternative to reporting a p-value is to calibrate and report a test at a particular nominal level. That can establish error control at that chosen level, but it does not by itself establish a p-value valid at every level. Conversely, an exact uniform distribution is stronger than needed: a superuniform p-value may be discrete or conservative and still guarantee the stated type I error bound. Neither this criterion nor a single numerical check constructs a suitable stopped-trial p-value; the assignment must first be defined and calibrated under the null.

Common Mistakes

Checking only a familiar significance level. Control at $\alpha=0.05$ alone does not prove superuniformity. The inequality must hold for every $u\in[0,1]$.
Calling a discrete p-value invalid because it is not uniform. Uniformity is sufficient but not necessary. A discrete distribution is valid when its cumulative probability never exceeds the cutoff.
Checking only the first few support points. One violation, including at a large cutoff, is enough to invalidate the p-value.
Using a null distribution that ignores stopping. Validity must be assessed under the sampling plan that actually generates the stopped outcome.
Interpreting $P$ as the probability that $H_0$ is true. The criterion concerns repeated-trial probabilities conditional on the null model, not the probability of a hypothesis.

Key takeaway: Under the null, a valid p-value satisfies $P(P\leq u)\leq u$ at every cutoff $u$. Therefore, rejecting when $P\leq\alpha$ has null rejection probability at most $\alpha$ at every nominal level. In a stopped trial, the null distribution used for this check must include the stopping rule.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

A p-value is uniform under a simple null. What is $P(P\leq0.07)$, and what does this imply for rejection at level 0.07?
A discrete p-value has mass 0.10 at 0.20 and mass 0.90 at 1.00. Is it superuniform? Check the support points and explain why that is enough.
Suppose $P(P\leq0.30)=0.34$ under the null. What conclusion follows about validity and rejection at nominal level 0.30?
For a composite null, the inequality holds at one parameter value but fails at another null parameter value. Is the p-value valid for the composite null?
Does unconditional superuniformity require superuniformity after conditioning on the stopping look? Explain briefly.

Solutions. (1) Uniformity gives $P(P\leq0.07)=0.07$, so rejection at level 0.07 has null probability 0.07. (2) At $u=0.20$, the cumulative probability is 0.10, which is at most 0.20; at $u=1.00$, it is 1.00. Between support points the cumulative probability is constant and the cutoff increases, so the condition holds everywhere; the p-value is valid. (3) Since 0.34 exceeds 0.30, superuniformity fails and rejection at level 0.30 has null probability 0.34. (4) No. Composite-null validity must hold at every null parameter value, so one failure is sufficient to invalidate it. (5) No. The definition controls the unconditional null rejection probability over stopped-trial outcomes; conditional validity at each stopping look is a stronger, separate requirement.