P-Value Calibration Under a True Null

Why P-Value Calibration Matters

A p-value is calculated from the observed test statistic and a null reference distribution. But if the null hypothesis is true, the statistic itself varies from sample to sample, so the p-value varies too. Understanding that variation explains what a p-value threshold can guarantee and why the guarantee may be exact for a continuous test but conservative for a discrete one.

The key calibration question is: when the null model is true, how often is the p-value at most a chosen number \(\alpha\)? For a correctly specified continuous test, that probability is exactly \(\alpha\). For a discrete test using ordinary nonrandomized tail probabilities, it is generally less than or equal to \(\alpha\), because probability can occur only in jumps.

Definition: Let \(P_{\mathrm{val}}\) denote the p-value as a random variable over repeated samples taken under a true null model. It is uniformly calibrated if \(P(P_{\mathrm{val}}\leq \alpha)=\alpha\) for every \(0\leq\alpha\leq1\). It is conservative if \(P(P_{\mathrm{val}}\leq\alpha)\leq\alpha\) for every such \(\alpha\).

The distinction concerns the distribution of p-values over repeated samples, not whether any one observed p-value is large or small. A p-value is not a probability that the null hypothesis is true. Here, the null model is assumed true in order to study the long-run behavior of the calculation.

Continuous Test Statistics: The Uniformity Result

Suppose a test statistic \(T\) has a continuous null distribution with cumulative distribution function \(F_0(t)=P_0(T\leq t)\). The subscript \(0\) means probabilities are calculated under the null model. For an upper-tail test, define the p-value from an observed statistic \(t\) by \(p(t)=P_0(T\geq t)=1-F_0(t)\). Since a continuous distribution assigns probability zero to any single point, using \(>\) instead of \(\geq\) gives the same tail probability.

Theorem: If \(T\) has a continuous distribution under the true null and the p-value is calculated from its exact null upper-tail probability, then \(P_{\mathrm{val}}=1-F_0(T)\) is uniformly distributed on \([0,1]\). The same conclusion holds for the exact lower-tail p-value \(F_0(T)\).

The result is a direct application of the probability integral transform. If \(T\) has continuous CDF \(F_0\), then \(U=F_0(T)\) is uniform on \([0,1]\). Reversing a uniform variable, \(1-U\), leaves it uniform as well. More explicitly, for \(0\leq u\leq1\),

\begin{aligned} P_0(P_{\mathrm{val}}\leq u) &=P_0(1-F_0(T)\leq u)\\ &=P_0(F_0(T)\geq 1-u)\\ &=P_0(U\geq 1-u)\\ &=u. \end{aligned}

This calculation shows why the null tail probability is calibrated: the probability of obtaining a p-value no greater than \(u\) is exactly \(u\). In particular, a threshold of \(0.05\) is reached or crossed in 5% of repeated samples under this exact continuous null model. That statement depends on using the correct null distribution and the exact tail probability; it is not automatic for every quantity someone labels a p-value.

Worked Example: Calibrating an Upper-Tail Normal Test

Under a hypothetical null model, a standardized test statistic \(Z\) has the exact distribution \(N(0,1)\), where the second parameter is the variance. An observed value is \(z=1.30\). Calculate its upper-tail p-value, determine whether it is below \(0.10\), and verify the repeated-sample probability of crossing the \(0.10\) threshold.

Method and assumptions. Use the exact standard normal null distribution and the upper-tail p-value \(p(z)=P_0(Z\geq z)=1-\Phi(z)\), where \(\Phi\) is the standard normal CDF. The distribution is continuous, so the uniform calibration result applies.

Observed p-value. From the standard normal distribution, \(\Phi(1.30)=0.9031995\ldots\). Thus

p(1.30)=1-0.9031995\ldots=0.0968005\ldots\approx 0.0968.

The p-value is below \(0.10\). To check the threshold directly, \(p(Z)\leq0.10\) is equivalent to \(1-\Phi(Z)\leq0.10\), or \(\Phi(Z)\geq0.90\). Therefore \(Z\geq\Phi^{-1}(0.90)=1.2816\ldots\), and

P_0(p(Z)\leq0.10)=P_0(Z\geq1.2816\ldots)=0.10.

The p-value is rounded to four decimal places; the threshold probability is exact by the definition of the 90th percentile. Check: \(1.30>1.2816\ldots\), consistent with the observed p-value being less than \(0.10\). The calculation avoids treating the observed p-value itself as a 10% probability about the null.

Two-Sided Tests and Continuous Calibration

For a two-sided test, the same principle applies when the p-value is the exact upper-tail probability of a continuous measure of extremeness. For example, in a symmetric standard normal test, larger values of \(|Z|\) are more extreme in either direction. The null distribution of \(W=|Z|\) is continuous, so the two-sided p-value can be calculated as the upper-tail probability of \(W\).

P_{\mathrm{val}}=P_0(|Z|\geq |z|) =2\Phi(-|z|).

The final equality uses the symmetry of the standard normal distribution. Calibration follows from the continuous upper-tail result applied to \(W\), not merely from the fact that the p-value has been doubled. In other continuous models, the appropriate two-sided p-value should likewise be based on a defined measure of extremeness and its null distribution; the normal formula is not universal.

Worked Example: Checking a Two-Sided Normal P-Value

A hypothetical two-sided test has the exact null distribution \(Z\sim N(0,1)\). The observed statistic is \(z=2.10\). Calculate the p-value to four decimal places, then verify the null probability of obtaining a p-value at most \(0.05\).

Method and assumptions. Use the exact symmetric standard normal null model. Define extremeness by \(|Z|\), so the p-value is \(P_0(|Z|\geq|z|)=2\Phi(-|z|)\). This is the continuous null-tail probability for \(|Z|\).

Observed p-value. The normal tail probability is \(\Phi(-2.10)=0.0178644\ldots\). Retaining the unrounded tail probability through multiplication gives

p(2.10)=2(0.0178644\ldots)=0.0357288\ldots\approx0.0357.

To verify calibration at \(0.05\), the event \(p(Z)\leq0.05\) is

2\Phi(-|Z|)\leq0.05 \quad\Longleftrightarrow\quad |Z|\geq\Phi^{-1}(0.975)=1.9600\ldots.

Under the null, each tail beyond these cutoffs has probability \(0.025\), so

P_0(p(Z)\leq0.05) =P_0(Z\leq-1.9600\ldots)+P_0(Z\geq1.9600\ldots) =0.025+0.025=0.05.

The observed p-value is rounded to four decimal places. Check: \(2.10>1.9600\ldots\), and \(0.0357<0.05\), so the observed statistic lies in the corresponding two-sided tail region. The calculation avoids rounding \(0.0178644\ldots\) to \(0.0179\) before doubling; if rounded first, that intermediate product would be \(0.0358\), not the correctly rounded \(0.0357\).

Discrete Tests: Why Calibration Is Usually Conservative

A discrete statistic can take only a limited or countable set of values. Its null distribution therefore has probability jumps, and its tail probability can move from above a chosen threshold to below it without taking that threshold exactly. With the usual nonrandomized upper-tail definition \(p(t)=P_0(T\geq t)\), the p-value itself has a discrete distribution.

Key result: For a discrete test using the exact inclusive null tail probability, \(P_0(P_{\mathrm{val}}\leq\alpha)\leq\alpha\). Thus its p-values are generally conservative: a threshold event can have probability less than the threshold because the null distribution cannot divide an outcome's probability mass into smaller pieces.

For an upper-tail test, order the possible values of \(T\) from largest to smallest. The p-value at any value is the total null probability at that value and all more extreme values. As the cutoff moves through the support of \(T\), the probability of the rejection event changes in jumps. At a given \(\alpha\), the selected tail may have probability below \(\alpha\), with no attainable cutoff giving exactly \(\alpha\). This is the source of conservatism. A discrete p-value can still happen to have exact calibration at some thresholds, but it is not generally uniform over the whole interval \([0,1]\).

Worked Example: A Discrete Binomial Test

A hypothetical quality check counts the number \(X\) of flagged items in four independent inspections. Under the true null model, each inspection is flagged with probability \(0.5\), independently, so \(X\sim\operatorname{Binomial}(4,0.5)\). Use the inclusive upper-tail p-value \(P_0(X\geq x)\). Find the p-value for an observation \(x=3\), and calculate the probability under the null that the p-value is at most \(0.30\).

Method and assumptions. Use the exact binomial model with \(n=4\) and success probability \(0.5\). For \(k=0,1,2,3,4\),

P_0(X=k)=\binom{4}{k}(0.5)^k(0.5)^{4-k} =\frac{\binom{4}{k}}{16}.

The probabilities at \(k=0,1,2,3,4\) are respectively \(1/16,4/16,6/16,4/16,1/16\). The upper-tail p-values at those outcomes are

\begin{array}{c|ccccc} x&0&1&2&3&4\\ \hline P_0(X\geq x)&16/16&15/16&11/16&5/16&1/16\\ &1&0.9375&0.6875&0.3125&0.0625 \end{array}

At the observed \(x=3\), the p-value is \(P_0(X\geq3)=(4+1)/16=5/16=0.3125\). To find \(P_0(P_{\mathrm{val}}\leq0.30)\), inspect the possible p-values. The values \(1,0.9375,0.6875,\) and \(0.3125\) exceed \(0.30\); only \(0.0625\), which occurs when \(X=4\), is at most \(0.30\). Therefore

P_0(P_{\mathrm{val}}\leq0.30)=P_0(X=4)=\frac{1}{16}=0.0625<0.30.

The probability is reported to four decimal places. Check: \(0.0625\leq0.30\), as required for a conservative p-value, and the event contains exactly the \(X=4\) outcome. The calculation avoids interpolating between discrete outcomes as if the p-value distribution were continuous.

Conditions and Common Calibration Traps

Exact uniform calibration requires more than a continuous-looking statistic. The null distribution used to calculate the p-value must be the correct distribution under the true null, and the p-value must be its exact tail probability under the stated ordering or extremeness rule. If a reference distribution is only an approximation, calibration is generally approximate rather than exact. If nuisance parameters are estimated, the resulting p-value need not be exactly uniform unless the method accounts for that estimation in a way that gives the claimed null distribution.

For a composite null hypothesis, the null contains more than one parameter value. A p-value that is calibrated at one parameter value is not thereby guaranteed to be calibrated across all values in the composite null. A valid composite-null procedure must control the probability of small p-values for each null parameter value, or use an appropriate worst-case null calculation. The uniformity derivation above is for a specified true null distribution, including an exact pivotal reference distribution when one is available.

Assuming all p-values are uniform. Uniformity is an exact result under specified continuous-null conditions; ordinary discrete tail p-values are generally conservative.
Ignoring the tail convention. Upper-tail, lower-tail, and two-sided tests use different definitions of extremeness. The p-value must match the test and its null distribution.
Rounding too early. Keep adequate precision in tail calculations and round only the final reported probability.
Calling conservatism exact uniformity. The inequality \(P_0(P_{\mathrm{val}}\leq\alpha)\leq\alpha\) allows strict inequality. A discrete test may not attain a chosen threshold probability exactly.

Key takeaway: Under a true, correctly specified continuous null, an exact null-tail p-value is uniform because the CDF transforms the statistic to a uniform variable. For a discrete statistic, inclusive tail probabilities usually produce a conservative, stepwise distribution: the chance of a p-value at most \(\alpha\) is no greater than \(\alpha\).

Check Your Understanding

For each problem, assume the stated null model is correct and use the specified p-value convention.

Under a null model, \(T\sim\operatorname{Uniform}(0,1)\), and the upper-tail p-value is \(1-T\). Find the probability that the p-value is at most \(0.20\), and state its null distribution. (Answer: \(P(P_{\mathrm{val}}\leq0.20)=0.20\); \(P_{\mathrm{val}}\sim\operatorname{Uniform}(0,1)\).)
A continuous upper-tail test has \(Z\sim N(0,1)\) under the null. For an observation \(z=-1.25\), calculate the p-value to four decimal places, and give the probability under the null that its p-value is at most \(0.10\). (Answer: \(1-\Phi(-1.25)=0.8944\); \(P(P_{\mathrm{val}}\leq0.10)=0.10\).)
Under a null model, \(X\sim\operatorname{Binomial}(3,0.5)\), and the p-value is \(P_0(X\geq x)\). List the p-value for observed \(x=2\), then find \(P_0(P_{\mathrm{val}}\leq0.20)\). (Answer: \(P_0(X\geq2)=4/8=0.5000\); the calibration probability is \(P_0(X=3)=1/8=0.1250\).)
A two-sided test has the exact null distribution \(Z\sim N(0,1)\) and p-value \(2\Phi(-|z|)\). For observed \(z=1.50\), calculate the p-value to four decimal places and verify the probability of a p-value at most \(0.10\). (Answer: \(2\Phi(-1.50)=2(0.0668072\ldots)=0.1336\); \(P(P_{\mathrm{val}}\leq0.10)=P(|Z|\geq1.6449\ldots)=0.10\).)
For the four-inspection binomial test in the worked example, find the null probability that the p-value is at most \(0.40\), and compare it with \(0.40\). (Answer: the outcomes \(X=3,4\) have p-values \(0.3125,0.0625\), so the probability is \((4+1)/16=0.3125<0.40\).)

What You'll Learn