P-Values as Graded Evidence, Not a Verdict

Introduction

A p-value is often reported as one number, but its meaning is not a verdict. It measures how often a specified null model would produce a result at least as extreme as the observed result, using a stated test procedure. Comparing p-values can therefore describe degrees of incompatibility with that null model: a smaller value corresponds to a more unusual result under the model. It does not establish that a hypothesis is true or false.

The comparison is most informative when the p-values come from the same null model, the same test statistic, and the same rule for deciding which results count as extreme. Even then, the comparison concerns the results’ compatibility with the model, not the probability that the model is correct. The sign or direction of the observed result also requires attention; a p-value by itself does not supply it.

What a P-Value Measures

Definition: A p-value is the probability, calculated under the null model and its assumptions, of obtaining a test statistic at least as extreme as the observed statistic according to the test’s specified extremeness rule. For a one-sided test, extremeness is measured in the direction specified by the alternative hypothesis. For a two-sided test, it is measured according to the chosen two-sided rule.

The probability is conditional on the null model being used for the calculation. For example, a p-value of 0.03 means that if the null model and the assumptions used to calculate the p-value hold, results at least as extreme as the observed result would occur with probability 0.03 under repeated sampling. It does not mean that the null hypothesis has a 3% probability of being true, or that there is a 97% probability that the alternative is true.

The value is also tied to how the test defines “at least as extreme.” A one-sided test counts results in its specified direction. A two-sided test counts departures in either direction according to its stated rule. Thus, p-values should not be compared as though they were independent of the test that produced them. First ask what null model, statistic, assumptions, and extremeness rule each value uses.

Comparing P-Values Under a Shared Test Setup

When those elements are held fixed, the comparison has a clear interpretation. If one observed statistic is more extreme under the test’s rule than another, its p-value cannot be larger. In a test with a continuous reference distribution, it will usually be smaller. With discrete distributions or tied extremeness rankings, a change in the observed outcome can leave the p-value unchanged.

The reason is a basic property of tail probabilities. Let \(T\) be a statistic for which larger values mean more extreme results, and let \(P_0\) denote probability under the null model. The one-sided p-value at cutoff \(t\) is \(P_0(T \ge t)\). If \(t_2>t_1\), then every outcome with \(T \ge t_2\) also has \(T \ge t_1\). Therefore:

\{T \ge t_2\} \subseteq \{T \ge t_1\} \quad\Longrightarrow\quad P_0(T \ge t_2) \le P_0(T \ge t_1).

The tail probability decreases strictly only if there is positive probability on statistic values between the two cutoffs. More precisely, the difference between the probabilities is \(P_0(t_1 \le T < t_2)\). If that probability is zero, the tail probability stays the same. This can occur for discrete statistics when no possible values lie between the cutoffs.

Worked Example: Graded P-Values in the Same Normal Test

Suppose, hypothetically, that a random sample of \(n=25\) independent observations comes from a normal population with known standard deviation \(\sigma=10\) units. The null hypothesis is that the population mean is \(\mu=50\) units, and the alternative is two-sided. For this model, the standard error of the sample mean is \(\sigma/\sqrt{n}\). Consider three possible observed sample means: 52.4, 54.0, and 56.0 units.

Check the model. This calculation assumes the sample is random and independent and that the stated normal model and known \(\sigma\) are appropriate. The standard error is \(10/\sqrt{25}=2\) units. Under the null hypothesis, the standardized statistic has a standard normal reference distribution, denoted \(Z\sim N(0,1)\).

For a two-sided test, the statistic is \(z=(\bar{x}-\mu)/SE\), and the p-value is \(2P(Z\ge |z|)\). For \(\bar{x}=52.4\) units:

z=\frac{52.4-50}{2}=1.20, \qquad p=2P(Z\ge 1.20)\approx 2(0.1150697)=0.2301.

For \(\bar{x}=54.0\) units:

z=\frac{54.0-50}{2}=2.00, \qquad p=2P(Z\ge 2.00)\approx 2(0.0227501)=0.04550.

For \(\bar{x}=56.0\) units:

z=\frac{56.0-50}{2}=3.00, \qquad p=2P(Z\ge 3.00)\approx 2(0.0013499)=0.002700.

These calculations use the same null model and test rule. The observed means are 2.4, 4.0, and 6.0 units above the null mean, respectively, and their standardized distances from the null value increase. The corresponding p-values decrease. The result at 56.0 units is more unusual under this particular null model and test procedure than the result at 54.0 units; it is not thereby proven that the null hypothesis is false.

Discreteness and Equal P-Values

With discrete data, several outcomes may have the same degree of extremeness under a test’s rule. In a two-sided test, outcomes on opposite sides of a null value may receive the same p-value, even though they point in different directions. A p-value comparison should not erase the direction shown by the observed statistic.

Worked Example: Equal P-Values, Opposite Directions

Suppose a hypothetical experiment consists of \(n=5\) independent trials, each with success probability \(p=0.5\) under the null hypothesis. Let \(X\) be the number of successes. The null model is \(X\sim\operatorname{Binomial}(5,0.5)\). Consider observing either \(X=1\) or \(X=4\). Use a two-sided probability-ordering rule: count outcomes whose null probability is no greater than the probability of the observed outcome.

Check the model. The calculation assumes independent trials with a common success probability of 0.5 under the null. The null probabilities for counts \(x=0,1,2,3,4,5\) are \(1/32, 5/32, 10/32, 10/32, 5/32, 1/32\), respectively.

If \(X=1\) is observed, its null probability is \(5/32\). Counts with probability no greater than \(5/32\) are 0, 1, 4, and 5. Thus:

p=P(X\in\{0,1,4,5\}\mid H_0) =\frac{1+5+5+1}{32} =\frac{12}{32} =0.375.

If \(X=4\) is observed, its null probability is also \(5/32\), and the same four counts are at least as extreme by this rule. The p-value is again \(12/32=0.375\). The results point in opposite directions relative to the null expectation of \(5(0.5)=2.5\) successes: one observed success is below it, while four are above it. The equal p-values reflect the symmetry of the null model and the chosen two-sided rule, not identical directions.

For discrete tests, the precise two-sided rule matters; other standard constructions can produce different values. The comparison is meaningful only when the rule is stated and used consistently.

A P-Value Is Not a Verdict

A small p-value describes a result that would be relatively unusual if the specified null model and assumptions were correct. That makes the result more incompatible with the model than a result associated with a larger p-value, when the tests are suitably comparable. It does not identify the reason for the discrepancy. The model or its assumptions may be inadequate, the data may reflect an unmodeled feature, or the result may be an unusual outcome under a model that is otherwise appropriate. The p-value alone cannot distinguish these explanations.

Likewise, a comparatively large p-value does not establish that the null model is true. It only says that the observed result is not especially far into the region that the test defines as extreme. The calculation does not give the probability of the null hypothesis, and it does not prove that the data match the null model. Those conclusions would require information and reasoning beyond the p-value itself.

Nor should a p-value be treated as a direct measure of how large or important an effect is. It is a probability under a reference model, not a measurement in the outcome’s units. Always report and interpret the observed effect or statistic in context as well as the p-value; the p-value cannot replace the direction and magnitude of the observed result.

Worked Example: Interpreting a P-Value in Context

Suppose, hypothetically, a study estimates a mean difference of 2.0 minutes between two groups. Under the null hypothesis of a mean difference of 0 minutes, suppose the stated model and assumptions give a standard error of 1.25 minutes and a standard normal reference distribution. The alternative is two-sided.

Check the model. For this calculation, assume the observations and study design justify the stated standard error and that the null distribution of the standardized statistic is standard normal. Given those assumptions, the observed statistic is:

z=\frac{2.0-0}{1.25}=1.60.

The two-sided p-value is:

p=2P(Z\ge 1.60) \approx 2(0.0547993) =0.1096.

In context, if the null model of a 0-minute mean difference and the assumptions used for the calculation hold, the probability of a test statistic at least as extreme as \(1.60\) in either direction is about 0.1096. This is not a 10.96% probability that the null hypothesis is true, and it is not a 10.96% probability that the result was “caused by chance.” The observed difference is 2.0 minutes in the positive direction; the p-value describes its extremeness under the specified null model.

Common Mistakes

Turning a graded comparison into a verdict. “The smaller p-value proves the null is false” is not justified. The correction is that a smaller p-value, under a comparable test setup, indicates greater incompatibility with the null model.
Reading a p-value as the probability of a hypothesis. A p-value is calculated assuming a null model; it is not the probability that the null is true or false.
Assuming every more extreme cutoff must have a strictly smaller tail probability. The tail probability cannot increase as the cutoff moves farther into the tail, but it may stay the same if no probability is assigned between cutoffs.
Using the p-value to infer direction. A two-sided p-value can be the same for results on opposite sides of the null value. Inspect the statistic or estimate to identify direction.
Comparing values from unlike tests as if they were on one universal scale. Check the null models, statistics, assumptions, and extremeness rules before drawing a comparison.

Key takeaway: A p-value grades how unusual the observed result is under a specified null model and test procedure. Smaller values indicate greater incompatibility when the comparison is appropriate, but no p-value by itself proves a hypothesis true or false.

Check Your Understanding

Answer each question by referring to the null model and test procedure, not just the reported number.

In your own words, what does a p-value of 0.04 mean under the model used to calculate it? What does it not mean?
Two results from the same two-sided test have p-values 0.20 and 0.03. What comparison is justified, and what conclusion would go too far?
Why can a tail probability remain unchanged as its cutoff moves farther into a tail of a discrete distribution?
In the binomial example, why did observing one success and four successes produce the same p-value, and what information about direction did the p-value leave out?
What should you check before comparing p-values from two different tests?