Why a Small P-Value Does Not Mean Chance Caused the Data

What “Chance Alone” Leaves Unspecified

When a result is surprising, it is tempting to ask whether “chance alone” could have produced it. The phrase sounds like a question about causes, but a p-value answers a narrower question: if a particular null model and its assumptions were correct, how often would a result at least this extreme occur? It does not calculate the probability that chance was the cause of the data, nor does it identify what did cause them.

The distinction matters because the word “chance” is not a complete statistical model. To calculate a p-value, we need to specify what outcomes are possible, how likely they are under the null hypothesis, and which outcomes count as at least as extreme as the observed result. A small p-value says that the defined tail of outcomes is relatively unlikely under those specifications. It cannot, by itself, tell us whether those specifications are appropriate.

Definition: A p-value is the probability, calculated under a specified null hypothesis and probability model, of obtaining a test statistic at least as extreme as the observed statistic according to the test’s chosen extremeness rule. In symbols, it has the form \(P(T \text{ is at least as extreme as } t_{\mathrm{obs}} \mid H_0,\text{ model})\), where \(T\) is the test statistic and \(t_{\mathrm{obs}}\) is its observed value.

The condition after the vertical bar is essential. A p-value is conditional on the null model; it is not a probability calculated across competing explanations of the data. In particular, it is not \(P(\text{chance caused the data}\mid\text{observed data})\). A p-value does not define “chance” as a general cause and compare it with every possible alternative cause.

A Tail Probability Is Not the Probability of the Observed Outcome

A p-value usually adds up the probabilities of multiple possible outcomes. It includes the observed outcome and other outcomes that the test regards as at least as extreme. Consequently, the probability of the observed outcome itself can be much smaller than the p-value. This is one reason why saying “the p-value is the chance that chance produced these exact data” is inaccurate.

The test also depends on its stated hypotheses and extremeness rule. For example, a two-sided test counts outcomes that are extreme in either direction, while a one-sided test counts outcomes in the direction specified by its alternative. Those rules determine which probabilities are added; they do not turn the resulting sum into a probability about what caused the data.

Worked Example: Ten Heads in Ten Hypothetical Coin Tosses

Suppose a hypothetical coin is tossed 10 times, independently, and all 10 tosses are heads. Test \(H_0:p=0.5\) against \(H_a:p\ne 0.5\), where \(p\) is the probability of heads on each toss. Under \(H_0\), the number of heads \(X\) follows a binomial model with \(n=10\) and \(p=0.5\). The model assumes a fair coin and independent tosses.

The observed count is \(X=10\). For a two-sided test based on distance from the null expectation of 5 heads, counts at least as extreme are \(X=0\) and \(X=10\). Thus:

\begin{aligned} p\text{-value} &=P(X=0\text{ or }X=10\mid H_0)\\ &=\binom{10}{0}(0.5)^{10}+\binom{10}{10}(0.5)^{10}\\ &=\frac{1}{1024}+\frac{1}{1024}\\ &=\frac{2}{1024}\approx 0.001953. \end{aligned}

Under this fair, independent-toss model, about 0.195% of sets of 10 tosses would produce a count at least this far from 5. The exact ordered sequence “heads, heads, heads, heads, heads, heads, heads, heads, heads, heads” has probability \(1/1024\approx0.0009766\), half the p-value. The p-value includes the equally extreme all-tails outcome too.

This calculation does not show that there is a 0.195% probability that chance caused the 10 heads. It shows how often the specified two-sided tail would occur if the coin were fair and the tosses independent. If the coin were biased, the tosses were not independent, or the recording were inaccurate, the calculation would not describe that situation.

Why the Model and Design Matter

A p-value has meaning only relative to the process used to generate the data and the model used to represent that process. Before interpreting it as evidence against a null model, ask what makes the proposed probability calculation suitable for the observations. Depending on the test, relevant conditions may include:

A clearly specified null model. The null hypothesis and probability distribution must determine the probabilities used in the calculation.
A suitable data-generation mechanism. Random sampling or random assignment may justify particular probability calculations. Without it, the model may not represent how the observations were obtained.
Appropriate dependence assumptions. If a model treats observations as independent, repeated or clustered measurements cannot simply be treated as independent without justification.
A valid test statistic and extremeness rule. The statistic and the outcomes counted in the tail must match the hypotheses and the test being used.
A reasonable approximation, when one is used. If the p-value comes from an approximate distribution, that approximation must be adequate for the data and sample size in question.

These are not technical details that can be separated from the interpretation. They are part of what the p-value is conditional on. A small number calculated under a model that poorly represents the study does not become a trustworthy description of the study merely because the arithmetic is correct.

Worked Example: A Small Normal-Model P-Value

Imagine a hypothetical sample of \(n=25\) independent measurements from a population modeled as normal, with known population standard deviation \(\sigma=20\) units. The sample mean is \(\bar{x}=112\) units. Test \(H_0:\mu=100\) against \(H_a:\mu\ne100\), using the stated normal model.

Under \(H_0\), the standard error of the sample mean is:

SE=\frac{\sigma}{\sqrt{n}}=\frac{20}{\sqrt{25}}=4\text{ units}.

The standardized test statistic is:

z=\frac{\bar{x}-\mu_0}{SE} =\frac{112-100}{4}=3.

For this two-sided test, results at least as extreme as the observed statistic have \(|Z|\ge3\). Under the standard normal reference distribution:

P(|Z|\ge3)=2P(Z\ge3)\approx 2(0.001350)=0.002700.

Under the assumed normal model, with independent observations and the stated known \(\sigma\), about 0.270% of samples would give a standardized result at least this far from zero in either direction. This does not mean that there is a 0.270% probability chance caused the sample mean of 112. If the observations were strongly dependent or the model were not suitable, the reference distribution and calculated p-value might not give the relevant tail probability.

A Randomization Model Gives “Chance” a Specific Meaning

In a randomized experiment, chance can refer precisely to the random assignment mechanism. Under a null hypothesis of no treatment effect for any experimental unit, the outcomes would remain fixed while the assignment labels vary according to the design. A randomization p-value measures how often that design would produce an assignment with a statistic at least as extreme as the observed one. It does not claim that chance created the outcome values.

Worked Example: Extreme Allocation in a Hypothetical Experiment

Consider a hypothetical randomized experiment with 10 units, five assigned to treatment and five to control. The observed responses are five 10s in the treatment group and five 0s in the control group, so the observed difference in group means is \(10-0=10\) response units. Use the absolute difference in means as the test statistic, and test the sharp null hypothesis that treatment changes no unit’s response.

Under that null, the 10 responses stay fixed and only the five treatment labels vary. There are:

\binom{10}{5}=252

equally likely ways to assign five of the 10 responses to treatment. The largest possible absolute difference is 10, reached only when all five 10s are assigned to treatment and all five 0s to control, or when all five 0s are assigned to treatment and all five 10s to control. Thus two of the 252 assignments are at least as extreme as the observed assignment:

p\text{-value}=\frac{2}{252}\approx0.007937.

This calculation is justified by the hypothetical experiment’s random assignment and the sharp null: under that null, the responses do not change with assignment. The p-value describes the proportion of permitted assignments that would produce a difference at least as large in absolute value. It does not say there is a 0.7937% probability that random assignment caused the observed responses. The responses are treated as fixed for this calculation; the random element is the assignment.

What a Small P-Value Can and Cannot Tell You

If a p-value is small, then results in the defined tail would be uncommon under the specified null model. That makes the observed statistic a poor match to that model, provided the model and calculation are appropriate. It does not establish which explanation is responsible for the mismatch. Possibilities include a genuine difference from the null, a model that does not fit, a faulty measurement or recording process, selection effects, or dependence that was not accounted for.

The phrase “chance alone” can be a convenient shorthand for “the variability represented by this null model and the study’s probability mechanism.” It should not be treated as a literal claim that chance is one competing cause whose probability has been measured. A null model describes a reference process; it does not list every possible source of error or bias in a study.

Key takeaway: A small p-value is a tail probability under a specified null model, data-generation process, statistic, and extremeness rule. It is not the probability that chance caused the data. If the model or design assumptions are not credible, the p-value may not describe the study’s actual uncertainty.

Common Mistake

Incorrect reasoning: “The p-value is 0.002, so there is a 0.2% chance that random variation produced these results and a 99.8% chance that something else caused them.”

This reverses the conditioning in the p-value and turns a tail probability into a probability about causes. The p-value instead says: if the null model and relevant assumptions were correct, the probability of a result at least this extreme by the specified rule would be 0.002. It does not assign probabilities to the null model, chance, or other explanations. A careful interpretation names the model and, where relevant, the sampling or assignment mechanism that make the calculation meaningful.

Check Your Understanding

Use the conditional meaning of a p-value and its assumptions to answer these questions.

A two-sided test gives a p-value of 0.01. State what the 0.01 describes, including the condition under which it is calculated.
Why can the p-value in a test be larger than the probability of the exact observed sequence of outcomes?
In a randomized experiment, what varies in a randomization calculation under the sharp null: the observed responses, the assignment labels, or both?
A test assumes independent observations, but the data consist of repeated measurements from the same individuals. Why might the reported p-value be unreliable?
Rewrite this statement accurately: “There is a 2% chance that chance alone caused the data.”