What a P-Value Does and Does Not Say
A p-value is often summarized as “how surprising the data are,” but that phrase is incomplete unless it identifies the model under which the data are being judged. A p-value is calculated assuming a specified null hypothesis and its associated sampling model. Its meaning is conditional: it describes how often the model would produce a test statistic at least as extreme as the one observed.
That conditional statement is useful evidence about compatibility between the observed statistic and the null model. It is not the probability that the null hypothesis is true. A small p-value indicates that the observed statistic would be relatively unusual if the specified null model were correct; it does not by itself say what probability should be assigned to the null hypothesis.
The phrase “under the null model” matters. It includes the null hypothesis and the probability model used to obtain the test statistic’s reference distribution. Independence assumptions, a specified variance, a distributional form, and the way the test was chosen can all matter. A reported p-value is not an unconditional probability about the study; it is conditional on those modeling choices.
Why the P-Value Is a Tail Probability
A test statistic reduces the sample to a number whose behavior under the null model is known or specified. The test’s direction determines which outcomes count as more extreme. For an upper-tail test, larger values count as more extreme; for a lower-tail test, smaller values do. For a two-sided test, extremeness is assessed in either direction according to the test’s definition.
For a continuous test statistic with null density \(f_0\), an upper-tail p-value is the area under that density at and above the observed value:
This follows directly from the event whose probability is being calculated: under the null model, the statistic can take many possible values, and the p-value adds up the probabilities of all values in the designated tail. For a discrete statistic, the corresponding probabilities are summed, including the observed value. Thus, a p-value is generally not the probability of obtaining exactly the observed data. It is the probability of the observed statistic and outcomes deemed at least as extreme.
For a symmetric, continuous null distribution and a two-sided test based on distance from the null value, the two tails have equal area. In that particular setting, the two-sided p-value is twice the one-sided tail area in the observed direction. That shortcut does not apply automatically to every test: the test’s definition of “at least as extreme” must be used.
Worked Interpretations
Worked Example: A One-Sided Test of a Sensor Mean
A hypothetical laboratory checks a sensor that should have a mean reading of 100 units. It takes an independent random sample of \(n=25\) readings. Assume readings are normally distributed with known variance \(100\), so each reading has standard deviation \(10\). The observed sample mean is 104. Test \(H_0:\mu=100\) against \(H_1:\mu>100\), and interpret the p-value.
Method. Use the standardized sample mean and the upper tail, because the alternative is that the population mean exceeds 100. Under the null model, \(\bar{X}\sim N(100,100/25)=N(100,4)\); the second parameter is the variance, so the standard error is \(\sqrt{4}=2\).
The observed standardized statistic is
Under \(H_0\), the standardized statistic has a standard normal distribution. Therefore,
Interpretation. If the sensor’s true mean were 100 and the stated normal, independent-sampling model were appropriate, the probability of a sample mean of at least 104 (equivalently, a statistic of at least 2) would be 0.0228. This is evidence that the observed statistic is unusual relative to that null model. It is not a 2.28% probability that \(\mu=100\).
Common mistake avoided. The alternative is one-sided, so the p-value is the upper-tail area, 0.0228. Doubling it would answer a different, two-sided question.
Check. Since the observed mean is above 100, the upper-tail p-value is below 0.5, as expected.
Worked Example: A Two-Sided Test of a Filling Process
A hypothetical filling process has target mean 50 millilitres. An independent random sample of 16 fills is modeled as normally distributed with known variance 64. The sample mean is 53 millilitres. For \(H_0:\mu=50\) against \(H_1:\mu\neq 50\), calculate and interpret the p-value.
Method. Use a two-sided standardized test. Under the null model, the sample mean has variance \(64/16=4\) and standard error \(2\). The standardized observed statistic is
Because the null distribution is standard normal and symmetric, statistics at least as far from zero as 1.5 lie in the two tails. Using \(\Phi(1.5)=0.9332\),
Interpretation. Under the specified null model, there is a 0.1336 probability of a standardized sample mean at least 1.5 units from zero in either direction. This p-value describes how often such a statistic or a more extreme one would occur if the null model held; it does not give the probability that the target mean is correct.
Common mistake avoided. Using only \(0.0668\) would count the upper tail but omit results equally far below the null value. The two-sided alternative requires both tails here.
Check. The two-sided p-value, 0.1336, is twice the one-sided area in the direction of the observed statistic, 0.0668, because this null distribution is continuous and symmetric and the test is defined by absolute distance.
Worked Example: An Exact Binomial Tail for a Rare Event
A hypothetical review samples 10 independent transactions. Under a specified baseline model, each transaction has probability \(p=0.20\) of requiring a particular manual review. Let \(X\) be the number requiring review. The observed count is 5. For \(H_0:p=0.20\) against \(H_1:p>0.20\), calculate and interpret the exact p-value.
Method. Under the null model, \(X\sim\operatorname{Binomial}(10,0.20)\). Since larger counts support the stated alternative, the p-value is the probability of 5 or more reviews under \(H_0\):
The six terms are
Adding the terms gives \(0.0327935\), or \(0.0328\) to four decimal places.
Interpretation. If each transaction independently had a 0.20 review probability, the probability of observing at least 5 reviews in 10 transactions would be 0.0328. This is the exact upper-tail probability for this discrete statistic under the null model. It is not the probability that the baseline probability is 0.20.
Common mistake avoided. The exact p-value includes the probability of 5 reviews as well as the probabilities of 6 through 10. Using only \(P(X=5)\) would calculate the chance of exactly the observed count, not the upper-tail p-value.
Check. The event \(X\geq5\) includes all counts at least as large as the observed count, and the sum is greater than the \(X=5\) term alone.
Why a P-Value Is Not a Probability of the Null
The conditional probability in a p-value runs from a model assumption to possible data: assuming \(H_0\), how probable are results at least this extreme? The probability that a hypothesis is true given observed data asks the reverse question: given the data, how probable is \(H_0\)? These are different conditional probabilities. A p-value calculation does not reverse the conditioning.
Assigning a probability to a hypothesis after seeing data requires additional ingredients, including a probability model for the alternatives and, in a Bayesian analysis, prior probabilities. The p-value alone supplies neither. For that reason, a p-value of 0.03 does not mean there is a 3% chance the null is true, nor does it mean there is a 97% chance the alternative is true.
| Statement | What it means | Does a p-value provide it? |
|---|---|---|
| Probability of a tail event under \(H_0\) | How often the specified null model produces a statistic at least as extreme as observed | Yes, this is the p-value’s meaning |
| Probability that \(H_0\) is true given the data | A probability assigned to the hypothesis after observing data | No |
| Size or importance of an effect | How large a difference is, and whether it matters in context | No; report an estimate and relevant scale separately |
| Probability the model assumptions are correct | Whether the distributional and sampling assumptions hold | No; those assumptions are used to calculate the p-value |
A small p-value is evidence against the particular null model used for the calculation, provided the test and model assumptions are appropriate. It does not identify which assumption, if any, is responsible for an unusual result. Nor does it establish that an effect is practically important: with enough precision, a small departure from a null value can produce a small p-value. Conversely, a p-value that is not small does not establish that the null hypothesis is true. It says only that this statistic is not especially unusual relative to the specified null distribution.
Common Overclaims to Avoid
- “The p-value is the chance the null is true.” It is a probability of data summaries conditional on the null model, not a probability of the null given the data.
- “The p-value is the chance the result happened by chance.” This is too vague. The p-value refers to a defined tail event under a specified model, not to every possible source of randomness or bias.
- “A small p-value proves the alternative.” It indicates that the statistic is unusual under the null model. The calculation does not prove a particular alternative or rule out model problems.
- “A large p-value proves there is no effect.” A p-value is not a measure of the probability that the null is true. Interpret it alongside the estimate, uncertainty, and study design.
- “A small p-value means a large or important effect.” Statistical evidence and practical importance are different questions. The p-value alone does not quantify the effect’s size.
Check Your Understanding
For each problem, focus on the event the p-value describes and the model on which that probability is conditioned.
- A test statistic is standard normal under \(H_0\), and the prespecified alternative is upper-tailed. The observed value is \(z=1.40\). Find the p-value and state its conditional interpretation. (Answer: \(p=0.0808\); under \(H_0\), the probability of \(Z\geq1.40\) is 0.0808.)
- Under a standard normal null distribution, a lower-tailed test gives \(z_{\text{obs}}=-1.75\). Calculate the p-value and explain why the upper tail is not used. (Answer: \(p=P(Z\leq-1.75)=0.0401\); the alternative concerns smaller values.)
- A two-sided test has a standard normal null distribution and observed statistic \(z=2.24\). Calculate the p-value to four decimals and state what event it represents. (Answer: \(p=2[1-\Phi(2.24)]=0.0251\); it is the probability under \(H_0\) of \(|Z|\geq2.24\).)
- Let \(X\sim\operatorname{Binomial}(6,0.10)\) under \(H_0\). For an upper-tailed test, the observed count is 2. Calculate the exact p-value by summing \(P(X=k)\) for \(k=2,\ldots,6\). (Answer: \(p=0.1143\), the null-model probability of at least 2 successes.)
- A correctly specified test reports \(p=0.031\). A colleague says, “There is a 3.1% probability that the null hypothesis is true.” Explain the error and give a correct interpretation. (Answer: The probability is conditional in the wrong direction; 0.031 is the probability under the specified null model of a statistic at least as extreme as observed, not \(P(H_0\mid\text{data})\).)