Tutorials › Introduction to Statistics › Why a P-Value Is Not the Probability the Null Is True

Home  /  Tutorials  /  The Logic of Tests and P-Values

The Logic of Tests and P-Values · Tutorial 610 of 996

Why a P-Value Is Not the Probability the Null Is True

Distinguish probability under the null from probability of the null given the data, and see why the latter requires a model and prior probabilities.

Intermediate 9 min read

What You'll Learn

  • State the conditional probability represented by a p-value.
  • Distinguish \(P(A\mid H_0)\) from \(P(H_0\mid A)\).
  • Use Bayes’ theorem to calculate a posterior probability when a complete model is specified.
  • See how different prior probabilities can produce different posterior probabilities from the same event.
  • Identify why a p-value alone cannot determine the probability that the null hypothesis is true.

Two Different Conditional Probabilities

A p-value is often described in ordinary language as “the probability of the result under the null hypothesis.” That wording can be useful shorthand, but it is easy to reverse the conditioning. A p-value is calculated assuming the null model; it does not directly tell us the probability that the null hypothesis is true after seeing the data.

The distinction is between two conditional probabilities. The first asks how likely an event in the data would be if the null hypothesis were the model. The second asks how likely the null hypothesis is after observing that event. These questions condition on different things, so their answers need not be equal.

Definition: Let \(H_0\) be a null hypothesis and let \(A\) be the event that the observed test statistic is at least as extreme as the one obtained, according to the test’s definition of extremeness. The p-value is \(P(A\mid H_0)\), calculated under the null model and the test’s assumptions. The reversed conditional probability is \(P(H_0\mid A)\): the probability assigned to the null after observing \(A\). A p-value does not, by itself, give this reversed probability.

The event \(A\) generally includes the observed result and other outcomes at least as extreme. Therefore, a p-value is not usually the probability of obtaining the exact data observed. In a continuous model, the probability of one exact numerical value can be zero, even though the probability of a tail event containing that value is positive.

This tutorial uses \(A\) to make the conditioning visible. The p-value concerns \(P(A\mid H_0)\). To calculate \(P(H_0\mid A)\), one needs more than that conditional probability: at a minimum, one needs prior probabilities for the competing hypotheses and the probability of \(A\) under the alternatives being considered.

Why Reversing the Condition Requires More Information

Bayes’ theorem relates the two conditional probabilities, but it does not make them interchangeable. Suppose, for illustration, that the possibilities are two models, \(H_0\) and \(H_1\), which are mutually exclusive and collectively exhaustive for the calculation. Let their prior probabilities be \(P(H_0)\) and \(P(H_1)\), and suppose \(P(A)>0\). The event \(A\) can occur under either model.

$$ P(H_0\mid A) = \frac{P(A\mid H_0)P(H_0)} {P(A\mid H_0)P(H_0)+P(A\mid H_1)P(H_1)} $$

The denominator comes from the law of total probability: if exactly one of \(H_0\) or \(H_1\) holds, then \(A\) occurs either under \(H_0\) or under \(H_1\). In particular,

$$ P(A)=P(A\mid H_0)P(H_0)+P(A\mid H_1)P(H_1). $$

For a short derivation, begin with the definition of conditional probability: \(P(H_0\mid A)=P(H_0\cap A)/P(A)\). The same definition gives \(P(A\mid H_0)=P(H_0\cap A)/P(H_0)\), so \(P(H_0\cap A)=P(A\mid H_0)P(H_0)\). Substituting this into the first expression and then using the law of total probability for \(P(A)\) gives Bayes’ formula above. The prior probability and the alternative-model probability appear because both affect how plausible \(H_0\) is among the models that could have produced \(A\).

Key distinction: \(P(A\mid H_0)\) asks about possible data while treating \(H_0\) as the model. \(P(H_0\mid A)\) asks about the hypothesis after observing data. The second probability requires a probability model for the hypotheses, including prior probabilities; it cannot generally be recovered from a p-value alone.

Worked Example: A Positive Test and a Rare Condition

Worked Example: A Positive Test and a Rare Condition

Suppose, hypothetically, that a condition affects 1% of a population. A test has sensitivity 95% and specificity 90%. What is the probability that a person has the condition given a positive result?

Let \(D\) be the event that a person has the condition and \(+\) the event of a positive test. The stated assumptions are \(P(D)=0.01\), \(P(D^c)=0.99\), \(P(+\mid D)=0.95\), and \(P(+\mid D^c)=0.10\). The last probability is 10% because specificity of 90% means \(P(-\mid D^c)=0.90\). We assume these rates apply to the population in question.

First find the overall probability of a positive result by accounting for both people with the condition and people without it:

$$ P(+)=P(+\mid D)P(D)+P(+\mid D^c)P(D^c) =(0.95)(0.01)+(0.10)(0.99)=0.1085. $$

The joint probability of having the condition and testing positive is \(P(+\mid D)P(D)=(0.95)(0.01)=0.0095\). Therefore,

$$ P(D\mid +) = \frac{P(+\mid D)P(D)}{P(+)} = \frac{(0.95)(0.01)}{0.1085} \approx 0.08756. $$

Under these hypothetical rates, the probability of having the condition given a positive result is about 8.756%. This is not the test’s sensitivity: sensitivity is \(P(+\mid D)=0.95\), the probability of a positive result among people who have the condition. The calculation illustrates how reversing the condition changes the question, and why the population prevalence matters.

Worked Example: A Tail Probability Is Not a Posterior Probability

Worked Example: A Tail Probability Is Not a Posterior Probability

Imagine ten independent tosses of a coin and suppose the observed result is nine heads. Let \(A\) be the event of getting at least nine heads. Under \(H_0\), assume the coin has probability 0.5 of heads on each toss. What is \(P(A\mid H_0)\)? Then consider whether that probability alone determines \(P(H_0\mid A)\).

Under the stated null model, the number of heads has a binomial distribution with \(n=10\) and probability of heads 0.5. Independence and the same probability of heads on each toss are assumptions of this model. The event \(A\) includes exactly nine heads or exactly ten heads, so

$$ \begin{aligned} P(A\mid H_0) &=\binom{10}{9}(0.5)^9(0.5)+(0.5)^{10}\\ &=10(0.5)^{10}+(0.5)^{10}\\ &=\frac{11}{1024}\\ &\approx 0.01074. \end{aligned} $$

Thus, the probability of at least nine heads under this null model is about 1.074%. This is the p-value for the specified upper-tail event. It is not yet the probability that the coin is fair given the result.

To calculate a posterior probability, add a specific alternative model and prior probabilities. For this illustration only, let \(H_1\) be a model in which tosses are independent and the probability of heads is 0.8. Under that model,

$$ \begin{aligned} P(A\mid H_1) &=\binom{10}{9}(0.8)^9(0.2)+(0.8)^{10}\\ &=10(0.134217728)(0.2)+0.1073741824\\ &=0.268435456+0.1073741824\\ &=0.3758096384. \end{aligned} $$

Now suppose the prior probabilities are \(P(H_0)=0.90\) and \(P(H_1)=0.10\). These are added assumptions, not quantities supplied by the p-value. Bayes’ theorem gives

$$ \begin{aligned} P(H_0\mid A) &=\frac{P(A\mid H_0)P(H_0)} {P(A\mid H_0)P(H_0)+P(A\mid H_1)P(H_1)}\\ &=\frac{(0.0107421875)(0.90)} {(0.0107421875)(0.90)+(0.3758096384)(0.10)}\\ &\approx \frac{0.00966797}{0.0472489}\\ &\approx 0.2046. \end{aligned} $$

Under this particular two-model setup and prior, the posterior probability assigned to \(H_0\) after event \(A\) is about 20.46%. It is not 1.074%, the probability of event \(A\) under \(H_0\).

Change only the prior probabilities to \(P(H_0)=0.99\) and \(P(H_1)=0.01\). The same event and the same conditional probabilities now give

$$ \begin{aligned} P(H_0\mid A) &=\frac{(0.0107421875)(0.99)} {(0.0107421875)(0.99)+(0.3758096384)(0.01)}\\ &=\frac{0.0106347656}{0.0143928620}\\ &\approx 0.7389. \end{aligned} $$

The p-value remains about 0.01074 because the null model and the event \(A\) have not changed. The posterior changes because the prior probabilities have changed. These calculations do not imply that this two-model Bayesian setup is required for ordinary hypothesis testing; they show why the p-value cannot, by itself, determine a posterior probability.

Worked Example: Same Null Probability, Different Posterior

Worked Example: Same Null Probability, Different Posterior

Suppose a test produces an event \(A\) defined as a result at least as extreme as the observed statistic, and its p-value is 0.03. To see what else is needed for a posterior calculation, consider a deliberately simplified model with only two possibilities. Assume \(P(A\mid H_0)=0.03\), \(P(A\mid H_1)=0.30\), and initially \(P(H_0)=P(H_1)=0.50\). The likelihood under \(H_1\) and the prior probabilities are assumptions supplied for this illustration.

Applying Bayes’ theorem,

$$ \begin{aligned} P(H_0\mid A) &=\frac{(0.03)(0.50)} {(0.03)(0.50)+(0.30)(0.50)}\\ &=\frac{0.015}{0.165}\\ &\approx 0.09091. \end{aligned} $$

Under these assumptions, the posterior probability assigned to \(H_0\) is about 9.091%. Now retain both conditional probabilities but change the prior to \(P(H_0)=0.95\) and \(P(H_1)=0.05\):

$$ \begin{aligned} P(H_0\mid A) &=\frac{(0.03)(0.95)} {(0.03)(0.95)+(0.30)(0.05)}\\ &=\frac{0.0285}{0.0435}\\ &\approx 0.6552. \end{aligned} $$

The p-value is 0.03 in both calculations, yet the posterior probabilities differ substantially. The difference is not a contradiction: the calculations answer different conditional-probability questions under different prior assumptions. Without specifying those assumptions and the alternative model, there is no unique posterior probability to calculate from the p-value.

Common Mistake: Reading the Condition Backwards

A common error is to see a p-value of 0.03 and conclude, “There is a 3% probability that the null hypothesis is true.” This treats \(P(A\mid H_0)\) as though it were \(P(H_0\mid A)\). The two expressions condition on different events and are not interchangeable.

A careful statement keeps the condition explicit: “Assuming the null model and the test’s assumptions, the probability of a result at least as extreme as the one observed is 0.03.” That statement describes the data’s probability under a specified model. It does not assign a probability to the truth of the null hypothesis.

A posterior probability can be meaningful when a probability model for hypotheses, prior probabilities, and likelihoods under the competing models have been specified. But a p-value alone supplies only a null-model tail probability. In particular, it does not specify the prior plausibility of the null or the probability of the result under an alternative. Keeping those ingredients separate prevents a conditional probability from being mistaken for its reverse.

Key takeaway: A p-value is a probability of data at least as extreme as observed, conditional on the null model. The probability of the null conditional on the data is a different quantity; calculating it requires additional modeling assumptions and prior probabilities.

Check Your Understanding

For each question, pay attention to what is being conditioned on.

  1. In words, what question does \(P(A\mid H_0)\) answer, and how does it differ from \(P(H_0\mid A)\)?
  2. Why is a p-value generally a probability of a tail event rather than the probability of the exact observed data?
  3. A test has sensitivity 90%. Does that mean that a person with a positive result has a 90% probability of having the condition? Explain what additional information would be needed.
  4. In the coin example, why did the posterior probability change when the prior probabilities changed, even though the p-value stayed the same?
  5. What additional quantities, besides \(P(A\mid H_0)\), appear in the two-model Bayes calculation for \(P(H_0\mid A)\)?

Previous

Next

Back to Tutorials