Defining a P-Value Under the Null

From an Observed Statistic to a Probability

A hypothesis test begins with data and a statistic calculated from those data. The statistic summarizes how the observed result compares with the null hypothesis. A p-value then asks a probability question: if the null model were correct, how often would the test produce a statistic at least as extreme as the one observed?

The null hypothesis alone does not determine this probability. We also need a statistical model that describes the statistic’s distribution when the null hypothesis holds, and a rule specifying which statistic outcomes count as at least as extreme. The earlier material on test statistics and measures of extremeness provides the statistic and the direction or ordering used for that comparison. This tutorial defines the probability those ingredients identify.

Definition: A p-value is the probability, calculated assuming the null hypothesis $H_0$ and the test model, of obtaining a test statistic at least as extreme as the observed statistic, according to the test’s extremeness rule. In notation, if $T$ is the test statistic, $t_{\mathrm{obs}}$ is its observed value, and $E(t_{\mathrm{obs}})$ is the set of outcomes counted as at least as extreme, then $$ p\text{-value}=P\bigl(T\in E(t_{\mathrm{obs}})\mid H_0,\text{ test model}\bigr). $$

The expression “assuming the null hypothesis” means that the probability is calculated from the distribution specified by the null model. It does not mean that we have established that the null is true. The statistic’s observed value comes from the actual data; the reference distribution describes the values that statistic could take across repetitions under the stated null model.

A p-value is therefore not generally the probability of obtaining exactly the observed data. For a continuous statistic, the probability of any one exact value is typically zero. Instead, the p-value is the probability of the observed statistic and all outcomes the test rule regards as at least as extreme.

How the Null Model Defines the Calculation

Before calculating a p-value, state the hypotheses, the statistic, and the assumptions that justify its null distribution. Once these are specified, the calculation is a probability from that distribution. For a right-tail rule, the p-value is $P(T\ge t_{\mathrm{obs}}\mid H_0)$; for a left-tail rule, it is $P(T\le t_{\mathrm{obs}}\mid H_0)$. Which rule is appropriate is determined by the test’s stated direction and extremeness convention, not selected after seeing which probability is smaller.

For example, if the null model says $T\sim N(0,1)$ and the observed value is $z_{\mathrm{obs}}=2$, a right-tail p-value is the area under the standard normal curve at 2 and to its right. In symbols, it is $P(Z\ge 2\mid H_0)$, not the height of the curve at 2 and not the probability that $H_0$ is true.

Worked Example: A Right-Tail Normal Test

Suppose a hypothetical filling process is being assessed against a null mean fill amount of 50 millilitres. A random sample of $n=36$ containers has sample mean $\bar{x}=53$ millilitres. Assume independent, normally distributed fill amounts and that the population standard deviation is known to be $\sigma=9$ millilitres. Test $H_0:\mu=50$ against $H_a:\mu>50$, using the standardized sample mean as the statistic.

Under the stated normal model, the sample mean has mean $\mu$ and standard error $\sigma/\sqrt{n}$. When $H_0$ holds, the standardized statistic has a standard normal distribution:

Z=\frac{\bar{x}-50}{9/\sqrt{36}},\qquad Z\sim N(0,1)\text{ under }H_0.

The assumptions are that the sample is random, the observations are independent, the fill amounts are normally distributed, and the population standard deviation is known. The example stipulates these conditions, so the standard normal reference distribution applies.

Substitute the observed values. The standard error is $9/\sqrt{36}=9/6=1.5$ millilitres, and the observed statistic is

z_{\mathrm{obs}}=\frac{53-50}{1.5}=\frac{3}{1.5}=2.

Because the alternative is an increase, the specified extremeness rule counts statistics at least this large. Thus

p\text{-value}=P(Z\ge 2\mid H_0)=1-\Phi(2)\approx 1-0.97725=0.02275.

Under the null model, the probability of obtaining a standardized sample mean of 2 or greater is about 0.02275, or 2.275%. In context, this is the null-model probability of seeing a sample mean at least as high as 53 millilitres, expressed through the test statistic.

Why the Statistic’s Null Distribution Matters

A raw difference is not enough to determine a p-value. The null distribution accounts for how variable the statistic is expected to be when $H_0$ holds. The same difference between a sample estimate and its null value can correspond to different probabilities when the sample size or variability changes. The p-value is calculated from the distribution of the chosen statistic, not from the difference alone.

A useful justification follows directly from the definition. Suppose the null model gives a continuous statistic $T$ with density $f_0(t)$, and the test’s rule for an observed $t_{\mathrm{obs}}$ counts values $t\ge t_{\mathrm{obs}}$. The probability of that event is the integral of the null density over the entire right-tail region:

P(T\ge t_{\mathrm{obs}}\mid H_0) =\int_{t_{\mathrm{obs}}}^{\infty} f_0(t)\,dt.

This integral explains why the p-value is an area, not a density height. It includes every value in the region counted by the test rule. For a discrete statistic, the corresponding probability is a sum of the probabilities of the included outcomes rather than an integral.

Worked Example: A Left-Tail One-Sample t Test

In a hypothetical exercise study, the null mean change in a measured outcome is 20 units. A random sample of 10 participants has $\bar{x}=18.5$ units and $s=\sqrt{10}$ units. Consider $H_0:\mu=20$ versus $H_a:\mu<20$, using the one-sample t statistic.

The population standard deviation is unknown, so the statistic uses the sample standard deviation. The one-sample t model applies if observations are independent random observations from a normally distributed population. These are the assumptions for this calculation; because the sample is small, the normal-population condition is particularly important.

Under $H_0$, the statistic follows a t distribution with $n-1=9$ degrees of freedom. Its standard error is

SE=\frac{s}{\sqrt{n}}=\frac{\sqrt{10}}{\sqrt{10}}=1\text{ unit}.

Therefore, the observed statistic is

t_{\mathrm{obs}}=\frac{\bar{x}-20}{s/\sqrt{n}} =\frac{18.5-20}{1} =-1.50.

The alternative is a decrease, so the left-tail probability is the p-value:

p\text{-value}=P(T_9\le -1.50\mid H_0) \approx 0.08393.

If the null mean and the t model were correct, about 8.393% of samples would produce a t statistic of $-1.50$ or lower under this test rule. The probability refers to the statistic’s null distribution, with the statistic calculated in units of its estimated standard error.

Discrete Statistics Use Sums of Probabilities

Not every test statistic varies continuously. Counts, for example, can take only whole-number values. The definition is unchanged: identify the outcomes included by the test rule and add their probabilities under the null model. The observed outcome is included, along with outcomes at least as extreme according to the stated rule.

Worked Example: A Poisson Count Test

A hypothetical service desk receives an average of 2 calls during a particular hour. Under the null model, the number of calls in an hour follows a Poisson distribution with rate $\lambda=2$. In one hour, 5 calls are recorded. Consider $H_0:\lambda=2$ against $H_a:\lambda>2$, with a larger count treated as more extreme.

This model assumes a constant call rate during the hour and independent arrivals, so that the call count in a fixed time interval follows the stated Poisson distribution. With those assumptions, $X\sim\operatorname{Poisson}(2)$ under $H_0$. The observed count is $x_{\mathrm{obs}}=5$, and the right-tail p-value is the probability of 5 or more calls.

\begin{aligned} p\text{-value} &=P(X\ge 5\mid \lambda=2)\\ &=\sum_{k=5}^{\infty}P(X=k\mid \lambda=2)\\ &=1-\sum_{k=0}^{4} e^{-2}\frac{2^k}{k!}\\ &=1-e^{-2}\left(1+2+2+\frac{8}{6}+\frac{16}{24}\right)\\ &=1-7e^{-2}\\ &\approx 0.05265. \end{aligned}

The null model assigns probability about 0.05265 to a count of at least 5 calls in an hour. This result is obtained by summing probabilities for the observed count and all larger counts specified by the test rule. It is not the probability of observing exactly 5 calls.

A General Calculation and Its Limits

The examples used different reference distributions, but the logic was the same each time. First, the hypotheses and model specify how the statistic behaves under $H_0$. Next, the data give an observed statistic. Finally, the test’s extremeness rule identifies an event in the statistic’s null distribution, and the p-value is the probability of that event.

State the null model.
Specify $H_0$, the sampling or measurement assumptions, and the reference distribution for the test statistic under $H_0$.

Calculate the observed statistic.
Use the data and the stated formula to obtain $t_{\mathrm{obs}}$, $z_{\mathrm{obs}}$, or another observed test-statistic value.

Find the null probability.
Apply the test’s extremeness rule to identify the relevant event, then calculate its probability under the null distribution.

The result is meaningful only to the extent that the model is appropriate. If the observations are dependent when independence is required, or the assumed distribution is not reasonable, the reference probabilities may not describe the actual sampling behavior. In such cases, a precisely calculated tail area can still be a poor p-value for the situation. Assumptions should therefore be checked before interpreting a numerical result.

Common Mistakes

Using the probability of the exact observed statistic. For a continuous statistic that probability is typically zero. The p-value includes the observed statistic and the outcomes counted as at least as extreme.
Using a probability without naming the null model. A tail area has meaning only relative to a specified distribution under $H_0$. State the model and its assumptions before calculating.
Confusing the statistic with the p-value. A statistic such as $t_{\mathrm{obs}}=-1.50$ is a location in a reference distribution; the p-value, such as 0.08393, is the probability of the specified event under that distribution.
Describing the result without context. The p-value is not the chance of the observed measurement alone. It is a probability for test-statistic outcomes under the null model, interpreted in the units and setting of the investigation.

Key takeaway: A p-value is a null-model probability. It is calculated by locating the observed test statistic in its reference distribution and finding the probability of that statistic and outcomes at least as extreme under the test’s stated rule.

Check Your Understanding

For each question, focus on what probability the p-value represents and which model supplies it.

In a right-tail test, the observed statistic is $z_{\mathrm{obs}}=1.4$ under a standard normal null model. Write the probability expression for the p-value.
Why is a p-value for a continuous statistic not usually the probability of observing exactly the observed statistic?
A one-sample t test has $n=16$, and its null model uses $df=15$. What information is needed, besides the degrees of freedom, to calculate its p-value?
For a count test, explain why a p-value may be calculated by adding probabilities for several count values.
A calculation produces a numerical tail area, but the observations were dependent even though the test model assumes independence. What concern does this raise?