Calculating P-Values From Null Distributions

From a Test Statistic to a Tail Probability

A p-value is calculated from the null distribution of a test statistic, but the calculation depends on the direction of the alternative hypothesis. For a right-tailed test, outcomes at or above the observed statistic count as at least as extreme in the specified direction. For a left-tailed test, outcomes at or below the observed statistic count. Discrete distributions require particular care because an outcome can have positive probability at the boundary.

The practical sequence is: identify the null distribution, record the observed statistic, establish the rejection direction, and then calculate the probability of the corresponding tail under the null. A tail probability is not calculated from whichever side looks smaller; its direction comes from the test being performed.

Definition: For a specified one-sided test with test statistic \(T\) and observed value \(t_{\text{obs}}\), the p-value is the probability under the null distribution of obtaining a value at least as extreme as \(t_{\text{obs}}\) in the direction specified by the alternative. For an upper-tail test this is \(P_{H_0}(T\ge t_{\text{obs}})\); for a lower-tail test it is \(P_{H_0}(T\le t_{\text{obs}})\).

The subscript \(H_0\) emphasizes that the calculation uses the distribution of the statistic under the null hypothesis. If the test specifies an upper-tail rejection direction, use the upper tail; if it specifies a lower-tail direction, use the lower tail. The same observed number can therefore produce different p-values for tests with different alternatives.

Discrete Null Distributions: Include the Boundary

For a discrete statistic, the probability mass function assigns probability to individual possible values. If the observed count is \(t_{\text{obs}}\), an upper-tail p-value is found by adding the probabilities of all possible values \(t\) satisfying \(t\ge t_{\text{obs}}\). A lower-tail p-value adds the probabilities for all possible values satisfying \(t\le t_{\text{obs}}\). The observed value itself is included in either sum.

\begin{aligned} p_{\text{upper}}&=\sum_{t\ge t_{\text{obs}}}P_{H_0}(T=t),\\ p_{\text{lower}}&=\sum_{t\le t_{\text{obs}}}P_{H_0}(T=t). \end{aligned}

The inclusive inequalities matter: discrete observations have positive probability, so omitting the observed outcome removes probability that belongs to the tail. For a discrete variable, a cumulative distribution function \(F_T(t)=P(T\le t)\) gives a lower-tail probability directly. The upper-tail probability is \(P(T\ge t_{\text{obs}})=1-P(T\le t_{\text{obs}}-1)\), when the statistic takes integer values. It is generally not \(1-F_T(t_{\text{obs}})\), because that expression excludes the probability at the observed value.

Worked Example: An Upper-Tail Binomial Test

A hypothetical quality check counts defective seals in a random sample of 10 seals. Assume the seals have independent outcomes and, under the null hypothesis, each seal is defective with probability \(p=0.20\). Thus \(X\sim\operatorname{Binomial}(10,0.20)\). The observed count is 5, and the specified alternative is that the defect probability is greater than 0.20. Calculate the p-value.

Method and direction. The test statistic is the defect count \(X\), and larger counts support the specified alternative. Therefore this is an upper-tail p-value:

p=P_{H_0}(X\ge 5)=\sum_{k=5}^{10}\binom{10}{k}(0.20)^k(0.80)^{10-k}.

Calculate each term using the binomial probability mass function:

\begin{aligned} P(X=5)&=\binom{10}{5}(0.20)^5(0.80)^5=0.0264241152,\\ P(X=6)&=\binom{10}{6}(0.20)^6(0.80)^4=0.0055050240,\\ P(X=7)&=\binom{10}{7}(0.20)^7(0.80)^3=0.0007864320,\\ P(X=8)&=\binom{10}{8}(0.20)^8(0.80)^2=0.0000737280,\\ P(X=9)&=\binom{10}{9}(0.20)^9(0.80)=0.0000040960,\\ P(X=10)&=\binom{10}{10}(0.20)^{10}=0.0000001024. \end{aligned}

Adding the six probabilities gives \(p=0.0327934976\), or 0.0328 to four decimal places. The calculation includes \(X=5\), the observed count, and every more extreme count in the specified direction.

Common mistake avoided: Using \(1-P(X\le5)\) would omit \(P(X=5)\). For this discrete upper tail, the complement must stop at \(X=4\): \(P(X\ge5)=1-P(X\le4)\).

Continuous Null Distributions: Use the CDF or Its Complement

For a continuous statistic with cumulative distribution function \(F_T(t)=P_{H_0}(T\le t)\), the lower-tail p-value is \(F_T(t_{\text{obs}})\). The upper-tail p-value is \(1-F_T(t_{\text{obs}})\). A continuous distribution assigns probability zero to any single point, so \(P(T\ge t_{\text{obs}})=P(T>t_{\text{obs}})\); the boundary convention does not change the result.

\begin{aligned} p_{\text{lower}}&=F_T(t_{\text{obs}}),\\ p_{\text{upper}}&=1-F_T(t_{\text{obs}}). \end{aligned}

For example, if a test statistic has a standard normal null distribution, write \(Z\sim N(0,1)\) and let \(\Phi\) denote the standard normal cumulative distribution function. A lower-tail result \(z_{\text{obs}}\) gives \(p=\Phi(z_{\text{obs}})\); an upper-tail result gives \(p=1-\Phi(z_{\text{obs}})\). A normal table or statistical calculator may provide either a CDF or an upper-tail area, so check what the reported function represents before using it.

Worked Example: A Lower-Tail Test for a Mean

A hypothetical laboratory measures a chemical concentration in 16 independent samples. Assume the measurements are normally distributed with known standard deviation \(\sigma=4\) units. The null hypothesis specifies a mean of 23 units, and the alternative is that the mean is less than 23. The observed sample mean is 21.5. Calculate the p-value using the null distribution of the sample mean.

Method and direction. Under the stated normal model and null hypothesis, the standardized sample mean is standard normal. Its standard error is

\frac{\sigma}{\sqrt{n}}=\frac{4}{\sqrt{16}}=\frac{4}{4}=1.

The observed standardized statistic is

z_{\text{obs}}=\frac{\bar{x}-\mu_0}{\sigma/\sqrt{n}} =\frac{21.5-23}{1} =-1.5.

The alternative is lower-tailed, so the p-value is the probability under \(H_0\) of a standard normal statistic at or below \(-1.5\):

p=P(Z\le -1.5)=\Phi(-1.5)=0.0668.

Thus the p-value is 0.0668 to four decimal places. The calculation uses the null mean and the known standard deviation, along with the stated sample size.

Common mistake avoided: Taking \(1-\Phi(-1.5)=0.9332\) would calculate the opposite tail. The alternative says “less than,” so the relevant probability is below the observed standardized value, not above it.

Justification: Why the Tail Formula Matches the Definition

The p-value definition asks for the probability, assuming the null model, of results at least as extreme as the observed result in the specified direction. In an upper-tail test, the set of results meeting that description is exactly the event \(\{T\ge t_{\text{obs}}\}\). In a lower-tail test, it is the event \(\{T\le t_{\text{obs}}\}\). Therefore the formulas follow by applying the probability model for \(T\) under \(H_0\) to those events.

For a continuous statistic, the CDF gives \(P(T\le t_{\text{obs}})\) directly, and the complement gives \(P(T>t_{\text{obs}})\). Since \(P(T=t_{\text{obs}})=0\) for a continuous distribution, the strict and inclusive upper-tail probabilities are equal. For a discrete integer-valued statistic, the event \(\{T\ge t_{\text{obs}}\}\) excludes only values less than \(t_{\text{obs}}\), so its complement is \(\{T\le t_{\text{obs}}-1\}\). This is why the discrete complement must account for the boundary mass.

Specify the null distribution.
Identify the distribution of the test statistic under the null hypothesis, including its parameters.

Locate the observed statistic.
Calculate or record the observed value using the definition of the test statistic.

Choose the specified tail.
Use the direction of the alternative: larger values for an upper-tail test or smaller values for a lower-tail test.

Calculate and check.
Sum discrete probability masses, or use the continuous CDF or complement. Confirm that the answer is between 0 and 1 and corresponds to the correct direction.

Worked Example: A Lower-Tail Poisson Test

A hypothetical transit operator records the number of signal faults during one fixed monitoring period. Assume independent fault arrivals follow a Poisson model. Under the null hypothesis, the expected count is \(\lambda=4\) faults per period, so \(X\sim\operatorname{Poisson}(4)\). Only 1 fault is observed, and the specified alternative is that the fault rate is lower than the null rate. Calculate the p-value.

Method and direction. Smaller counts support the alternative, so include all possible counts at or below the observation:

p=P_{H_0}(X\le1)=P(X=0)+P(X=1).

For a Poisson random variable with mean \(\lambda\), \(P(X=k)=e^{-\lambda}\lambda^k/k!\). Substituting \(\lambda=4\):

\begin{aligned} P(X=0)&=e^{-4}\frac{4^0}{0!}=e^{-4},\\ P(X=1)&=e^{-4}\frac{4^1}{1!}=4e^{-4},\\ p&=e^{-4}+4e^{-4}=5e^{-4}\\ &=5(0.0183156)=0.091578. \end{aligned}

The p-value is 0.0916 to four decimal places. Both 0 and 1 are included because both are at least as small as the observed count.

Common mistake avoided: Calculating \(P(X=1)\) alone gives approximately 0.0733, but it omits the possible outcome \(X=0\), which is also in the lower tail. The p-value concerns the full directional tail, not just the probability of the exact observed result.

Checks That Prevent Tail Errors

Before reporting a p-value, verify that the model and the calculation match the test. The null distribution must be appropriate for the statistic and assumptions. For example, a binomial model requires a fixed number of independent trials with the same success probability; a Poisson model uses a specified mean count for the observation period; and the normal calculation for a sample mean above relied on normally distributed measurements and a known standard deviation.

The following checks are especially useful in exam calculations:

Read the alternative direction. “Greater than” calls for an upper tail; “less than” calls for a lower tail.
Use the null parameter. For these examples, the probabilities and standardization are computed at the null value, not at an alternative value or at the observed estimate.
Include discrete boundary probability. An observed count belongs in its own directional tail.
Distinguish a CDF from a survival probability. A CDF is a lower-tail probability; an upper tail is its complement for a continuous statistic.
Check the scale and units. Standard errors and count-model parameters must match the statistic and observation period.

These procedures apply when a one-sided rejection direction has been specified. Calculating a p-value for a two-sided test requires a rule for what counts as equally or more extreme in both directions; it is not obtained by choosing an arbitrary tail.

Key takeaway: A p-value is a null-model probability over the outcomes at least as extreme as the observation in the specified direction. For discrete statistics, sum the appropriate probability masses and include the observed value. For continuous statistics, use the CDF for a lower tail and its complement for an upper tail.

Check Your Understanding

For each problem, identify the direction before calculating. Round each final probability to four decimal places.

Under a null model, \(X\sim\operatorname{Binomial}(5,0.30)\). An observed count is 3, and the alternative is that the success probability is greater. Calculate the p-value. (Answer: 0.1631)
Under a null model, \(X\sim\operatorname{Binomial}(4,0.50)\). The observed count is 1, and the alternative is that the success probability is less. Calculate the p-value. (Answer: 0.3125)
A test statistic has a standard normal null distribution and the specified alternative is upper-tailed. If \(z_{\text{obs}}=1.10\), calculate the p-value. (Answer: 0.1357)
Assume independent normal measurements with known standard deviation \(\sigma=6\). For a sample of \(n=9\), the null mean is 50 and the observed sample mean is 52. The alternative is that the population mean is greater than 50. Calculate the standardized statistic and the p-value. (Answer: \(z_{\text{obs}}=1.00\), \(p=0.1587\))
Under a Poisson null model with mean \(\lambda=2.5\) events per fixed period, zero events are observed. The alternative is a lower event rate. Calculate the p-value. (Answer: \(e^{-2.5}=0.0821\))

What You'll Learn