The P-Value as a Tail Probability

What a P-Value Measures

A test statistic summarizes the sample in a way that is relevant to the hypotheses. A p-value then measures how much probability the null model assigns to outcomes at least as extreme as the observed statistic. The key phrase is at least as extreme: the direction or measure of extremeness must come from the test design and its alternative, not be chosen after seeing the data.

For a one-sided test, the alternative identifies a direction. If larger values of the statistic support the alternative, the relevant probability is an upper tail. If smaller values support it, the relevant probability is a lower tail. For a two-sided test, the test must specify how departures in either direction will be measured.

Definition: Let $T$ be a test statistic, let $t_{\text{obs}}$ be its observed value, and suppose the null hypothesis specifies a distribution for $T$. The p-value is the null probability of the event that the statistic is at least as extreme as $t_{\text{obs}}$, where “at least as extreme” is defined by the prespecified alternative and test statistic. For a simple null and an upper-tail test, $p=P_{H_0}(T\ge t_{\text{obs}})$; for a lower-tail test, $p=P_{H_0}(T\le t_{\text{obs}})$.

The notation $P_{H_0}$ means probability calculated under the null model. The p-value is calculated after the data are observed, so it is a number summarizing the observed result relative to that model. It is not the probability that the null hypothesis is true.

Tail Direction and the Alternative

The test statistic and alternative work together. An upper-tail test is appropriate when large values of $T$ support the alternative; a lower-tail test is appropriate when small values support it. The tail includes the observed value itself. In a discrete model, outcomes tied with the observed statistic therefore contribute to the p-value.

Test direction	What supports the alternative?	Tail event for observed value $t_{\text{obs}}$
Upper-tail	Larger values of $T$	$T\ge t_{\text{obs}}$
Lower-tail	Smaller values of $T$	$T\le t_{\text{obs}}$
Two-sided	Departures in either direction	Defined by the test’s prespecified measure of extremeness

For a two-sided test, there is not one universal rule that applies to every statistic and null distribution. A test might define extremeness through the absolute value of a signed statistic, or through another statistic designed to measure departure from the null. The rule must be set before examining the result. Simply doubling a one-sided probability is not generally justified, especially when the null distribution is asymmetric or discrete.

Worked Example: An Upper-Tail Test for an Alarm Count

A hypothetical monitoring unit produces an alarm on each independent trial with probability $0.20$ when operating normally. Five trials are run. Test $H_0:p=0.20$ against $H_a:p>0.20$, using the number of alarms $X$ as the test statistic. Assume the trials are independent Bernoulli observations with the same alarm probability, and suppose three alarms are observed. Find the p-value.

Method. The alternative says the alarm probability is higher than the null value, so unusually large counts support $H_a$. Under $H_0$, $X\sim\operatorname{Binomial}(5,0.20)$. The p-value includes every count at least as large as the observed count:

p=P_{H_0}(X\ge 3)=P(X=3)+P(X=4)+P(X=5).

Using the binomial probability formula, the three terms are

\begin{aligned} P(X=3)&=\binom{5}{3}(0.20)^3(0.80)^2 =10(0.008)(0.64)=0.0512,\\ P(X=4)&=\binom{5}{4}(0.20)^4(0.80) =5(0.0016)(0.80)=0.0064,\\ P(X=5)&=\binom{5}{5}(0.20)^5 =0.00032. \end{aligned}

Thus $p=0.0512+0.0064+0.00032=0.05792$, or 0.0579 to four decimal places. The calculation includes three alarms as well as counts of four and five, because ties with the observed statistic belong in the upper tail.

Check. The terms are nonnegative, and their sum is less than 1, as a probability must be. A common mistake would be to use only $P(X=3)=0.0512$; that is the probability of the exact observed count, not the probability of an outcome at least as extreme in the specified direction.

Two-Sided Extremeness Must Be Specified

For a two-sided alternative, a large positive statistic and a large negative statistic may both count as evidence against the null. One possible design uses the magnitude $|T|$ as the measure of departure. With that design, the p-value is the null probability that $|T|$ is at least as large as the observed magnitude. This definition should not be silently substituted for another two-sided convention.

Worked Example: A Two-Sided Test With a Discrete Null Distribution

A hypothetical calibration study uses a signed statistic $T$, with larger positive values indicating one kind of departure from the target and larger negative magnitudes indicating departure in the other direction. The test was specified in advance to use $|T|$ as its measure of extremeness. Under the simple null hypothesis, $T$ has this exact distribution:

Value of $T$	$-2$	$-1$	0	1	2
Null probability	0.08	0.17	0.50	0.15	0.10

The probabilities sum to $0.08+0.17+0.50+0.15+0.10=1.00$, so this is a probability distribution. Suppose the observed statistic is $T=-1$. Its magnitude is $|-1|=1$. By the prespecified rule, all outcomes with magnitude at least 1 count:

p=P_{H_0}(|T|\ge 1) =P(T=-2)+P(T=-1)+P(T=1)+P(T=2).

Substituting the four relevant probabilities gives $p=0.08+0.17+0.15+0.10=0.50$. The p-value is therefore 0.5000 to four decimal places. The outcome $T=0$ is excluded because its magnitude is below the observed magnitude.

Check. The included probabilities total one half, and the excluded probability is $P(T=0)=0.50$. A common mistake would be to take only the lower tail $P(T\le -1)=0.25$. That ignores outcomes in the opposite direction even though this test explicitly defines extremeness using $|T|$.

Why the Tail Probability Matches a Tail Test

The p-value’s connection to a test decision follows directly from how a one-sided test orders outcomes. Consider a simple null hypothesis and a test that rejects for large values of $T$. If its rejection region is $T\ge c$, then the observed value lies in that region exactly when it reaches the critical threshold. The p-value records the null probability of reaching or exceeding the observed value, so it also records the null probability of the tail beyond that observation.

t_{\text{obs}}\ge c \quad\Longrightarrow\quad \{T\ge t_{\text{obs}}\}\subseteq\{T\ge c\} \quad\Longrightarrow\quad P_{H_0}(T\ge t_{\text{obs}}) \le P_{H_0}(T\ge c).

Thus, if the test’s null rejection probability is at most a chosen level $\alpha$, an observation in the rejection region has an upper-tail p-value no larger than $\alpha$. Conversely, for a test that rejects when the upper-tail p-value is at most $\alpha$, the p-value rule selects outcomes from the high-statistic tail. With a discrete statistic, the tail probability can jump between attainable values; this can make the rejection probability smaller than the target level.

This argument depends on the test being ordered by the same statistic and direction used to define the p-value. It does not justify changing direction after seeing the data or choosing whichever tail gives a smaller probability.

Composite Null Hypotheses

A simple null hypothesis specifies a single probability model. A composite null allows more than one parameter value, so the distribution of the statistic may vary within the null. To obtain a p-value that accounts for all null values, a common definition takes the largest relevant tail probability over the null parameter set. This is a conservative choice: it does not report only the most favorable null value for making the observation seem unusual.

Definition: For a composite null parameter set $\Theta_0$ and a prespecified tail event $E(t_{\text{obs}})$, a conservative p-value is $$ p=\sup_{\theta\in\Theta_0}P_{\theta}\bigl(T\in E(t_{\text{obs}})\bigr). $$ The event $E(t_{\text{obs}})$ is determined by the statistic and alternative. The supremum takes the largest null tail probability across parameter values allowed by $H_0$.

In some models, the largest probability occurs at the boundary of the null set. That must be justified from the model; it is not automatic. The next example verifies the boundary choice for a binomial upper-tail test.

Worked Example: A Composite Null for a Success Rate

A hypothetical device is tested on four independent items, each classified as a success or failure. Let $X$ be the number of successes. Test $H_0:p\le 0.40$ against $H_a:p>0.40$, and suppose three successes are observed. Assume a common success probability $p$ for all four independent trials. Find the conservative p-value.

Method. Large values of $X$ support the upper-tail alternative. For a particular null value $p$, the relevant tail probability is $P_p(X\ge3)$. Since the null includes every $p\le0.40$, the p-value is the largest such probability over that range.

For $X\sim\operatorname{Binomial}(4,p)$,

P_p(X\ge3)=\binom{4}{3}p^3(1-p)+p^4 =4p^3-3p^4.

To check where this expression is largest on $0\le p\le0.40$, differentiate:

\frac{d}{dp}(4p^3-3p^4)=12p^2(1-p)\ge0 \qquad (0\le p\le0.40).

The tail probability is therefore nondecreasing throughout the null parameter range, so its maximum occurs at $p=0.40$. At that boundary,

p=P_{0.40}(X\ge3) =4(0.40)^3(0.60)+(0.40)^4 =4(0.064)(0.60)+0.0256 =0.1536+0.0256 =0.1792.

The conservative p-value is 0.1792. The calculation uses the null boundary because the tail probability increases over the null range, not merely because the boundary is convenient.

Check. The exact-three and four-success probabilities are both included, and their sum is $0.1792$. A common mistake is to evaluate the tail at an arbitrary null value, such as $p=0.20$, and report that result; the composite-null p-value must account for the null value giving the largest tail probability.

Common Interpretation Traps

Using only the exact observed outcome. A p-value includes outcomes at least as extreme as the observed statistic, including ties when the statistic is discrete.
Choosing a tail after seeing the result. The alternative and the statistic’s ordering determine the relevant tail before the data are examined.
Treating the p-value as a probability that $H_0$ is true. It is a probability calculated assuming the null model, not a probability assigned to the hypothesis itself.
Assuming every two-sided p-value is twice a one-sided p-value. That shortcut requires suitable symmetry and a compatible test construction; it is not a general definition.
Ignoring the rest of a composite null. When null parameter values produce different tail probabilities, use the prescribed composite-null method rather than selecting one convenient value.

Key takeaway: A p-value is a null tail probability for outcomes at least as extreme as the observed statistic. The alternative and prespecified test design define “extreme”; they determine which outcomes are counted.

Check Your Understanding

For each problem, identify the relevant null tail and include all outcomes counted by the stated test design.

Under a simple null, $X\sim\operatorname{Binomial}(4,0.25)$. An upper-tail test observes $X=2$. What is the p-value? (Answer: 0.2617)
A discrete statistic has null probabilities $P(T=-2)=0.10$, $P(T=-1)=0.20$, $P(T=0)=0.40$, $P(T=1)=0.20$, and $P(T=2)=0.10$. A two-sided test uses $|T|$ and observes $T=1$. Find the p-value. (Answer: 0.6000)
Under a simple null, $Y\sim\operatorname{Binomial}(5,0.60)$. A lower-tail test observes $Y=2$. Find the p-value $P(Y\le2)$. (Answer: 0.3174)
For $X\sim\operatorname{Binomial}(3,p)$, test $H_0:p\le0.30$ against $H_a:p>0.30$. If $X=2$, the upper-tail probability is $3p^2(1-p)+p^3$. Verify that it is nondecreasing on $0\le p\le0.30$, then find the conservative p-value. (Answer: 0.2160)
A simple-null distribution assigns probabilities 0.12, 0.28, 0.35, and 0.25 to statistic values $-2,-1,0,1$, respectively. A prespecified lower-tail test observes $T=0$. Find its p-value and explain why the value at $T=0$ is included. (Answer: 0.7500; the lower-tail event is $T\le0$)

Test direction	What supports the alternative?	Tail event for observed value \(t_{\text{obs}}\)
Upper-tail	Larger values of \(T\)	\(T\ge t_{\text{obs}}\)
Lower-tail	Smaller values of \(T\)	\(T\le t_{\text{obs}}\)
Two-sided	Departures in either direction	Defined by the test’s prespecified measure of extremeness

What You'll Learn