Critical-Value and P-Value Decisions

Two Ways to State a Test Decision

A hypothesis test can be reported by comparing a test statistic with a critical value, or by comparing its p-value with the significance level. These are not two different tests when they use the same statistic, null reference distribution, direction of extremeness, and boundary convention. They are two ways to describe the same rejection rule.

The qualification matters. In a continuous, strictly increasing part of a null distribution, a critical value can be chosen so that its tail probability is exactly the significance level. For discrete distributions, that equality may be impossible. Even with a continuous distribution, gaps in its support can make a quantile convention important. To ensure agreement, the critical region must be defined to match the p-value rule, rather than selected using an incompatible cutoff convention.

Definition: Suppose \(T\) is a test statistic, \(H_0\) is the null hypothesis, and \(\alpha\) is the significance level. A p-value is the null probability of observing a statistic at least as extreme as the observed value, according to the test’s specified direction or ordering. A critical-value rule rejects \(H_0\) when the statistic falls in a specified critical region. The two rules agree when that critical region is precisely the set of statistic values whose p-values are at most \(\alpha\).

For an upper-tail test, larger values of \(T\) count as more extreme. Write \(P_0\) for probability under the null model. The p-value at an observed value \(t\) is \(P_0(T\geq t)\), and the p-value rule rejects when this probability is at most \(\alpha\). If the null distribution is continuous and strictly increasing in the relevant region, a critical value \(c_\alpha\) can be chosen with \(P_0(T\geq c_\alpha)=\alpha\). In that setting, the two rules reduce to \(T\geq c_\alpha\) and \(P_0(T\geq T_{\mathrm{obs}})\leq\alpha\), respectively.

Why the Rules Agree

The key is that an upper-tail probability decreases as the observed statistic increases. Thus, if \(t\) is at least as large as the critical value, its upper-tail probability is no larger than the upper-tail probability at the critical value. Conversely, if its tail probability is no larger than the critical value’s tail probability, strict increase of the null CDF in the relevant region ensures \(t\) is at least as large as that critical value.

t\geq c_\alpha \quad\Longleftrightarrow\quad P_0(T\geq t)\leq P_0(T\geq c_\alpha)=\alpha.

The equivalence in this display uses the stated strict-increase condition to make the reverse implication valid for the cutoff \(c_\alpha\). Mere continuity is not enough: a continuous distribution can have gaps in its support, leaving its CDF flat over an interval. In such an interval, several statistic values can have the same tail probability, so a cutoff chosen at one edge need not describe every value satisfying the p-value rule.

A reliable general convention is to define the rejection region directly as \(\{t:p(t)\leq\alpha\}\), where \(p(t)\) is the p-value function for the test. If the statistic is ordered so that more extreme values are in one tail, this set is an upper or lower region in that ordering. A critical-value description must use the boundary of that same set and include or exclude the boundary consistently.

Specify the test.
Identify the null reference distribution, which direction or ordering counts as extreme, and the significance level.

Find the p-value rule.
For an observed statistic, calculate the null probability of an outcome at least as extreme under the test’s definition.

Match the critical region.
Reject by critical value exactly for the statistic values whose p-values are at most \(\alpha\), respecting any ties or unattainable tail probabilities.

Worked Examples

Worked Example: Upper-Tail Test With a Standard Normal Statistic

A hypothetical process audit uses a statistic \(Z\) whose null distribution is \(N(0,1)\). The alternative is that the statistic tends to be unusually large. At significance level \(\alpha=0.05\), the observed value is \(z_{\mathrm{obs}}=1.80\). Decide using both methods.

Method and assumptions. Under \(H_0\), \(Z\sim N(0,1)\), and the test is upper-tailed. The standard normal distribution is continuous and strictly increasing, so the critical value with upper-tail probability \(0.05\) is approximately \(1.6449\).

Critical-value comparison. Since \(1.80>1.6449\), the observed statistic lies in the critical region. Reject \(H_0\).

P-value comparison. The upper-tail p-value is

p=P_0(Z\geq1.80)=1-\Phi(1.80)=1-0.9641=0.0359.

Since \(0.0359<0.05\), reject \(H_0\) by the p-value rule as well. Both calculations use the same upper tail and produce the same decision. The common mistake this avoids is comparing an upper-tail p-value with a lower-tail critical value.

Worked Example: Lower-Tail Test for a Mean

Suppose a hypothetical sample of \(n=25\) independent measurements comes from a normal population with known standard deviation \(\sigma=10\). Test \(H_0:\mu=100\) against \(H_1:\mu<100\) at \(\alpha=0.05\). The observed sample mean is \(\bar{x}=96.5\). Use both decision methods.

Method and assumptions. Under \(H_0\), the standardized sample mean \(Z=(\bar{X}-100)/(\sigma/\sqrt{n})\) has distribution \(N(0,1)\). Independence, normal sampling, and known \(\sigma\) justify this reference distribution. This is a lower-tail test.

Calculate the statistic. The standard error under the null is \(10/\sqrt{25}=10/5=2\). Therefore,

z_{\mathrm{obs}}=\frac{96.5-100}{2}=\frac{-3.5}{2}=-1.75.

Critical-value comparison. The lower-tail critical value at \(\alpha=0.05\) is approximately \(-1.6449\). Since \(-1.75<-1.6449\), reject \(H_0\).

P-value comparison. The lower-tail p-value is

p=P_0(Z\leq-1.75)=\Phi(-1.75)=0.0401.

Since \(0.0401<0.05\), reject \(H_0\). The decisions match. A frequent error is to use the upper-tail probability \(1-\Phi(-1.75)\), which is about \(0.9599\); that calculation asks how likely a statistic at least as large is, not how likely one at least as small is.

Worked Example: Exact Upper-Tail Test With a Discrete Statistic

A hypothetical review records the number \(X\) of flagged items among eight independently inspected items. Under \(H_0\), \(X\sim\operatorname{Binomial}(8,0.25)\); the alternative is that the flag probability is greater than \(0.25\). If \(X=4\), decide at \(\alpha=0.10\), using the exact null distribution.

Method and assumptions. The inspections are independent, each has the same null flag probability \(0.25\), and \(X\) counts flags. For this upper-tail test, the p-value at \(x\) is \(P_0(X\geq x)\). Because the binomial distribution is discrete, its tail probabilities need not equal \(0.10\) at any attainable count.

Find the critical region. At \(X=4\), the exact upper-tail probability is

\begin{aligned} P_0(X\geq4) &=\sum_{k=4}^{8}\binom{8}{k}(0.25)^k(0.75)^{8-k}\\ &\approx 0.1138153. \end{aligned}

This exceeds \(0.10\), so \(X=4\) is not in the rejection region. At \(X=5\),

\begin{aligned} P_0(X\geq5) &=\sum_{k=5}^{8}\binom{8}{k}(0.25)^k(0.75)^{8-k}\\ &\approx 0.0272980. \end{aligned}

Since the upper-tail probability decreases as \(X\) increases, the critical region is \(X\geq5\). The observed count is 4, so the critical-value method does not reject.

Check by the p-value. The p-value is \(0.1138153\), which is greater than \(0.10\). Do not reject \(H_0\). Thus both methods agree. The test’s actual null rejection probability is \(P_0(X\geq5)\approx0.0273\), not \(0.10\); discreteness makes the attainable size fall below the chosen significance level. The common mistake is to use an approximate tail probability or to assume the critical boundary must have tail probability exactly \(\alpha\).

Other Tail Rules and Important Conditions

For a lower-tail test, the p-value is \(P_0(T\leq t_{\mathrm{obs}})\), and smaller statistic values are more extreme. With a continuous, strictly increasing null CDF near the cutoff, the critical-value rule \(T\leq c_\alpha\) is equivalent to a lower-tail p-value at most \(\alpha\). The direction changes, but the logic is the same: moving farther into the lower tail cannot increase the lower-tail probability.

For a two-sided test, the statistic’s extremeness ordering must account for departures in both directions. For example, if \(Z\sim N(0,1)\) under \(H_0\) and extremeness is measured by \(|Z|\), the p-value is \(P_0(|Z|\geq|z_{\mathrm{obs}}|)\). The matching critical-value rule rejects for \(|Z|\geq c\), where \(P_0(|Z|\geq c)=\alpha\). The equivalence follows because this two-sided tail probability decreases as \(|z_{\mathrm{obs}}|\) increases. Using a one-sided tail probability with a two-sided critical value would not be the same test.

The argument requires more than putting two numbers beside \(\alpha\): both calculations must refer to the same null model, statistic, tail definition, and boundary convention. With composite null hypotheses, the p-value must also be calibrated for the composite null—for example, by taking the largest relevant tail probability over null parameter values when that is the prescribed construction. Once a valid p-value function is specified, its rejection set still consists of outcomes with \(p\leq\alpha\).

Common Mistakes and Exam Checks

Mixing tails. For an upper-tail alternative use \(P_0(T\geq t_{\mathrm{obs}})\); for a lower-tail alternative use \(P_0(T\leq t_{\mathrm{obs}})\). The critical region must point in the same direction.
Forgetting the two-sided definition. A two-sided p-value measures extremeness in both directions under the specified test, not just the tail containing the observation.
Assuming every critical boundary has size exactly \(\alpha\). A discrete statistic can skip over \(\alpha\). Use the attainable rejection region and calculate its null probability.
Ignoring gaps or ties. Continuity alone does not guarantee that an arbitrarily chosen quantile cutoff matches every value whose p-value is at most \(\alpha\). State the cutoff convention or define the critical region directly from the p-value rule.
Changing the test between calculations. The equivalence does not excuse using different null distributions, tails, or definitions of extremeness in the two methods.

Key takeaway: A p-value decision and a critical-value decision agree when they describe the same rejection region. For a continuous, strictly increasing null reference distribution, the usual tail cutoff gives this equivalence directly. For discrete statistics or flat portions of a distribution, define the boundary carefully and make the critical region match \(p\leq\alpha\).

Check Your Understanding

For each problem, identify the relevant tail, compare the statistic with the matching critical region, and check the p-value decision.

Under \(H_0\), \(Z\sim N(0,1)\). An upper-tail test uses \(\alpha=0.025\) and observes \(z=1.90\). Decide by both methods. (Answer: Critical value \(\approx1.960\), p-value \(\approx0.0287\); do not reject.)
Under \(H_0\), \(Z\sim N(0,1)\). A lower-tail test uses \(\alpha=0.025\) and observes \(z=-2.05\). Decide by both methods. (Answer: Critical value \(\approx-1.960\), p-value \(\approx0.0202\); reject.)
Under \(H_0\), \(X\sim\operatorname{Binomial}(6,0.30)\). For an upper-tail test with \(\alpha=0.10\), the observed count is \(X=4\). Find the exact p-value and the critical region. (Answer: \(P_0(X\geq4)=0.059535+0.010206+0.000729=0.070470\); critical region \(X\geq4\); reject.)
Under \(H_0\), \(X\sim\operatorname{Binomial}(8,0.25)\). For an upper-tail test with \(\alpha=0.10\), the observed count is \(X=4\). Decide by both methods. (Answer: \(P_0(X\geq4)\approx0.1138\), while the critical region is \(X\geq5\); do not reject.)
Under \(H_0\), \(Z\sim N(0,1)\). A two-sided test uses \(\alpha=0.05\) and observes \(z=2.30\). Decide using \(|Z|\) and its two-sided p-value. (Answer: Critical rule \(|Z|\geq1.960\), p-value \(2[1-\Phi(2.30)]\approx0.0214\); reject.)

What You'll Learn