Constructing Two-Sided P-Values

Why Two-Sided P-Values Use Both Directions

A two-sided alternative says that a parameter differs from its null value, without specifying in advance whether it is larger or smaller. Results in either direction can therefore count against the null model. The p-value must add the null probability of the observed result and outcomes judged at least as extreme on the opposite side, according to the test’s prespecified measure of extremeness.

For a symmetric reference distribution and a statistic measured as a signed distance from its null value, outcomes equally far above and below that value have matching probabilities. In this familiar case, the two-sided p-value is twice the tail probability in the direction of the observed statistic. Symmetry is essential to that shortcut. For an asymmetric reference distribution, the two tails need not have equal probabilities, so the extremeness rule must be applied directly.

Definition: A two-sided p-value is the probability, calculated under the null model, of obtaining the observed test statistic or another outcome that the test’s prespecified rule counts as at least as extreme, in either direction. For a continuous statistic \(T\) and the rule that outcomes are extreme when \(|T|\geq |t_{\mathrm{obs}}|\), this is \(P(|T|\geq |t_{\mathrm{obs}}|\mid H_0)\). The rule for extremeness is part of the test; it is not determined by the phrase “two-sided” alone.

The p-value is calculated from the null reference distribution, not from a distribution chosen after seeing which calculation gives a smaller number. Earlier material on null models, test statistics, and defining p-values provides the general framework: specify the null distribution, state what counts as extreme, then add the null probabilities of those outcomes. In the examples below, the extremeness rule is stated explicitly before the calculation.

The Symmetric Case: Doubling One Tail

Suppose \(T\) has a continuous null distribution symmetric about zero, and the test defines extremeness by distance from zero. If the observed statistic is \(t_{\mathrm{obs}}>0\), outcomes at least as extreme are \(T\geq t_{\mathrm{obs}}\) or \(T\leq -t_{\mathrm{obs}}\). Symmetry makes these two tail probabilities equal. If \(t_{\mathrm{obs}}<0\), the same reasoning applies with the tails reversed.

\begin{aligned} p &=P(T\geq |t_{\mathrm{obs}}|\mid H_0) +P(T\leq -|t_{\mathrm{obs}}|\mid H_0)\\ &=2P(T\geq |t_{\mathrm{obs}}|\mid H_0). \end{aligned}

The equality follows because symmetry about zero gives equal probabilities to the two equally distant tails. Continuity means the boundary points have probability zero, so it does not matter here whether the inequalities include equality. This calculation is appropriate for a symmetric reference distribution and the stated distance-based extremeness rule. It is not a universal definition of a two-sided p-value.

Worked Example: A Two-Sided Standard Normal Test

Suppose a hypothetical test has \(H_0\) reference distribution \(Z\sim N(0,1)\), and its two-sided rule counts values with \(|Z|\geq |z_{\mathrm{obs}}|\) as at least as extreme. The observed statistic is \(z_{\mathrm{obs}}=1.80\). Find the p-value.

The standard normal distribution is continuous and symmetric about zero, so doubling the upper tail at 1.80 is justified:

\begin{aligned} p &=P(|Z|\geq 1.80)\\ &=2P(Z\geq 1.80)\\ &=2(0.0359303)\\ &=0.0718606\approx 0.0719. \end{aligned}

Thus the two-sided p-value is approximately 0.0719. The calculation includes results at least 1.80 standard normal units from zero in either direction. The tail probability was kept unrounded until after doubling, so the reported four-decimal answer is not changed by premature rounding.

The same symmetry argument applies to a Student \(t\) reference distribution. When a one-sample mean test uses an estimated standard deviation, its null statistic follows a \(t\) distribution under the model assumptions. The degrees of freedom determine which symmetric \(t\) curve supplies the tail probabilities; they do not change the logic of combining the two tails.

Worked Example: A Two-Sided One-Sample t Test

In a hypothetical sample of \(n=16\) measurements, suppose \(\bar{x}=52.4\) units and \(s=8.0\) units. Test \(H_0:\mu=50\) units against \(H_a:\mu\neq 50\) units. Assume the observations are independent and come from a normal population. These conditions justify using the one-sample \(t\) reference distribution with \(df=n-1\).

The standard error is \(s/\sqrt{n}=8.0/\sqrt{16}=2.0\) units. The test statistic is:

t_{\mathrm{obs}} =\frac{\bar{x}-\mu_0}{s/\sqrt{n}} =\frac{52.4-50}{8.0/\sqrt{16}} =\frac{2.4}{2.0} =1.20, \qquad df=16-1=15.

The \(t\) distribution with 15 degrees of freedom is symmetric about zero. Under the distance-from-zero rule, the p-value is twice its upper-tail probability at 1.20:

p=2P(T_{15}\geq 1.20)\approx 2(0.1244)=0.2488\approx 0.249.

The p-value is approximately 0.249. It accounts for test statistics at least 1.20 units from zero in either direction. The statistic’s positive sign shows that the sample mean is above the null value; the two-sided p-value calculation also counts equally distant negative statistics.

When the Reference Distribution Is Asymmetric

If a null distribution is asymmetric, equal distances from the null value do not generally have equal probabilities. A two-sided test still considers departures in both directions, but it cannot assume that the two relevant tails match. The test needs a prespecified rule for deciding which outcomes count as at least as extreme. One possible rule is distance from the null value: count outcomes whose absolute deviations are at least as large as the observed absolute deviation. Other tests can use a different rule, appropriate to their statistic and null model.

For a continuous statistic \(T\), the distance-based rule gives:

p=P(|T|\geq |t_{\mathrm{obs}}|\mid H_0) =P(T\leq -|t_{\mathrm{obs}}|\mid H_0) +P(T\geq |t_{\mathrm{obs}}|\mid H_0).

This equation adds both tails, but it does not say they are equal. In an asymmetric distribution, one tail may be much larger than the other, and sometimes one side of the distribution may have no probability beyond the matching cutoff. The probabilities must be calculated from the actual null distribution.

Worked Example: Distance-Based Extremeness in an Asymmetric Model

Consider a hypothetical test statistic whose null model is \(T=G-2\), where \(G\) has a gamma distribution with shape 2 and scale 1. Thus \(G\) has density \(f_G(g)=g e^{-g}\) for \(g\geq0\), and \(T\) has mean zero but an asymmetric distribution supported on values \(T\geq-2\). Suppose the test’s prespecified two-sided rule counts outcomes with \(|T|\geq|t_{\mathrm{obs}}|\) as extreme, and the observed value is \(t_{\mathrm{obs}}=-1.20\).

The two-sided event is \(T\leq-1.20\) or \(T\geq1.20\). Since \(T=G-2\), these conditions correspond to \(G\leq0.80\) and \(G\geq3.20\), respectively. For this gamma distribution, the cumulative probability is \(P(G\leq g)=1-e^{-g}(1+g)\), and the upper-tail probability is \(P(G\geq g)=e^{-g}(1+g)\) for \(g\geq0\). Therefore:

\begin{aligned} p &=P(G\leq0.80)+P(G\geq3.20)\\ &=\left[1-e^{-0.80}(1+0.80)\right] +e^{-3.20}(1+3.20)\\ &\approx 0.1912+0.1712\\ &\approx 0.3624. \end{aligned}

The two-sided p-value under this specified rule is approximately 0.3624. The two tail probabilities differ, even though the cutoffs are the same distance from zero. Doubling the smaller tail would give about 0.3424, not the p-value defined by this test’s distance-based rule. Doubling the observed-side tail would also be unjustified because the null distribution is not symmetric.

Why the Extremeness Rule Matters

“Two-sided” specifies that departures in both directions matter; it does not, by itself, specify a unique ranking of all possible outcomes. For a symmetric statistic centered at its null value, absolute distance is a natural rule and leads to the familiar doubled-tail calculation. For an asymmetric statistic, the test must state how outcomes in different parts of the reference distribution are compared. A distance-based rule is one clear option when signed distance has a meaningful interpretation, but it is not automatically appropriate for every statistic.

This is why a p-value cannot always be reconstructed from the observed statistic and the words “two-sided” alone. One must know the null reference distribution and the extremeness rule. In particular, using twice the smaller one-sided tail is a convention used in some settings, not a general mathematical identity for every asymmetric distribution. The exact test definition determines which outcomes are counted.

Key takeaway: Add the null probabilities of all outcomes counted as at least as extreme under the test’s stated two-sided rule. Double one tail only when symmetry and the chosen extremeness rule justify treating the two tails as equal.

Common Mistakes

Doubling a tail without checking symmetry. The reasoning “two-sided means multiply by two” is incomplete. Doubling follows for symmetric reference distributions and matching tail cutoffs; an asymmetric null model requires calculating the probabilities according to the specified rule.
Using only the observed direction. For a two-sided test, counting only outcomes beyond the observed statistic on its own side ignores outcomes that are comparably extreme in the opposite direction.
Choosing the rule after seeing the data. Selecting whichever tail combination produces a preferred answer changes the test. State the extremeness rule as part of the test, then apply it to the observed result.
Rounding before combining tails. In the normal example, doubling 0.0359 gives 0.0718, while using the unrounded tail gives 0.0718606, which rounds to 0.0719. Keep sufficient precision through the calculation.

Specify the null model.
Identify the reference distribution for the test statistic under \(H_0\), including any relevant degrees of freedom or model assumptions.

State what counts as extreme.
For a distance-based test, this may be outcomes with \(|T|\geq|t_{\mathrm{obs}}|\). Use the test’s actual prespecified rule.

Calculate the null probability.
Add the probabilities of all outcomes counted as at least as extreme. Double one tail only when symmetry makes that shortcut valid.

Check Your Understanding

Use the stated reference distribution and extremeness rule in each question.

For a continuous distribution symmetric about zero, why are the two tail probabilities equal when the cutoffs are the same distance from zero?
A standard normal test has \(z_{\mathrm{obs}}=-2.00\) and uses absolute distance from zero. Write the p-value as a probability involving \(Z\), without looking up its numerical value.
In a two-sided one-sample \(t\) test, what information determines the reference distribution used to find the p-value?
For an asymmetric null distribution, explain why \(2\) times the smaller one-sided tail is not necessarily the p-value under a distance-based rule.
In the gamma example, which two ranges of \(G\) correspond to outcomes at least 1.20 units from zero on the \(T\) scale?