Exact and Approximate Signed-Rank P-Values

From Signed-Rank Statistics to P-Values

The signed-rank statistic summarizes how the signs of paired differences line up with their ranks. To turn that summary into a p-value, we need to know what values the statistic could take if the null hypothesis were true. For a small sample, we can find that null distribution by enumerating sign assignments. For a larger sample, a normal approximation is often more practical.

The calculations in this tutorial use the signed-rank statistic \(W_+\), the sum of the ranks attached to positive differences. A small or large \(W_+\) can be evidence against a null center, depending on the alternative. For a two-sided test, values far from the center of the null distribution count as extreme.

Definition: Let \(d_i\) be the difference for paired unit \(i\), measured relative to the hypothesized center. Remove any zero differences, rank the remaining absolute differences from smallest to largest, and assign tied absolute differences their average ranks. Write \(r_i\) for the rank attached to \(d_i\). Then \(W_+=\sum_{d_i>0}r_i\), and \(W_-=\sum_{d_i<0}r_i\). If the ranks sum to \(R\), then \(W_++W_-=R\).

The null model matters. For exact signed-rank inference, assume the paired differences are independent and symmetric about the hypothesized center. Conditional on the observed nonzero absolute differences, each sign is then positive or negative with probability \(1/2\), independently of the other signs. Thus all \(2^m\) sign assignments have equal probability, where \(m\) is the number of nonzero differences. Tied absolute differences do not invalidate this enumeration: each observation still has its own sign, even when some observations share a rank.

A one-sided exact p-value is the proportion of sign assignments producing a statistic at least as extreme in the direction specified by the alternative. For a two-sided test, a convenient statistic is \(T=\min(W_+,W_-)\). The exact p-value is the probability, under the null distribution, of obtaining \(T\) no larger than the observed \(T\). Equivalently, it is twice the smaller inclusive tail probability for \(W_+\), capped at 1. Inclusive means that assignments giving exactly the observed statistic are counted.

Exact P-Values by Enumerating Signs

For a small number of nonzero differences, listing the possible positive-sign subsets is a direct way to obtain an exact p-value. Each subset contributes the sum of its ranks to \(W_+\), and each subset has probability \(1/2^m\). If tied observations have equal average ranks, count each sign assignment separately; do not treat distinct statistic values as if they were equally likely.

Worked Example: An Exact Two-Sided P-Value

Suppose four hypothetical paired differences, in minutes, are \(+1,-2,+3,+4\). Assume the paired differences are independent and symmetric about the null center of zero. There are no zero differences or tied absolute differences, so their ranks are 1, 2, 3, and 4. The observed positive differences have ranks 1, 3, and 4, giving \(W_+=1+3+4=8\). The total rank sum is \(R=1+2+3+4=10\), so \(W_-=10-8=2\) and \(T=2\).

There are \(2^4=16\) equally likely sign assignments. To find the two-sided exact p-value, count assignments with \(T\leq2\). Equivalently, count assignments with \(W_+\leq2\) or \(W_+\geq8\). The rank subsets giving \(W_+\leq2\) are the empty subset, \(\{1\}\), and \(\{2\}\). There are three such assignments. By complementing each subset, there are also three assignments with \(W_+\geq8\).

Therefore,

p_{\text{exact}}=\frac{3+3}{16}=\frac{6}{16}=0.375.

The exact two-sided p-value is 0.375. Under the stated null model, 37.5% of sign assignments produce a statistic at least this far from the center. This is not strong evidence against the null center at common significance levels.

This enumeration also explains why exact p-values are discrete: they are multiples of \(1/2^m\) when all sign assignments are equally likely. For small \(m\), the possible p-values can be quite coarse. Ties can make several assignments produce the same \(W_+\), but the denominator remains the number of sign assignments, not the number of different values of \(W_+\).

Normal Approximation and Continuity Correction

When the number of nonzero differences is larger, the null distribution of \(W_+\) can often be approximated by a normal distribution. The approximation uses the mean and variance of \(W_+\) under the independent, symmetric-sign null model. The ranks are treated as fixed after conditioning on the absolute differences.

Theorem: Under the null sign model, with fixed ranks \(r_1,\ldots,r_m\), \(W_+\) has mean \(\mu_{W_+}=\frac{1}{2}\sum_{i=1}^m r_i\) and variance \(\sigma^2_{W_+}=\frac{1}{4}\sum_{i=1}^m r_i^2\). The normal approximation uses \(Z=(W_+-\mu_{W_+})/SE\), where \(SE=\sqrt{\sigma^2_{W_+}}\). A continuity correction adjusts the observed statistic toward the mean before calculating \(Z\).

With no tied absolute differences, the ranks are \(1,2,\ldots,m\), so \(\mu_{W_+}=m(m+1)/4\) and \(\sigma^2_{W_+}=m(m+1)(2m+1)/24\). With ties, average ranks change the sum of squared ranks, so the variance must be calculated using those actual ranks.

The continuity correction accounts for the fact that \(W_+\) takes discrete values while the normal distribution is continuous. For untied ranks, the statistic changes in integer steps, so the usual correction is \(0.5\). For a lower-tail observation, add the correction to \(W_+\) to move it toward the mean; for an upper-tail observation, subtract it. With average ranks, the attainable values may have a different lattice spacing. A suitable correction is half that spacing; for example, if all attainable values lie on a half-unit lattice, the correction is \(0.25\). When a two-sided statistic is equally far into either tail, apply the correction toward the mean in the relevant tail and double the corresponding one-sided normal tail probability.

Worked Example: A Normal Approximation With Untied Ranks

Consider 30 hypothetical paired differences in minutes. Suppose the absolute differences are all distinct, and the positive differences have ranks 1 through 15 while the negative differences have ranks 16 through 30. Assume independent paired differences and a symmetric null model centered at zero. There are no zeros or ties, and the observed statistic is

W_+=1+2+\cdots+15=\frac{15(16)}{2}=120.

The ranks sum to \(R=30(31)/2=465\), so the null mean is \(465/2=232.5\). The null variance and standard error are

\sigma^2_{W_+}=\frac{30(31)(61)}{24}=2363.75, \qquad SE=\sqrt{2363.75}\approx48.62.

The observation is below the mean, so the continuity-corrected value is \(120+0.5=120.5\). The standardized statistic is

Z=\frac{120.5-232.5}{48.62}\approx-2.304.

For a two-sided test, the normal-approximation p-value is \(2P(Z_{\text{standard}}\leq-2.304)\), approximately 0.0212. Under the null model, this approximation suggests that a result at least as far from the center is uncommon. It is an approximate p-value, not the exact sign-enumeration probability.

How Ties Change the Normal Approximation

A tie in absolute differences gives multiple observations the same average rank. The mean of \(W_+\) remains half the sum of the ranks, but the variance is based on the sum of their squares. The usual tie-correction formula expresses the same adjustment relative to the variance for untied ranks. If a block of \(t_j\) observations is tied, the variance is reduced by \((t_j^3-t_j)/48\) for that block.

\sigma^2_{W_+} = \frac{m(m+1)(2m+1)}{24} - \sum_j\frac{t_j^3-t_j}{48}.

This adjustment follows directly from the average ranks. If \(I_i\) equals 1 for a positive sign and 0 for a negative sign, then \(W_+=\sum_i r_iI_i\). Each \(I_i\) has mean \(1/2\) and variance \(1/4\), and the indicators are independent under the null sign model. Therefore, \(E(W_+)=\sum_i r_i/2\), and \(\operatorname{Var}(W_+)=\sum_i r_i^2/4\). Replacing the untied ranks within each tied block by their average reduces the sum of squared ranks; the reduction for a block of size \(t_j\), after dividing by 4, is \((t_j^3-t_j)/48\). This gives the stated tie correction.

Worked Example: A Tie-Corrected Approximation

Suppose there are 20 hypothetical nonzero paired differences in minutes, with independent signs under a symmetric null model centered at zero. Two observations have absolute difference 1, and the other absolute differences are 2 through 19. The two tied observations receive average rank 1.5 each; the remaining ranks are 3 through 20. Suppose one rank-1.5 observation and the observations with ranks 3, 5, 7, 9, 11, 13, 15, 17, and 19 are positive. Then

$$ W_+=1.5+3+5+7+9+11+13+15+17+19=100.5. $$

The ranks sum to 210, giving a null mean of 105. The untied variance for 20 ranks would be \(20(21)(41)/24=717.5\). The single tie block has size 2, so the variance reduction is \((2^3-2)/48=0.125\). Thus

\sigma^2_{W_+}=717.5-0.125=717.375, \qquad SE=\sqrt{717.375}\approx26.784.

Because the average ranks are multiples of 0.5, the attainable statistics lie on a half-unit lattice, for which the half-spacing continuity correction is 0.25. Since \(W_+=100.5\) is below the mean, correct toward the mean by adding 0.25:

Z=\frac{100.5+0.25-105}{26.784}\approx-0.159.

The two-sided normal-approximation p-value is approximately \(2P(Z_{\text{standard}}\leq-0.159)=0.874\). The tie correction is part of the standard error; it does not change the observed \(W_+\). This calculation illustrates the approximation and its corrections, but with only 20 nonzero differences the normal result should still be understood as approximate.

Choosing and Reporting the Calculation

For a small sample, exact enumeration avoids relying on the normal approximation. It is especially useful when only a limited number of sign assignments exist and the hypotheses and symmetry assumptions support the exact null distribution. For larger samples, the normal approximation is more convenient, provided the distribution is not being treated as exactly normal. Use average ranks in the variance when absolute differences are tied, and apply a continuity correction when using a discrete statistic with a continuous approximation.

A p-value describes how extreme the observed statistic would be under the specified null model. It is not the probability that the null hypothesis is true, and a small p-value by itself does not establish that the paired measurements differ in a practically important way. The signed-rank procedure also relies on a symmetric distribution of paired differences for its usual center-based interpretation; the exact sign calculation does not remove that modeling assumption.

Common Mistakes

Counting different statistic values instead of sign assignments. Assignments are equally likely under the exact null model, not distinct values of \(W_+\). Count every assignment, including multiple assignments that produce the same statistic.
Assuming symmetry alone makes the signs independent. Symmetry gives fair signs conditional on magnitudes, but independence between paired units is also needed for the stated enumeration.
Using untied ranks in the variance when ties are present. Use the average ranks actually assigned, or subtract the tie adjustment from the untied variance.
Applying a continuity correction in the wrong direction. Move the observed statistic toward the null mean: add the correction for a lower-tail observation and subtract it for an upper-tail observation.
Calling an approximate p-value exact. A normal tail probability with continuity and tie corrections remains an approximation to the signed-rank distribution.

Key takeaway: For exact signed-rank p-values, enumerate the \(2^m\) equally likely sign assignments under independent, symmetric differences. For a normal approximation, use the mean and variance based on the actual ranks, adjust for ties, and move the statistic toward its mean with an appropriate continuity correction.

Check Your Understanding

Try these questions without looking back.

Why are there \(2^m\) equally likely sign assignments under the exact null model, and what assumptions justify that statement?
A two-sided exact test has 32 sign assignments, of which 10 produce a statistic at least as extreme as observed. What is the exact p-value?
For an observation below the null mean, which direction should a continuity correction move the statistic, and why?
Why must tied average ranks be used when calculating the variance of the normal approximation?
Explain why counting distinct \(W_+\) values rather than sign assignments can give an incorrect exact p-value.