Pairwise Comparisons After Friedman Tests

Why Follow a Friedman Test With Pairwise Comparisons?

A Friedman test can provide evidence that outcomes differ across conditions in a blocked or repeated-measures study. Its overall result does not say which conditions differ. If a study compares several treatments, times, or methods, pairwise follow-up comparisons can help identify where the differences lie.

For each pair of conditions, the observations remain paired within blocks: compare the two measurements from the same person, site, or other block. A common rank-based follow-up is the Wilcoxon signed-rank test applied separately to each pair. Because several tests may be conducted, their p-values should be adjusted to control the chance of making one or more false rejections across the family of comparisons.

Definition: A pairwise comparison after a Friedman test compares two conditions using their paired observations within each block. A common procedure is a Wilcoxon signed-rank test of the paired differences. When several pairs are tested, a multiplicity adjustment such as Holm’s procedure is used to control the family-wise error rate.

A significant Friedman result is a reason to examine follow-up comparisons, not proof that every pair differs. Nor does it establish which pair differs. Pairwise results must be computed and interpreted on their own, with the multiplicity adjustment made explicit.

Paired Signed-Rank Comparisons

For conditions A and B, define one difference per block, using the same direction throughout: for example, \(d_i=B_i-A_i\). The signed-rank test ranks the absolute values of the nonzero differences, then combines those ranks according to the signs of the differences. A positive difference favors B on the measured scale; a negative difference favors A.

Assumptions and interpretation: The blocks are independent of one another, and each block contributes a paired measurement under both conditions being compared. For a Wilcoxon signed-rank comparison, paired differences must be meaningfully defined, and their absolute magnitudes must be meaningfully rankable. Ordinal labels alone do not guarantee this: arbitrary recodings of ordinal scores can change the differences and their absolute ranks. The test’s usual location interpretation also relies on a distribution of paired differences that is symmetric about its center. Under that model, the null is that the center of the difference distribution is zero. The test is not generally a test of equal means or of arbitrary differences between distributions.

For a pairwise test, zero differences are commonly omitted from ranking. Rank the remaining absolute differences from smallest to largest; tied magnitudes receive their average rank. Let \(W_+\) be the sum of ranks for positive differences and \(W_-\) the sum for negative differences. When there are no ties in absolute magnitude and \(r\) nonzero differences, the ranks sum to \(r(r+1)/2\). With tied magnitudes, the average ranks still sum to that same total.

Under the null model of symmetric differences centered at zero, the signs are equally likely to be positive or negative, conditional on the observed absolute differences. For an exact test, enumerate the possible sign assignments and calculate the proportion whose test statistic is at least as extreme as the observed one. If zeros or tied ranks occur, the enumeration must use the actual nonzero midranks. A normal approximation may instead be used when appropriate, with tie and zero handling specified by the method used.

Worked Example: An Exact Pairwise Test With Seven Blocks

Suppose seven hypothetical blocks each have a measurement under conditions A and B, in seconds. The differences \(B-A\), in block order, are \(-1,\ 2,\ -3,\ -4,\ -5,\ -6,\ -7\) seconds. Assume blocks are independent, the paired differences are meaningful, and the signed-rank model is appropriate for these differences. Test the two-sided null that their symmetric distribution is centered at zero.

The absolute differences are \(1,2,3,4,5,6,7\), so their ranks are \(1,2,3,4,5,6,7\). Only the difference with rank 2 is positive. Therefore,

W_+=2,\qquad W_-=1+3+4+5+6+7=26.

There are \(2^7=128\) equally likely sign assignments under the null. For a two-sided test, assignments at least as extreme in either direction as the observed result have \(W_+\leq 2\) or \(W_+\geq 26\), because all ranks sum to 28. For the lower tail, the only assignments with rank sum at most 2 are the empty set of positive ranks, rank 1 alone, and rank 2 alone. Rank 1 and rank 2 together sum to 3, so that assignment is not in the tail. The upper tail has the same count by symmetry. Thus,

p=\frac{3+3}{128}=0.046875.

This is a raw, unadjusted p-value. At the 0.05 level, it would be below the threshold if this were the only planned test. If it is one of several pairwise comparisons, however, it must be considered with the other p-values and the chosen multiplicity adjustment before drawing a family-wide conclusion.

Adjusting a Family of Pairwise Tests

With \(k\) conditions, there are \(m=k(k-1)/2\) distinct pairs if every pair is tested. Testing each at level \(\alpha\) without adjustment can make the probability of at least one false rejection exceed \(\alpha\). This probability is the family-wise error rate. Holm’s step-down procedure controls it at no more than \(\alpha\), provided each individual p-value is valid under its own null. The tests do not need to be independent for Holm’s control.

Holm’s procedure: Order the \(m\) raw p-values from smallest to largest, \(p_{(1)}\leq\cdots\leq p_{(m)}\). Compare \(p_{(1)}\) with \(\alpha/m\). If it passes, compare \(p_{(2)}\) with \(\alpha/(m-1)\), and continue, comparing \(p_{(i)}\) with \(\alpha/(m-i+1)\). Stop at the first p-value that exceeds its threshold; do not reject that null or any of the remaining nulls. Equivalently, adjusted p-values in sorted order are \(p^{\mathrm{adj}}_{(i)}=\min\{1,\ \max_{j\leq i}[(m-j+1)p_{(j)}]\}\).

The step-down rule is less conservative than comparing every p-value with the single Bonferroni threshold \(\alpha/m\), while retaining family-wise error control. When reporting results, provide the adjustment method and adjusted p-values, or the sequential thresholds and decisions. Do not compare every raw p-value with 0.05 and call that a family-wise analysis.

Worked Example: Holm Adjustment for Three Pairwise Tests

Consider three hypothetical pairwise signed-rank tests, one for each pair among three conditions, with seven nonzero differences per pair and no tied absolute magnitudes. Suppose their positive-rank sums are 0, 2, and 5, respectively. The exact two-sided raw p-values are \(2/128=0.015625\), \(6/128=0.046875\), and \(20/128=0.15625\). The middle result follows from the tail count in the preceding example. For \(W_+=0\), there is one assignment in each tail; for \(W_+=5\), there are ten assignments in each tail, with sums at most 5 or at least 23.

Sort the p-values as shown and set \(\alpha=0.05\). There are \(m=3\) comparisons, so Holm’s successive thresholds are \(0.05/3=0.01667\), \(0.05/2=0.02500\), and \(0.05/1=0.05000\).

0.015625\leq0.01667,\qquad 0.046875>0.02500.

The first null is rejected. The second p-value exceeds its threshold, so the procedure stops: the second and third nulls are not rejected. The Holm-adjusted p-values in sorted order are

\begin{aligned} p^{\mathrm{adj}}_{(1)}&=3(0.015625)=0.046875,\\ p^{\mathrm{adj}}_{(2)}&=\max\{0.046875,\ 2(0.046875)\}=0.093750,\\ p^{\mathrm{adj}}_{(3)}&=\max\{0.093750,\ 1(0.15625)\}=0.15625. \end{aligned}

Only the pair associated with the smallest raw p-value is significant by Holm’s procedure at the 0.05 family-wise level. These results do not establish that the other pairs are equal; they indicate that the evidence for those differences did not meet the adjusted rejection rule.

Zeros, Ties, and a Pairwise Calculation

Some paired data contain zero differences or repeated absolute magnitudes. A zero provides no direction and is ordinarily excluded from the signed-rank ranking. Ties among nonzero absolute differences receive average ranks. The exact sign-enumeration method can account for these midranks directly, although some software uses a large-sample approximation or a specified convention instead. The analysis should state which method produced the p-value.

Worked Example: Zeros and Tied Absolute Differences

In a hypothetical repeated-measures comparison, condition B minus condition A gives differences in seconds of \(2,\ -2,\ 5,\ 0,\ -8,\ 8\) across six independent blocks. The zero is omitted, leaving five nonzero differences. The magnitudes are \(2,2,5,8,8\). The two values of 2 receive average rank \((1+2)/2=1.5\), the value 5 receives rank 3, and the two values of 8 receive average rank \((4+5)/2=4.5\).

The positive differences have ranks \(1.5,3,4.5\), and the negative differences have ranks \(1.5,4.5\). Thus,

W_+=1.5+3+4.5=9,\qquad W_-=1.5+4.5=6.

For an exact two-sided calculation, there are \(2^5=32\) sign assignments. The total rank is 15. The observed \(W_+=9\) is at least as extreme as sums no greater than 6 or no less than 9. To count the lower tail, double the ranks to avoid fractions: the weights are \(3,3,6,9,9\), and a lower-tail sum of 6 corresponds to a doubled sum no greater than 12. There are 14 such subsets: one empty subset; five singletons; seven pairs; and one triple. By symmetry the upper tail also has 14 assignments, so \(p=28/32=0.875\).

This comparison provides little evidence of a systematic difference under the signed-rank model. The calculation also shows why a zero and tied magnitudes cannot simply be treated as if all six differences had distinct, nonzero ranks. In a full set of comparisons, this raw p-value would still be included in the multiplicity adjustment.

Justification for Holm’s Error Control

Holm’s guarantee follows from the validity of the individual p-values and a union-bound argument. Suppose \(t\) of the \(m\) null hypotheses are true, and order all p-values. If the procedure rejects any true null, consider the first true null in this ordering, at position \(j\). At least \(t\) true nulls occur from position \(j\) onward, so \(m-j+1\geq t\). For this true null to be rejected, its p-value must satisfy

p_{(j)}\leq\frac{\alpha}{m-j+1}\leq\frac{\alpha}{t}.

Therefore, the event of any false rejection is contained in the event that at least one of the \(t\) true-null p-values is no greater than \(\alpha/t\). Each is a valid p-value, so its probability of being that small under its null is at most \(\alpha/t\). By the union bound, the probability that any of the \(t\) p-values is that small is at most \(t(\alpha/t)=\alpha\). This establishes family-wise error control without requiring independence among the pairwise tests.

Common Pitfalls and Reporting

Assuming the Friedman test identifies the differing pair. It does not; it is an overall test. Conduct and adjust appropriate pairwise comparisons to identify which contrasts have evidence against their nulls.
Using every raw p-value at 0.05. That ignores the family of tests. Apply a specified adjustment, such as Holm’s procedure, and report adjusted results.
Treating ordinal scores as sufficient for signed ranks. Ordinal labels alone do not make paired differences or their absolute magnitudes meaningful. The signed-rank procedure requires rankable absolute paired differences; a suitable measurement scale is typically interval-like.
Calling a nonsignificant pair equal. Failure to reject a null is not proof of equality. It means the data did not provide sufficient evidence for a difference under the procedure and adjustment used.
Describing a rank-based finding as a mean difference. The signed-rank test concerns signed ranks and, under its symmetry condition, a centered location difference. It does not automatically estimate or test a difference in means.

A clear report identifies the conditions being compared, the direction used to define differences, the number of blocks, the signed-rank method and treatment of ties or zeros, and the multiplicity adjustment. Interpret a significant result as evidence of a systematic paired difference under the model, in the study’s measurement context—not as proof that the condition caused the outcome to change.

Key takeaway: After a Friedman test, compare conditions in pairs using the matched observations within each block. Check that the paired differences and their absolute magnitudes are meaningful, calculate valid signed-rank p-values, and use Holm’s procedure when making a family of comparisons.

Check Your Understanding

Answer the questions using the definitions and procedures in this tutorial.

A Friedman test across four repeated-measures conditions is significant. How many distinct pairwise comparisons are there if every pair is tested?
For a pairwise signed-rank test, what is ranked, and how are tied absolute differences handled?
Why is an ordinal outcome by itself not enough to justify ranking absolute paired differences?
Under Holm’s procedure with four tests, what threshold is used for the smallest ordered p-value when \(\alpha=0.05\)?
In a family of adjusted pairwise tests, one comparison is not rejected. What can and cannot be concluded about that pair?