Start With the Design and the Question
When a distribution is strongly skewed or contains an influential outlier, a procedure based on a normal model may not be a sensible choice—especially with a small sample. But “use a nonparametric test” is not a complete decision. Different tests suit different study designs and answer different questions. A test for paired measurements is not interchangeable with a test for independent groups, and a test about a mean does not automatically answer a question about a median.
A reliable choice begins with two questions: How are the observations related? and What feature of the population is the question about? Then consider the outcome’s scale, distribution shape, and outliers. Nonparametric procedures can reduce reliance on a particular distributional model, but they still have assumptions about sampling, independence, pairing, and often distribution shape.
A Practical Choice Map
First identify the design. One sample, two independent groups, paired measurements, several independent groups, and repeated measurements in blocks are different structures. Treating related observations as if they were independent can give an inappropriate standard error and an inappropriate test.
Next identify the target. A one-sample or paired t procedure concerns a mean or a mean difference. A sign test concerns a median or, for paired data, the median difference under suitable conditions. The signed-rank procedure uses both the directions and magnitudes of nonzero differences and needs a symmetry condition for a straightforward location interpretation. Rank-sum and Kruskal–Wallis procedures compare independent groups using ranks; Friedman is used for related treatments arranged in blocks.
| Design and data situation | Common candidate | Key caution |
|---|---|---|
| One quantitative sample; inference about a mean | One-sample t procedure | Small-sample inference needs an approximately normal population without severe outliers; large-sample robustness is not unlimited. |
| Two paired measurements; inference about a mean difference | Paired t procedure | Assess the distribution of the within-pair differences, not the two sets of measurements separately. |
| One sample or paired differences; median or directional question | Sign test | Uses direction, not the size of a nonzero difference; ties at the hypothesized value need attention. |
| Paired differences with a roughly symmetric distribution | Signed-rank procedure | Its simple location interpretation relies on symmetry; severe asymmetry or outliers can undermine it. |
| Two independent groups | Rank-sum procedure | A median or location-shift interpretation usually needs comparable distribution shapes. |
| Three or more independent groups | Kruskal–Wallis procedure | It can detect distributional differences; it is not automatically a test of medians alone. |
| Several treatments measured within the same blocks | Friedman procedure | Measurements are related within blocks; blocks must be independent of one another. |
What the Main Choices Require
For a one-sample t procedure, the observations should be independent and drawn in a way that supports inference to the population of interest. Exact small-sample inference assumes a normal population. For larger samples, the procedure can be reasonably robust to moderate nonnormality, but strong skewness, unusual outliers, or a small sample can make the mean and its standard error unreliable. A paired t procedure applies the same reasoning to the differences within pairs.
For a sign test, the observations (or the nonzero paired differences) must be independent across individuals. The test uses whether each observation is above or below a specified value, not how far away it is. For a continuous population, testing whether a specified value is the median can be framed through the probability of falling above that value. With ties, the usual simple interpretation needs adjustment; ties should not be silently counted as positive or negative.
A signed-rank procedure also requires independent observations across pairs and uses the magnitudes of nonzero differences as well as their signs. For its common interpretation as a test of a location shift or center, the difference distribution should be approximately symmetric. If differences are very skewed, a sign test may be more defensible when the median is the target, though it gives up information about the sizes of differences.
The rank-sum procedure requires independent samples from the two groups and an outcome that can be meaningfully ordered. It compares the groups’ distributions. If their shapes and spreads are comparable, a difference in location is a useful interpretation; if shapes differ substantially, a result may reflect more than a median difference. Kruskal–Wallis extends this rank-based comparison to three or more independent groups, with the same caution about interpreting it as a test of medians. Friedman is instead for related measurements: observations are compared within each block, and different blocks must be independent. The block structure and treatment assignment must support the intended comparison.
Worked Examples
Worked Example: Skewed Paired Differences
In a hypothetical pilot exercise, seven volunteers record a task time before and after a practice session. Define each difference as before minus after, so a positive value represents a shorter time after practice. The differences, in minutes, are \(1, 1, 2, 2, 3, 4, 20\). The question concerns the typical change, not the mean change.
The design is paired: each volunteer contributes both measurements, so the analysis should use one difference per volunteer. The sample mean difference is
The sorted differences have a median of \(2\) minutes. The value \(20\) is far larger than the other six differences, making the mean especially sensitive to this observation. Since the stated target is a typical change and the differences are strongly right-skewed, a paired sign test is a reasonable candidate if the volunteer differences are independent and the median is the intended target. All seven differences are positive, so the observed directions favor shorter times after practice. This choice does not establish that practice caused the change; the design described here does not by itself rule out other explanations.
A signed-rank procedure would require a more credible symmetry assumption for these differences. The large positive difference and much smaller remaining values make that assumption questionable. A paired t procedure targets the mean difference, which is particularly affected by the value of \(20\).
Worked Example: Symmetric Paired Differences
In a separate hypothetical setting, six people have a measurement recorded under two conditions. Let each difference be condition A minus condition B. The differences are \(-6, -4, -2, 1, 3, 5\) units. The question concerns a typical within-person shift.
The observations are paired, and the six differences are independent across people. Their mean is
The median is the midpoint of \(-2\) and \(1\), also \(-0.500\) units. The differences are arranged in matching pairs around \(-0.5\): \(-6\) and \(5\), \(-4\) and \(3\), and \(-2\) and \(1\). This pattern is compatible with approximate symmetry, with no single value standing far apart from the rest. A signed-rank procedure is therefore a defensible nonparametric candidate for a location question, provided the symmetry assumption is reasonable beyond this small sample.
A paired t procedure would be another candidate if the target were the mean difference and the population of differences were plausibly normal. With only six observations, the sample alone cannot establish that model. The choice depends on the target and the plausibility of the assumptions, not simply on whether one procedure is called parametric.
Worked Example: Two Independent Groups With Similar Shapes
A hypothetical comparison records a quantitative score in two separate groups, with five unrelated participants sampled per group. Group A scores are \(2, 3, 3, 4, 10\); Group B scores are \(4, 5, 5, 6, 12\). Suppose the scientific question concerns a location difference, and the sampling process makes observations independent within and between groups.
The group medians are \(3\) and \(5\): for Group A, the middle of the ordered five scores is \(3\), and for Group B it is \(5\). Each Group B score is two units above the corresponding ordered Group A score, so the observed distributions have a similar pattern and spread, shifted in location. The data are independent, not paired; matching the second ordered scores does not create participant pairs.
A rank-sum procedure is a reasonable candidate for comparing these independent groups if the samples are representative and their population distributions have comparable shapes. In that setting, a location-shift interpretation is plausible. If the shapes differed substantially, the procedure would instead be understood as comparing distributions, not as a test of medians alone. A two-sample t procedure could target the difference in means, but its assumptions and sensitivity to the high score of \(10\) and \(12\) should be considered.
Worked Example: Repeated Treatments in Blocks
In a hypothetical laboratory exercise, each of four independent specimens is measured under three treatments. The recorded responses are:
| Specimen (block) | Treatment A | Treatment B | Treatment C |
|---|---|---|---|
| 1 | 8 | 10 | 13 |
| 2 | 6 | 7 | 9 |
| 3 | 11 | 12 | 14 |
| 4 | 5 | 8 | 8 |
Each specimen receives all three treatments, so measurements within a row are related. The independent units are the four specimens, not the twelve measurements. Because there are more than two treatments measured within each block, Friedman is a candidate for comparing the treatments, assuming the blocks are independent and the outcome can be ordered. Its conclusion concerns treatment differences across blocks; it does not turn the repeated measurements into independent groups. With only four blocks, conclusions would also be limited by the small amount of information.
Why Pairing Changes the Analysis
Pairing is not a cosmetic feature of a data table. If \(X\) and \(Y\) are measurements on the same individual, their difference has variance
This identity follows by expanding the variance of a difference: the two variance terms contribute positively, while the covariance appears twice with a negative sign. When measurements on the same person tend to move together, the covariance is positive, so the variance of the within-person differences can be smaller than the variability expected from treating the two sets as unrelated. A paired t procedure uses the differences directly; paired nonparametric procedures likewise preserve which observations belong together. Ignoring pairing discards part of the design and can change the uncertainty calculation.
Common Mistakes
- Choosing a test from normality alone. A skewed distribution does not automatically imply one particular nonparametric test. First identify whether observations are paired, independent, or blocked, and then identify the target.
- Calling every rank-based result a median comparison. Rank-sum and Kruskal–Wallis compare distributions. A median or location interpretation needs additional comparability of distribution shapes.
- Assuming nonparametric means assumption-free. Independence, valid sampling, the correct pairing structure, and procedure-specific conditions still matter.
- Using signed-rank for any paired data. Its usual location interpretation depends on approximate symmetry of the differences. A sign test may be more appropriate when the differences are highly asymmetric and the median is the target.
- Treating paired measurements as independent groups. Use the within-pair structure; do not count repeated observations from the same person as independent units.
Check Your Understanding
For each situation, name a plausible procedure and state one assumption or interpretation caution.
- A small sample of paired before-and-after differences is strongly skewed, and the question concerns the median difference. Which procedure might be considered, and why?
- Two independent groups have similarly shaped score distributions, and the question concerns a location shift. Which rank-based procedure is a candidate?
- Three independent groups are compared on an ordered outcome. Which procedure may be considered, and why should its result not automatically be described as a median test?
- Each of several participants is measured under four conditions. What feature of the design makes Friedman relevant?
- Explain why a paired t procedure should assess the differences rather than the separate distributions of the two measurements.