Two Tests, Two Questions
For two independent groups, the Mann–Whitney rank-sum test and Welch’s two-sample t test can both compare outcomes across groups, but they do not generally answer the same question. Welch’s test concerns the difference in population means. The Mann–Whitney test concerns whether the groups’ outcome distributions are the same; its rank comparisons can also be described through how often a randomly chosen observation from one group exceeds one from the other.
This distinction matters when outcomes are skewed, contain outliers, are recorded on an ordinal scale, or have different shapes across groups. The choice should follow the scientific target and the data structure, not a rule that one test is always preferable whenever data are non-normal.
Targets and Assumptions
Both procedures require a design with independent groups: an observation in one group is not paired with a particular observation in the other, and observations are independent within each group. A paired or repeated-measures design needs a method that accounts for the pairing; treating paired measurements as independent does not meet either test’s design requirement. Random sampling or random assignment supports the relevant interpretation, but does not by itself guarantee independence if the sampling or assignment design creates clusters.
| Feature | Welch’s two-sample t test | Mann–Whitney rank-sum test |
|---|---|---|
| Primary target | Difference in population means | Equality of population outcome distributions |
| Outcome scale | Quantitative, with meaningful differences and means | At least ordinal, so observations can be ordered |
| Variance assumption | Does not require equal population variances | Does not require equal variances for its distributional null |
| Shape assumption | For small samples, each group’s outcomes should be reasonably compatible with a normal model; larger samples often make inference more robust | No normal model is required, but a rejection indicates a distributional difference, not necessarily a difference in medians |
Welch’s test is designed to compare means even when the group variances differ. Its small-sample inference relies on the group outcomes being approximately normal, while with larger samples it is often reasonably robust to moderate non-normality. Severe skewness or influential outliers can still make the mean and its estimated standard error unstable, especially in small samples.
The Mann–Whitney test uses the ordering of observations rather than their numerical distances. It is appropriate when the outcome is ordinal or quantitative and a distributional comparison is meaningful. If both distributions have the same shape and spread and differ only by a location shift, a rank-sum result can be interpreted as evidence about that shift. Without this additional shape condition, the test is not simply a test of equal medians: different spreads or shapes can produce different distributions even when medians agree.
Welch’s Statistic and Its Degrees of Freedom
Let \(\bar{x}_A\) and \(\bar{x}_B\) be the sample means, \(s_A\) and \(s_B\) the sample standard deviations, and \(n_A\) and \(n_B\) the sample sizes. The estimated standard error of the difference in means is calculated from the two groups’ separate variances. Under \(H_0\), the test statistic is:
Welch’s method uses an approximate degrees-of-freedom value rather than assuming a common variance or using \(n_A+n_B-2\) degrees of freedom.
The \(p\)-value is found from a \(t\) distribution with this approximate \(df\), using the alternative hypothesis selected for the question. For a two-sided alternative, it is the probability under \(H_0\) of a \(t\) statistic at least as far from zero as the observed statistic, in either direction.
Worked Comparisons
Worked Example: A Mean Difference and a Rank Comparison
Suppose hypothetical independent samples record a quantitative outcome in arbitrary units: Group A has values 12, 13, 14, 15, 16, and Group B has values 8, 10, 11, 12, 14. Assume the observations were independently sampled and there is no pairing. The sample means are \(\bar{x}_A=14\) and \(\bar{x}_B=11\). Group A has sample variance \(s_A^2=2.5\), since its squared deviations from the mean sum to 10 and \(10/(5-1)=2.5\); Group B has \(s_B^2=5\), since its squared deviations sum to 20 and \(20/(5-1)=5\).
For Welch’s test, the standard error is \(\sqrt{2.5/5+5/5}=\sqrt{1.5}\approx1.225\). Thus \(t=(14-11)/\sqrt{1.5}\approx2.449\). The degrees of freedom are:
For a two-sided test, the \(t\) distribution with about 7.2 degrees of freedom gives \(p\approx0.043\). At \(\alpha=0.05\), this is evidence against equal population means; the estimated sample mean in A is 3.00 units higher, though this p-value is close to the 0.05 threshold. The calculation is appropriate if the independent-sample design holds and the small samples are reasonably compatible with the normal model. These five observations per group do not by themselves establish that shape condition, so plots and study context should be checked.
The rank comparison describes a different feature. Among the 25 cross-group pairs, count a pair as an A win if its A value is larger than its B value and as half a win if the values tie. The A value 12 contributes 3.5 wins (it exceeds 8, 10, and 11, and ties 12); 13 contributes 4; 14 contributes 4.5 (including half credit for the tie at 14); 15 contributes 5; and 16 contributes 5. Therefore \(U_A=3.5+4+4.5+5+5=22\). This rank summary indicates that A values tend to exceed B values in cross-group comparisons. It is not a test of the 3-unit difference in means.
Worked Example: An Outlier Can Separate the Targets
Consider hypothetical independent samples A: 1, 2, 3, 4, 20 and B: 5, 6, 7, 8, 9. Both groups have the same sample size, but A has one unusually large value. The means are \(\bar{x}_A=6\) and \(\bar{x}_B=7\). The sample variances are \(s_A^2=62.5\), because the squared deviations sum to 250 and \(250/4=62.5\), and \(s_B^2=2.5\), because its squared deviations sum to 10.
For Welch’s test, \(SE=\sqrt{62.5/5+2.5/5}=\sqrt{13}\approx3.606\), so \(t=(6-7)/\sqrt{13}\approx-0.277\). The approximate degrees of freedom are:
The two-sided \(p\)-value is approximately 0.80. These small samples are not persuasive evidence of a mean difference, and the large outlier makes the normal-model check especially important. The estimated mean difference is also sensitive to that observation.
For the rank comparison, each of A’s values 1, 2, 3, and 4 is below every B value, so each contributes zero A wins. The value 20 exceeds all five B values, contributing five wins. Thus \(U_A=5\), compared with a null expectation of \(n_An_B/2=12.5\). The rank summary emphasizes that A’s observations are mostly low relative to B, despite A’s large value. It does not make the mean difference disappear; it answers a different question. A formal rank-sum conclusion would require its own test and \(p\)-value.
Worked Example: Choosing a Rank-Based Comparison for Ordinal Data
A hypothetical study records independent ordinal ratings from 1 (lowest) to 5 (highest). Group A’s ratings are 1, 2, 2, 3, 5; Group B’s are 2, 3, 3, 4, 5. The ratings have a natural order, but the gaps between consecutive categories need not represent equal amounts. If the question concerns whether the rating distributions differ, the rank-sum test has a target that respects the ordering without treating the scale as equally spaced.
To obtain the pairwise rank summary, compare every A rating with every B rating, counting a larger A rating as one win and a tie as half a win. A rating of 1 contributes 0; the A rating of 2 is below two B ratings, ties one, and exceeds two, contributing \(2+0.5=2.5\); the other A rating of 2 contributes another 2.5. The A rating of 3 is below one B rating, ties two, and exceeds two, contributing \(2+2(0.5)=3\). The rating of 5 exceeds four B ratings and ties one, contributing \(4.5\). Therefore \(U_A=0+2.5+2.5+3+4.5=12.5\), exactly the null-center value \(5(5)/2=12.5\).
This calculation alone does not establish that the distributions are equal; a test requires the sampling distribution or an appropriate approximate \(p\)-value. It does show that the observed pairwise ordering is centered at the null value. A Welch test could calculate a difference between numerical averages, but that would treat the ordinal scores as quantitative measurements with meaningful equal spacing, an assumption not justified by the rating scale as described.
Diagnostics and Practical Choice
Begin by confirming that the groups are genuinely independent and the outcome is measured comparably in each. Then use group-specific dotplots, boxplots, or histograms to inspect location, spread, skewness, gaps, and outliers. A single pooled plot can hide a problem that occurs in only one group. With small samples, plots provide limited evidence about population shape, so they should be interpreted cautiously.
If the question concerns a difference in average quantitative outcome, Welch’s test directly targets that difference. If it concerns an ordinal ordering or a difference in outcome distributions, consider Mann–Whitney.
Verify independent groups and observations. For Welch, confirm that numerical means are meaningful. For a rank-sum test, verify that the outcome can be ordered meaningfully.
Assess whether outliers or severe skewness make a small-sample mean comparison fragile. For a rank-sum interpretation as a location shift, also assess whether the groups have reasonably similar shapes and spreads.
Describe a Welch result in terms of evidence about means, or a rank-sum result in terms of evidence about distributions or ordering. Do not translate one test’s result into the other test’s target without additional assumptions.
Common Mistake: “Nonparametric Means No Assumptions”
A common mistake is to see skewed data, choose Mann–Whitney automatically, and then report that the test proves the medians differ. The correction is to identify the actual target: the standard rank-sum test compares distributions. If the distributions differ in spread or shape, its result need not be evidence of a median difference. Conversely, Welch’s test does not require equal variances, and moderate non-normality does not automatically rule it out, particularly with adequate sample sizes.
A useful report names the groups, outcome, and target explicitly. For example: “Welch’s test found evidence of a difference in mean recovery time,” or “The rank-sum test found evidence that the outcome distributions differed.” Neither result establishes that group membership caused the difference unless the study design supports a causal conclusion.
Check Your Understanding
For each question, identify the relevant target or assumption before choosing a procedure.
- A researcher wants to compare the mean number of hours two independent groups of students slept. What does Welch’s test evaluate, and what should be checked in group-specific plots?
- Why does a Mann–Whitney result not automatically establish a difference in population medians?
- Two groups have very different sample variances. Does that alone violate Welch’s variance assumption? Explain.
- A pain rating is recorded on an ordered scale, but equal spacing between categories is not defensible. Which test has a more natural measurement-scale target, and what does it compare?
- Measurements come in before-and-after pairs for the same patients. Why would treating the two sets as independent groups be inappropriate for these procedures?