Common Errors in Multiple Comparisons

Why Multiple-Comparison Errors Matter

A conclusion from a set of comparisons depends on more than the individual p-values. It also depends on which comparisons were considered together, how the testing procedure was selected, and what a nonsignificant result can legitimately tell us. Overlooking any of these points can turn a technically correct calculation into an unsupported conclusion.

This tutorial focuses on three recurring errors: treating several unadjusted pairwise tests as though they form a protected family, choosing a procedure only after seeing the results, and interpreting failure to reject as proof of equality. The key discipline is to define the comparison family and decision rule in advance, then interpret the resulting evidence at the family level.

Model and Comparison Family

For a one-factor ANOVA with \(k\) groups, suppose observation \(j\) in group \(i\) follows the model

Y_{ij}=\mu_i+\varepsilon_{ij}, \qquad \varepsilon_{ij}\sim N(0,\sigma^2),

with errors independent across observations and a common variance \(\sigma^2\). The usual pooled pairwise comparisons rely on these assumptions: independent observations, approximately normal errors within groups, and equal population variances. The pooled error mean square \(MSE\) estimates \(\sigma^2\), and the error degrees of freedom are \(df\). The assumptions should be supported by the study design and examination of the data; a calculation alone does not establish them.

Definition: A comparison family is the set of hypotheses or intervals treated as one inferential group for controlling an error rate. The family-wise error rate is the probability of making at least one Type I error among the comparisons in that family when their null hypotheses are true.

For \(k\) groups, all pairwise comparisons comprise \(m=k(k-1)/2\) distinct differences between population means. For example, three groups produce three comparisons: A–B, A–C, and B–C. If the scientific question concerns only a prespecified subset, that smaller set may instead define the family. The family is determined by the inferential plan, not by whichever results look interesting after analysis.

Error 1: Treating Unadjusted Pairwise Tests as a Protected Family

A common mistake is to run each pairwise test at significance level \(\alpha\), then report all of the results as if the chance of any false positive were still \(\alpha\). Each test may have Type I error rate \(\alpha\) individually, but the chance of at least one false positive across multiple tests can be larger. The exact amount depends on how the tests relate; the individual level does not itself guarantee family-wise control.

Bonferroni adjustment provides one direct way to control the family-wise rate. For \(m\) comparisons, test each at level \(\alpha/m\), or multiply each unadjusted p-value by \(m\), limiting the adjusted value to 1. The justification is the union bound:

P(\text{at least one false rejection}) \leq \sum_{r=1}^{m}P(\text{false rejection on comparison }r) \leq m\left(\frac{\alpha}{m}\right)=\alpha.

This guarantee does not require the tests to be independent. It can be conservative, but it makes clear why several unadjusted tests cannot simply be treated as one protected family.

Worked Example: An Unadjusted Pairwise Result

Suppose a hypothetical study compares mean recovery time, in days, among three treatment groups. Assume the one-way ANOVA model is appropriate: observations were independently assigned and measured, and prior process checks support approximately normal errors with a common variance. The hypothetical summary gives \(n_A=2\), \(n_B=2\), \(n_C=26\), \(MSE=1.8\) days squared, and \(df=27\). The sample means are \(\bar{x}_A=12\) days and \(\bar{x}_B=15\) days. Consider the A–B difference, defined as A minus B.

The estimated difference is \(12-15=-3\) days. Under the pooled model, its standard error and t statistic are

SE=\sqrt{MSE\left(\frac{1}{n_A}+\frac{1}{n_B}\right)} =\sqrt{1.8\left(\frac{1}{2}+\frac{1}{2}\right)} =\sqrt{1.8}=1.342, \qquad t=\frac{-3}{1.342}=-2.236.

With 27 error degrees of freedom, the two-sided unadjusted p-value is approximately \(0.03381\). Considered as a single test at \(\alpha=0.05\), it would be significant. But if the family is all three pairwise comparisons, Bonferroni uses \(0.05/3=0.01667\) per test. Equivalently, the adjusted p-value for A–B is

p_{\text{adj}}=\min(3(0.03381),1)=0.1014.

The A–B result does not meet the family-wise 0.05 criterion. The unadjusted result is not a protected family-wise conclusion. Because the two groups in this illustration have very small sample sizes, the normal-error assumption would be difficult to assess from these data alone; the stated inference is conditional on the model assumptions being credible.

Error 2: Choosing a Procedure After Seeing Results

A procedure should be selected to match the scientific aim and comparison family, not because it happens to produce the preferred answer once the data are visible. For instance, an analyst might inspect several pairwise p-values, try different adjustments, and report the method that makes the most comparisons significant. The nominal error guarantee of any one procedure applies when that procedure is used as specified; it does not automatically cover the extra opportunity to select among procedures based on the observed results.

This is a form of data-dependent selection. Even if each candidate method has an advertised family-wise error rate when chosen in advance, selecting whichever method gives the smallest adjusted p-values can increase the overall chance of a false discovery. The issue is not that one particular method is always preferable. It is that the decision rule has changed after observing the evidence.

Worked Example: Procedure Shopping After Results

A hypothetical experiment compares three teaching formats, and the researchers are interested in all three pairwise differences in mean quiz score. Before collecting data, they could specify all pairwise comparisons as the family and choose a procedure suited to that goal. Suppose instead that, after looking at the sample means and unadjusted p-values, they try several methods and keep the one that labels the largest number of differences significant.

The mistake is to report the selected method as though it had been the sole, prespecified analysis. Its nominal guarantee applies to its own stated procedure, not necessarily to a process that searches across procedures and selects a favorable result. The extra selection step can make the probability of at least one false positive exceed the advertised level.

The correction is to define the scientific comparisons and select the method before inspecting inferential results. If a data-dependent choice has already been made, the analysis should disclose that choice and avoid presenting the nominal guarantee as if the selection had not occurred. A method should be chosen for the question—for example, all pairwise comparisons versus a targeted set—not for the significance pattern it produces.

State the scientific question.
Specify whether the goal concerns every pair, a set of targeted comparisons, or another defined family.

Choose the error-control procedure.
Select a method appropriate to the family and the desired inferential guarantee before reviewing which findings are significant.

Apply and interpret consistently.
Use the stated procedure for the family, and do not switch methods selectively to obtain a preferred conclusion.

Error 3: Interpreting Nonsignificance as Equality

A nonsignificant result means the data do not provide sufficient evidence to reject the particular null hypothesis at the stated error level and under the chosen procedure. It does not prove that the two population means are equal, nor does it show that their difference is exactly zero. A test may fail to detect a real difference because the estimate is imprecise.

A simultaneous confidence interval helps show the range of differences compatible with the family-wise procedure. If the interval includes zero, the corresponding null difference is not rejected by that procedure. But if the interval also includes differences that would matter in practice, it would be misleading to describe the groups as equivalent. Establishing practical equivalence requires a prespecified margin and an analysis designed to assess equivalence; ordinary nonsignificance is not that analysis.

Worked Example: A Nonsignificant Difference Is Not Proof of Equality

In a hypothetical three-group study of blood pressure reduction, suppose each group has \(n=10\), the ANOVA error mean square is \(MSE=4\) \((\text{mm Hg})^2\), and \(df=27\). The estimated difference between groups A and B is \(1.0\) mm Hg, defined as A minus B. Assume the observations are independent, the within-group errors are approximately normal, and the group variances are sufficiently similar for the pooled model.

For equal group sizes, the standard error is

SE=\sqrt{4\left(\frac{1}{10}+\frac{1}{10}\right)} =\sqrt{0.8}=0.8944\text{ mm Hg}.

The test statistic is \(t=1.0/0.8944=1.118\). Its two-sided p-value with 27 degrees of freedom is about \(0.273\), so the result is not significant under a family-wise 0.05 procedure. For all three pairwise comparisons, a Bonferroni simultaneous interval uses a two-sided critical value of about \(2.55\), corresponding to \(1-0.05/(2\cdot3)\) in each tail. The A–B interval is therefore

1.0\pm 2.55(0.8944) =1.0\pm2.281 =(-1.281,\ 3.281)\text{ mm Hg}.

This interval includes zero, but it also includes differences as large as about 3.28 mm Hg in the A-minus-B direction and about 1.28 mm Hg in the other direction. The data do not establish equality; they leave a range of plausible differences. The interval procedure gives simultaneous coverage for the three pairwise differences under the model, rather than a 95% chance that this particular fixed interval contains the parameter.

How to Diagnose the Three Errors

Before drawing conclusions from multiple comparisons, ask a small set of questions. These checks do not replace a suitable procedure; they help reveal when a result is being given a stronger interpretation than the analysis supports.

What is the family? Name the comparisons included, and check whether the reported adjustment matches that family.
Was the method chosen independently of the observed significance pattern? A method selected because it gives favorable results needs to be identified as data-dependent.
What does nonsignificance establish? It indicates insufficient evidence to reject under the stated procedure, not proof of no difference.
Are the model assumptions credible? Pairwise inference based on pooled ANOVA error relies on independent observations, approximately normal errors, and a common variance.

The most important practical correction is to keep the inferential plan coherent from question to conclusion. An adjustment cannot repair an undefined family, and a family-wise guarantee cannot justify a conclusion about comparisons that were not included. Likewise, a large p-value does not convert a difference estimate into zero.

Key takeaway: Define the comparison family, choose an appropriate error-control procedure before examining the results, and interpret nonsignificance as insufficient evidence—not as proof of equality.

Check Your Understanding

Answer each question using the definitions and reasoning in this tutorial.

There are four groups and all pairwise differences are of interest. How many comparisons are in the family, and why is testing each at level \(\alpha\) not automatically a family-wise \(\alpha\) procedure?
An analyst tries three adjustment methods after seeing the p-values and reports only the method with the most significant findings. What is the error in treating its nominal guarantee as fully applicable?
A Bonferroni-adjusted p-value is 0.12 when the family-wise significance level is 0.05. State the conclusion without claiming that the means are equal.
A simultaneous interval for a mean difference includes zero and also includes values considered important in context. What does this say about the evidence?
Name the model assumptions that should be considered when pairwise comparisons use the pooled ANOVA error mean square.