Why Clear Reporting Matters
A post hoc analysis can produce a list of comparisons, estimates, intervals, and p-values. A useful results summary does more than reproduce that list: it tells readers which groups were compared, how large the estimated differences were, how uncertainty was handled, and what the results mean in context. Without those details, even a correctly calculated analysis can be easy to misread.
Multiple comparisons require particular care. If many comparisons are considered, reporting ordinary, unadjusted intervals or p-values as though only one comparison had been planned can make the evidence appear stronger than it is. A clear report names the method used to account for the comparison family and avoids turning statistical significance into a claim of practical importance or causation.
What a Results Summary Should Include
Before reporting numbers, make the target and the comparison family clear. The family might be all pairwise comparisons among the groups, or only comparisons of each treatment group with a control. Those are different questions and can call for different procedures. State the family explicitly rather than using a vague phrase such as “the groups were compared.”
For every comparison you report, include the group labels, the estimated difference and its direction, and the outcome’s units. Then report either an adjusted interval, an adjusted p-value, or both. Name the comparison procedure and, when useful, the confidence level or significance level. If the analysis was conducted on transformed values, specify the scale on which the reported differences are expressed; an interpretation on the original scale may require additional care.
The assumptions depend on the method. For example, Tukey or Tukey–Kramer comparisons use the equal-variance one-way ANOVA framework: observations are independent, the within-group errors are approximately normal, and the groups share a common variance. Games–Howell comparisons are designed for unequal variances and unequal sample sizes; they still require independent observations and are most reliable when the within-group distributions are reasonably well behaved. Dunnett comparisons address a family of treatment-versus-control comparisons under the common-variance ANOVA framework. The method should match the design and assumptions; the report should not imply that an adjustment itself repairs a poor design or invalid model.
A simultaneous 95% family of intervals is constructed so that, under the method’s assumptions, the procedure has at least 95% probability of covering all the true differences in that family. This is not a claim that there is a 95% probability that a particular fixed parameter lies in its observed interval. An adjusted p-value accounts for the stated family; it is not the probability that the null hypothesis is true.
Say whether the analysis covers all pairwise comparisons or a restricted set, such as each treatment versus a control.
Report the procedure used, such as Tukey–Kramer, Games–Howell, or Dunnett, and ensure it fits the variance and design assumptions.
Give differences in a consistent direction with units, then provide adjusted intervals or adjusted p-values and the relevant confidence or significance level.
Describe the size and uncertainty of the differences, distinguish statistical evidence from practical importance, and avoid causal claims unsupported by the study design.
Worked Examples
Worked Example: Reporting All Pairwise Comparisons
In a hypothetical study, three independent groups of participants complete a task, and completion time is recorded in minutes. Their sample means are 12.4 minutes for A, 15.1 minutes for B, and 18.0 minutes for C. Suppose the equal-variance ANOVA assumptions are considered reasonable and a Tukey–Kramer analysis is used for the family of all three pairwise comparisons. The hypothetical software output gives the following simultaneous 95% intervals and adjusted p-values.
First calculate each difference in the stated direction, using group A minus group B, then A minus C, and then B minus C:
Suppose the adjusted output is: A minus B, interval \([-5.2,-0.2]\) minutes and adjusted \(p=0.031\); A minus C, interval \([-8.1,-3.1]\) minutes and adjusted \(p<0.001\); B minus C, interval \([-5.4,-0.4]\) minutes and adjusted \(p=0.019\). All three intervals exclude zero, and all three adjusted p-values are below 0.05. The comparisons therefore provide evidence of differences for this stated family at the 5% familywise level.
A clear summary could read: “Using Tukey–Kramer simultaneous 95% intervals for all pairwise comparisons, mean completion time was lower in A than B by an estimated 2.7 minutes (95% interval for A minus B: −5.2 to −0.2; adjusted \(p=0.031\)) and lower in A than C by 5.6 minutes (−8.1 to −3.1; adjusted \(p<0.001\)). B was lower than C by 2.9 minutes (−5.4 to −0.4; adjusted \(p=0.019\)).” The negative differences match the stated order of subtraction. These results support differences in mean task time; they do not by themselves establish that the differences matter practically.
Worked Example: Reporting a Borderline Games–Howell Comparison
A hypothetical comparison of three independent clinics records a patient satisfaction score. The sample means are 42.0 points for A, 45.4 for B, and 46.3 for C. Group spreads and sample sizes differ, so suppose the analyst uses Games–Howell for all pairwise comparisons after considering the study design and data. Assume independence is reasonable and the within-group distributions are sufficiently well behaved for the procedure. Hypothetical adjusted output for A minus B is a difference of −3.4 points, a 95% adjusted interval from −7.0 to 0.2 points, and adjusted \(p=0.11\).
The estimate is calculated directly from the means:
The interval includes zero, and the adjusted p-value exceeds 0.05. Thus, at the 5% familywise level, this comparison does not provide sufficient evidence of a difference in mean satisfaction scores. The result does not show that the group means are equal. The interval remains compatible with A’s mean being as much as 7.0 points lower than B’s, or as much as 0.2 points higher.
A suitable report is: “For the all-pairs family, the Games–Howell comparison estimated that clinic A’s mean satisfaction score was 3.4 points lower than clinic B’s (A minus B = −3.4; 95% adjusted interval: −7.0 to 0.2; adjusted \(p=0.11\)). This comparison was not statistically significant at the 5% familywise level.” The wording reports the estimate even though the test is nonsignificant, and it does not describe the result as proof of no difference.
Worked Example: Reporting Treatment Comparisons With a Control
In a hypothetical experiment, an outcome is measured in score points for a control group and two treatment groups. The sample means are 100.0 points for control, 106.2 for treatment A, and 103.1 for treatment B. Suppose observations are independent, the common-variance ANOVA assumptions are reasonable, and Dunnett’s procedure is used for the two treatment-versus-control comparisons. The hypothetical output gives adjusted 95% intervals of 1.0 to 11.4 points for A minus control, with adjusted \(p=0.018\), and −2.1 to 8.3 points for B minus control, with adjusted \(p=0.29\).
The estimated differences and their directions are:
For treatment A, the adjusted interval is entirely above zero, and the adjusted p-value is below 0.05. For treatment B, the interval contains zero and the adjusted p-value is above 0.05. A concise report is: “In Dunnett-adjusted comparisons with the control, treatment A had a mean score 6.2 points higher than control (95% adjusted interval: 1.0 to 11.4; adjusted \(p=0.018\)). The estimated difference for treatment B was 3.1 points (95% adjusted interval: −2.1 to 8.3; adjusted \(p=0.29\)), which was not statistically significant at the 5% familywise level.” The evidence for A does not establish that a 6.2-point change is practically important; that judgment needs context, such as a meaningful-change threshold.
Why an Adjusted Interval and Test Should Agree
When an adjusted two-sided test and an adjusted confidence interval are based on the same comparison procedure and the same family, they should give consistent decisions at matching levels. A 5% test rejects a zero difference exactly when the corresponding 95% adjusted interval excludes zero. This agreement is useful both for interpretation and for checking a report.
The reason is the definition of the interval. It contains the values of the difference that would not be rejected by the corresponding two-sided test. In particular, if zero lies in the interval, the test does not reject a zero difference at that level. If zero lies outside it, the test rejects. The adjustment changes the critical values or reference distribution to account for the comparison family, but the same adjusted procedure must be used for the interval and test to preserve this correspondence.
Common Reporting Pitfalls
- Leaving out the adjustment. A list of p-values is not enough if readers cannot tell whether they are adjusted. Name the procedure and which comparisons belong to its family.
- Changing the subtraction order. “A is 3 points above B” corresponds to A minus B being positive. If the reported estimate is B minus A, its sign must be negative. State the order and keep it consistent.
- Calling a nonsignificant result “no difference.” A nonsignificant comparison may still have an interval that includes differences of practical importance. Describe the estimate and uncertainty instead.
- Equating statistical significance with importance. A small difference can be statistically detectable, while a potentially important difference can remain uncertain. Interpret size using relevant subject-matter context.
- Overstating the design. A difference between group means does not by itself show that group membership caused the outcome. Causal language requires a design that supports causal conclusions.
Check Your Understanding
Use the direction of subtraction, comparison family, and adjusted uncertainty to guide each answer.
- If \(\bar{x}_A=8.2\) and \(\bar{x}_B=10.0\), what is the estimated difference A minus B, and how should its sign be interpreted?
- A 95% adjusted interval for A minus B is −4.0 to 1.0. What does the interval say about evidence of a difference at the matching two-sided 5% level?
- Why should a report say whether its family consists of all pairwise comparisons or only comparisons with a control?
- An adjusted p-value is 0.03. Does this mean there is a 3% probability that the null hypothesis is true? Explain.
- What additional information is needed before calling a statistically significant difference practically important?