From Test Calculation to Statistical Finding
A test result is useful only when a reader can tell what was tested, how the reference distribution was chosen, how strong the evidence was, and what the conclusion does—and does not—establish. A concise report of a Kruskal–Wallis test or a runs test therefore needs more than a statistic and the words “significant” or “not significant.”
For Kruskal–Wallis, report the test statistic, the chi-square approximation and degrees of freedom when that approximation is used, the p-value, and a conclusion in context. For a runs test, report the two symbol counts, the observed number of runs, whether the p-value is exact or approximated, and the direction of the alternative. In either case, note a material limitation, such as the approximation used or the scope of the conclusion.
What to Include for Kruskal–Wallis
Suppose \(k\) independent groups have sample sizes \(n_j\), with \(N=\sum_{j=1}^{k}n_j\) observations in total. Rank all observations together, and let \(R_j\) be the sum of the ranks in group \(j\). With no tied observations, the Kruskal–Wallis statistic is
For sufficiently large samples, the usual reference distribution is approximately chi-square with \(k-1\) degrees of freedom. If there are tied observations, use the tie-corrected statistic. If \(t_m\) is the number of observations in tie group \(m\), the correction factor is
The chi-square p-value is the area to the right of the observed statistic. State that it is an approximation when using this reference distribution. The test requires independent samples and an outcome that can be ranked. Its null hypothesis is that the group distributions are identical. A conclusion specifically about differences in location, such as medians, is most defensible when the group distributions have similar shapes and spreads. A significant omnibus result says that the data provide evidence against identical distributions; by itself, it does not identify which groups differ.
Worked Example: Reporting a Kruskal–Wallis Test
Method and question. In a hypothetical study, four independent measurements are recorded for each of three treatment groups. The observations, in increasing order across all groups, have the following group labels: A, B, C, A, B, C, C, A, B, C, A, B. There are no ties. At the 5% significance level, assess whether the group distributions differ using the large-sample Kruskal–Wallis test.
Rank and sum. The labels are already in rank order from 1 through 12. Therefore \(N=12\), each group has \(n_j=4\), and the rank sums are \(R_A=1+4+8+11=24\), \(R_B=2+5+9+12=28\), and \(R_C=3+6+7+10=26\). As a check, \(24+28+26=78=N(N+1)/2\).
Calculate the statistic. Substitution gives
There are \(k-1=2\) degrees of freedom. For a chi-square random variable with 2 degrees of freedom, the upper-tail probability at \(H=0.1538\) is \(\exp(-0.1538/2)=0.9260\), to four decimal places. This is an approximate p-value from the chi-square reference distribution.
Report and conclusion. “A Kruskal–Wallis test gave \(H=0.1538\), with an approximate \(\chi^2_2\) reference distribution and \(p=0.9260\). At the 5% level, we fail to reject the null hypothesis that the three group distributions are identical; these data do not provide evidence of a difference. The p-value is based on the large-sample approximation.”
Check. The test statistic is nonnegative, the rank sums total 78, and the p-value is large for a statistic close to zero. The conclusion does not prove the distributions are identical.
What to Include for a Runs Test
A run is a maximal consecutive sequence of one symbol. For example, in A A B B A, there are three runs: A A, B B, and A. In a two-symbol runs test, report the observed run count \(R\) and the counts \(n_1\) and \(n_2\) of the two symbols. These counts define the conditional reference distribution under random ordering.
When the test uses an exact method, state that the p-value was calculated from the exact distribution conditional on the observed symbol counts. If the test uses a normal approximation, report the approximation and, when used, the continuity correction. The expected number of runs and its variance under random ordering are
For an upper-tail alternative, a continuity-corrected normal statistic is \(z=(R-0.5-E[R])/\sqrt{\operatorname{Var}(R)}\). For a lower-tail alternative, use \(z=(R+0.5-E[R])/\sqrt{\operatorname{Var}(R)}\). The alternative must match the question: unusually few runs suggest clustering, while unusually many runs suggest alternation. An exact method is preferable when the sample counts are small or when the approximation is unsuitable.
Why the Exact Runs Probability Uses Arrangement Counts
Under the null hypothesis of random ordering, conditional on having \(n_1\) symbols of one type and \(n_2\) of the other, each distinct ordering is equally likely. There are \(\binom{N}{n_1}\) such orderings. Thus, the exact probability of a set of run counts is the number of compatible orderings divided by \(\binom{N}{n_1}\). This explains why an exact runs-test report should name both the method and the symbol counts: together, they specify the conditional reference distribution.
For example, with four A symbols and three B symbols, there are \(\binom{7}{4}=35\) possible orderings. Exactly two orderings have the minimum possible number of runs, \(R=2\): AAAABBB and BBBAAAA. Therefore, the exact lower-tail probability at \(R=2\) is \(2/35\). This count-based reasoning is also a useful check on an exact-test p-value.
Worked Example: Reporting an Exact Runs Test for Clustering
Method and question. A hypothetical sensor records the category of each of seven successive events as A, A, A, A, B, B, B. Assume the event order is being tested for clustering, with a lower-tail alternative to randomness. Use an exact test at \(\alpha=0.05\).
Count symbols and runs. There are \(n_A=4\) A symbols and \(n_B=3\) B symbols. The sequence contains one A run followed by one B run, so \(R=2\). This is the smallest possible number of runs when both symbols are present.
Find the exact p-value. Conditional on the counts, the number of equally likely orderings is
Only AAAABBB and BBBAAAA have \(R\leq 2\), so there are 2 orderings in the lower tail. The exact one-sided p-value is \(2/35=0.0571\), to four decimal places.
Report and conclusion. “An exact lower-tail runs test, conditional on 4 A symbols and 3 B symbols, gave \(R=2\) and \(p=0.0571\). At the 5% level, we fail to reject random ordering in favor of clustering. The test assesses the ordering of these two categories, not the mechanism that generated the events.”
Check. The p-value is the lower-tail probability, as required by the clustering alternative. Although the observed sequence is maximally clustered, its exact p-value is slightly above 0.05, so the stated decision follows the chosen significance level.
Worked Example: Reporting a Normal-Approximation Runs Test for Alternation
Method and question. A hypothetical quality-control log classifies 15 consecutive readings as A or B. It contains 8 A symbols and 7 B symbols, and the observed sequence has \(R=13\) runs. Suppose the specified alternative is unusually high alternation. Use a normal approximation with a continuity correction and test at the 5% level.
Calculate the reference mean and variance. Here \(N=8+7=15\). Therefore
The standard deviation is \(\sqrt{3.4489}=1.8571\). Because the alternative is an upper-tail alternative and \(R=13\) is an integer, the continuity-corrected boundary is \(12.5\). Hence
The upper-tail standard normal probability is approximately \(0.0149\), to four decimal places. This is an approximate p-value.
Report and conclusion. “A continuity-corrected normal-approximation runs test, with 8 A symbols and 7 B symbols, gave \(R=13\), \(z=2.172\), and an upper-tail \(p\)-value of approximately 0.0149. At the 5% level, we reject random ordering in favor of the specified alternative of unusually high alternation. The conclusion relies on a normal approximation.”
Check. The observed number of runs is above its null mean, and the upper-tail statistic is positive, matching the direction of the alternative. The reported p-value is small, consistent with rejection at 0.05.
Common Reporting Traps
- Calling a p-value the probability the null is true. The p-value is calculated under the null model; it is not the probability that the null hypothesis is correct.
- Omitting the reference method. A Kruskal–Wallis p-value from a chi-square approximation and a runs-test p-value from exact enumeration are not interchangeable descriptions. Name the method used.
- Using the wrong tail. A low-run clustering alternative calls for a lower-tail probability; a high-run alternation alternative calls for an upper-tail probability. A two-sided alternative requires a two-sided method.
- Overstating a nonsignificant result. “Fail to reject” means the test did not provide sufficient evidence at the chosen level. It does not establish that the null is true.
- Overstating a significant Kruskal–Wallis result. The omnibus test does not identify which groups differ. A claim about particular groups requires suitable follow-up comparisons, with appropriate attention to multiple testing.
- Ignoring design and assumptions. Rank-based comparisons require independent groups, while a runs test concerns the order of a two-symbol sequence conditional on its counts. Neither test repairs dependence or biased sampling in the data collection.
Writing a Finding That Can Be Checked
A report should be brief but complete enough for another reader to verify the decision. Include the test name, statistic, reference distribution or exact method, degrees of freedom or symbol counts, p-value, significance level, and contextual conclusion. State whether the p-value is approximate or exact, and avoid claims broader than the test supports.
Specify the distributions being compared or the randomness alternative being assessed.
For Kruskal–Wallis, include \(H\) and the chi-square degrees of freedom. For a runs test, include \(R\), the two symbol counts, and whether the method is exact or approximate.
Name the significance level and state “reject” or “fail to reject” in the direction of the alternative.
State a relevant limitation, such as an approximation, the need for follow-up comparisons, or the fact that a runs test only evaluates the observed sequence.
The central discipline is to separate evidence from interpretation. A p-value summarizes how unusual the observed statistic would be under the specified null model; the conclusion addresses that model and alternative, not every possible explanation for the data. This reporting approach is aligned with the SOA VEE Mathematical Statistics syllabus.
Check Your Understanding
For each problem, identify the method and report the conclusion using the stated significance level. Answers are given for checking.
- A Kruskal–Wallis test compares three independent groups and gives \(H=6.00\), with no ties. Use the chi-square approximation at the 5% level. State the degrees of freedom, p-value, and decision. (Answer: approximate \(\chi^2_2\), \(p=e^{-3}=0.0498\); reject the null hypothesis of identical group distributions at 5%.)
- A Kruskal–Wallis analysis of four independent groups gives \(H=4.20\), with no ties. Use the chi-square approximation at the 5% level. State the reference degrees of freedom and conclusion. (Answer: approximate \(\chi^2_3\), upper-tail \(p\approx0.2408\); fail to reject the null hypothesis at 5%.)
- A two-symbol sequence has \(n_A=3\), \(n_B=2\), and \(R=2\). For a lower-tail exact test of clustering, how many arrangements are possible and what is the p-value? State the decision at 5%. (Answer: \(\binom{5}{3}=10\) arrangements; 2 have \(R\leq2\), so \(p=0.2000\); fail to reject random ordering at 5%.)
- A sequence contains 4 A symbols and 3 B symbols, with \(R=7\). For an upper-tail exact test of alternation, determine the p-value and conclusion at 5%. (Answer: there are \(\binom{7}{4}=35\) arrangements and only 1 has \(R\geq7\), so \(p=1/35=0.0286\); reject random ordering in favor of alternation at 5%.)
- A sequence contains 10 A symbols and 10 B symbols and has \(R=6\). Use a lower-tail normal approximation with continuity correction. The null mean is \(11\), and the null variance is \(4.7368\). Find the approximate p-value and decision at 5%. (Answer: \(z=(6.5-11)/\sqrt{4.7368}\approx-2.068\), lower-tail \(p\approx0.0193\); reject random ordering in favor of clustering at 5%, noting that the p-value is approximate.)