Why Use a Chance-Adjusted Agreement Measure?
Two runs of a subgroup-discovery procedure can assign different labels to the same patients even when they produce the same partition: cluster numbers are arbitrary. A useful run-to-run comparison must therefore compare who is grouped together, not whether a patient was called “cluster 1” in both runs. The adjusted Rand index (ARI) does this using pairs of observations and corrects the observed agreement for agreement expected by chance under a specified fixed-margin model.
The ARI is a single summary of agreement between two hard partitions of the same observations. It is useful when comparing two runs on a common set of patients, but it does not explain which subgroup changed or why. The co-clustering matrices introduced in the earlier material on co-clustering matrices provide a complementary view of pairwise membership.
A partition must assign each included observation to exactly one group. The comparison requires the same observations in both partitions, although the two runs may use different numbers of groups. The group labels themselves do not matter. For convenience, define \(M=\binom{n}{2}\), the total number of distinct observation pairs; the usual pair-count formula requires \(n\geq 2\).
Definition and Chance Correction
Let \(S=\sum_{i,j}\binom{n_{ij}}{2}\) be the number of pairs placed together in both partitions. Also let \(S_A=\sum_i\binom{a_i}{2}\) be the number placed together in \(A\), and \(S_B=\sum_j\binom{b_j}{2}\) the number placed together in \(B\). The ARI is
The term \(S\) is observed joint pair agreement. The correction \(S_A S_B/M\) is its expected value under a random-assignment model that holds the group sizes in both partitions fixed. The denominator scales the difference between observed and expected agreement by a conventional normalization based on the two partitions’ within-group pair counts.
The chance reference is conditional on the observed group sizes. It is not a universal model for random clustering: it does not account for the clustering algorithm, the omics features, or how the data were resampled. Interpret “near zero” as near chance under this particular fixed-margin model, not as proof that two runs are unrelated in every relevant sense.
Why the Expected Count Has This Form
The chance correction follows from counting pairs under random allocation with fixed margins. Consider any one of the \(S_A\) pairs placed together in partition \(A\). Under random allocation to the groups of \(B\), with group sizes \(b_1,\ldots,b_s\) fixed, the probability that the pair is together in \(B\) is
Here the first factor is the probability that the first observation lands in group \(j\); the second is the conditional probability that the other observation lands there too. Summing the indicator of joint agreement over all \(S_A\) pairs gives
Subtracting this conditional expected count from the observed count gives the numerator. The denominator is the average of \(S_A\) and \(S_B\), less the same expected count. Thus the formula measures excess joint agreement relative to the fixed-margin reference and applies a symmetric normalization. This derivation justifies the chance term; it does not imply that the fixed-margin randomization describes every source of run-to-run variability.
Calculating the Index
Worked Example: A Partial Merge Across Two Runs
Suppose six hypothetical patients are partitioned as follows: run \(A\) assigns patients 1–3 to its first group and 4–6 to its second; run \(B\) assigns patients 1–2 to its first group and 3–6 to its second. The contingency table is
| Run B: group 1 | Run B: group 2 | Run A total | |
|---|---|---|---|
| Run A: group 1 | 2 | 1 | 3 |
| Run A: group 2 | 0 | 3 | 3 |
| Run B total | 2 | 4 | 6 |
The observations are the same in both runs, each is assigned once, and \(n=6\), so \(M=\binom{6}{2}=15\). The joint pair count is \(S=\binom{2}{2}+\binom{1}{2}+\binom{0}{2}+\binom{3}{2}=1+0+0+3=4\). The margins give \(S_A=\binom{3}{2}+\binom{3}{2}=6\) and \(S_B=\binom{2}{2}+\binom{4}{2}=7\). Therefore,
The runs show positive, but not especially strong, pair agreement relative to the fixed-margin chance reference. This value is not the proportion of patients whose labels match: labels are arbitrary, and the index counts pairs.
Worked Example: Relabeling Does Not Reduce Agreement
Suppose run \(A\) places patients \(\{1,2,3\}\) and \(\{4,5\}\) into two groups. Run \(B\) makes exactly the same split but calls the groups in the reverse order. Its contingency table has nonzero cells 3 and 2, with the other cells zero. Thus \(S=3+1=4\), \(S_A=S_B=4\), and \(M=\binom{5}{2}=10\). Substitution gives
The conclusion is exact agreement, despite the swapped names. A label-by-label comparison would incorrectly suggest disagreement; a partition comparison based on pairs is invariant to renaming.
Worked Example: Less Agreement Than the Chance Reference
For four hypothetical patients, let run \(A\) split them into \(\{1,2\}\) and \(\{3,4\}\), while run \(B\) splits them into \(\{1,3\}\) and \(\{2,4\}\). Every cell of the \(2\)-by-\(2\) contingency table is 1, so \(S=0\). Each run has \(S_A=S_B=\binom{2}{2}+\binom{2}{2}=2\), and \(M=\binom{4}{2}=6\). Hence,
No pair is together in both runs, even though the fixed-margin reference expects \(2(2/6)=2/3\) such pairs. The negative value records less joint agreement than that reference expects; it does not mean that half the patients were “negatively assigned.”
Group Sizes, Diagnostics, and Alternatives
The margins affect both the chance correction and the scale of the ARI. In particular, large groups contribute many pairs: a group of size \(m\) contributes \(\binom{m}{2}\), which grows quadratically with \(m\). Consequently, a high overall ARI can coexist with substantial instability in a small subgroup. Conversely, changing group sizes changes the fixed-margin expectation, even if a familiar unadjusted agreement summary stays the same.
Worked Example: Same Rand Index, Different Group-Size Correction
Consider two hypothetical comparisons, each with \(n=10\). The first has contingency table \(\begin{pmatrix}4&1\\1&4\end{pmatrix}\), with margins 5 and 5 in both partitions. Its joint pair count is \(S=6+0+0+6=12\), and each margin pair total is \(S_A=S_B=20\). The number of pairs separated in both partitions is \(45-20-20+12=17\), so the unadjusted Rand index, the proportion of pairs either together in both or separated in both, is \((12+17)/45=29/45\approx0.6444\). The ARI is
The second comparison has table \(\begin{pmatrix}7&1\\1&1\end{pmatrix}\), with margins 8 and 2 in both partitions. Here \(S=21\), \(S_A=S_B=29\), and the number separated in both is \(45-29-29+21=8\). Its unadjusted Rand index is also \((21+8)/45=29/45\approx0.6444\), but its ARI is
The same raw proportion of agreeing pairs does not imply the same chance-adjusted agreement. The different margins produce different expected pair counts and normalizations. The comparison illustrates why group sizes must be examined alongside the overall ARI; it does not establish that one particular margin pattern is inherently better.
Before calculating or interpreting the index, check the following conditions and features of the comparison:
- Common observations: Both partitions must refer to the same included patients. If only the intersection of two resamples is compared, state that the result describes agreement on that intersection, not on every patient in either run.
- Valid partitions: Each included patient must have one hard assignment in each run. Missing assignments, overlapping groups, or soft membership probabilities require a clearly specified alternative handling.
- Margins and small groups: Inspect the contingency table and group sizes. The total score can conceal changes concentrated in a small subgroup because the measure aggregates pairs.
- Degenerate cases: If the denominator is zero, the displayed formula is undefined. This can occur, for example, when both partitions put every observation in a single group or when both put every observation in a singleton group. Software may impose a convention for such cases; report that convention rather than treating the formula as defined.
- Repeated observations: If a resampling procedure includes the same patient more than once, decide whether the unit of comparison is the patient or the resampled occurrence. Applying the ordinary formula to repeated rows counts occurrences as separate units.
The unadjusted Rand index is easier to describe as the fraction of agreeing pairs, but it does not subtract the agreement expected under fixed margins. Label matching after optimal relabeling is another possible comparison, but it focuses on matching whole groups and can depend on how unmatched or split groups are handled. The ARI instead treats all pairs symmetrically and requires no label alignment. A co-clustering matrix can retain pair-specific agreement across runs rather than compressing it into one number. Each summary answers a different question; none by itself diagnoses the biological meaning of a discovered subgroup.
Common Mistake and Practical Interpretation
A common mistake is to read an ARI of 0.8 as “80% of patients received the same cluster label,” or to describe an ARI near zero as proof that the runs are independent. Neither conclusion follows. The score is a normalized, chance-adjusted pair-count statistic. Its reference assumes random assignments conditional on the observed margins; it does not measure causal relevance, clinical validity, or uncertainty in subgroup membership.
In a stability analysis, report the ARI together with the two group-size profiles, the observations compared, and the resampling or perturbation context. If the index is high, that supports overall partition agreement for those observations and that comparison. It does not guarantee that every subgroup is stable. If it is low, inspect the contingency table to distinguish a broad rearrangement from a change concentrated in one group. A single scalar is a summary, not a substitute for those checks.
Check Your Understanding
Answer each question, then compare with the solutions below.
- Two runs assign the same observations to the same groups but use different group names. What ARI should the partition comparison give, assuming the formula is defined?
- For \(n=5\), a comparison has \(S=3\), \(S_A=4\), and \(S_B=5\). Calculate the ARI.
- What does a negative ARI mean, and which chance reference is relevant?
- Why can two comparisons with the same unadjusted Rand index have different ARIs?
- What should be checked if an ARI calculation has a zero denominator?
Solutions. (1) The ARI is 1: changing group names does not change pair memberships. (2) \(M=\binom{5}{2}=10\); the expected count is \(S_A S_B/M=4(5)/10=2\). Thus \(\operatorname{ARI}=(3-2)/[(4+5)/2-2]=1/2.5=0.4000\). (3) The observed joint pair count is below its expected value under random allocation conditional on the two observed sets of group sizes. It is not a probability or a percentage. (4) The margins determine the fixed-margin expected count and the normalization, so different group sizes can yield different ARIs for the same raw pair agreement. (5) Check for degenerate partitions and confirm \(n\geq2\); the formula is undefined when its denominator is zero, so report any software convention used rather than claiming a value from the formula.