Consensus Clustering And Its Diagnostics

From Repeated Runs to a Consensus

Repeated clustering runs can disagree even when they use the same nominal number of clusters. The runs may differ because of resampling, feature perturbation, or other specified changes to the analysis. A consensus summary asks a pairwise question: how often are two samples assigned to the same cluster when both appear in a run? It turns a collection of label-dependent partitions into pairwise frequencies that can be inspected and used to construct a summary partition.

The resulting consensus is conditional on the runs that were performed. A frequency such as 0.8 means that the pair was co-clustered in 80% of the runs in which both samples were available. It is not a probability that the samples belong to a true biological subtype, nor does it establish that the clustering reflects meaningful biology. Its interpretation depends on the perturbation design and the analysis pipeline that generated the runs.

The matrix \(C\) is symmetric because co-clustering is a pairwise relation. Its off-diagonal entries lie between 0 and 1. A value near 1 indicates that a pair was usually grouped together; a value near 0 indicates that it was usually separated. Intermediate values record disagreement across runs. Different pairs can have different denominators, so it is important to report or inspect \(N_{ij}\) as well as \(C_{ij}\). A frequency based on one eligible run is less informative than the same frequency based on many runs.

Constructing a consensus partition requires an additional choice. One common approach is to treat \(1-C_{ij}\) as a dissimilarity and apply a clustering method to it, then cut the resulting hierarchy at a specified number of clusters. This is a construction rule, not a guarantee that the chosen partition is uniquely best. The consensus matrix is a summary of repeated pairwise outcomes; the partition is a further simplification of that summary.

A Pairwise Loss and Its Scope

A useful way to compare a candidate partition \(P\) with the consensus frequencies is to count pairwise disagreements. Write \(z_{ij}(P)=1\) when \(i\) and \(j\) are in the same cluster under \(P\), and 0 otherwise. For pairs with defined consensus, the pair-specific disagreement frequency is \(1-C_{ij}\) if \(P\) puts the pair together, and \(C_{ij}\) if \(P\) separates it. Summing these contributions gives a loss that weights each observed pair equally.

This loss is a descriptive criterion for comparing candidate partitions on the same set of defined pairs. It is not automatically the objective minimized by a hierarchical clustering algorithm. In particular, a linkage method applied to \(1-C_{ij}\) need not return the global minimizer of \(L(P)\), and imposing a cluster count can select a partition even when the consensus pattern does not support a clear grouping.

When every pair is available in every run, \(L(P)\) equals the mean number of pairwise disagreements between \(P\) and a run-specific partition. To see why, let \(d_{ij}^{(b)}(P)\) be 1 if the candidate and run \(b\) disagree about pair \(i,j\), and 0 otherwise. With complete availability, the average disagreement for pair \(i,j\) is \(1-C_{ij}\) if the candidate joins the pair and \(C_{ij}\) if it separates them. Therefore,

The equality follows by interchanging two finite sums. If availability varies, each pair-specific frequency still has a clear meaning, but summing those frequencies does not generally equal a fixed rescaling of the mean number of disagreements per run. One may instead define a run-level count \(D_b(P)=\sum_{i<j:I_{ij}^{(b)}=1}d_{ij}^{(b)}(P)\), and report its average over runs. If runs have different numbers of available pairs, a separate choice is needed: for example, report \(D_b(P)\) as a count, or report \(D_b(P)/M_b\), where \(M_b=\sum_{i<j}I_{ij}^{(b)}\) is the number of available pairs in run \(b\). These run-level summaries answer different questions from the pair-equally-weighted \(L(P)\).

Diagnosing Separation and Cluster Boundaries

A consensus partition should be examined alongside the matrix from which it was derived. A basic diagnostic compares the distribution of \(C_{ij}\) for pairs assigned to the same cluster with the distribution for pairs assigned to different clusters. Strongly separated distributions—with within-cluster values concentrated toward 1 and between-cluster values toward 0—are more consistent with a clear consensus than distributions that overlap broadly. The full distributions matter: a mean can conceal a substantial number of ambiguous pairs.

A simple descriptive summary for a sample \(i\) in a proposed cluster \(G\) is the difference between its average consensus with other members of \(G\) and its average consensus with samples outside \(G\). When both groups of comparison samples are nonempty and all relevant entries are defined, write

A large positive value indicates that sample \(i\) is more consistently co-clustered with its assigned cluster than with the rest of the samples. A value near zero indicates little contrast in this summary and can flag a weak boundary. This is a diagnostic contrast, not a calibrated membership probability or a test of whether the sample belongs to the cluster. If entries are undefined, the calculation must state how missing pair information is handled; silently treating undefined values as zero biases the comparison.

When the Partition Is Forced

A partition can be returned even when the consensus matrix contains no convincing separation. This occurs, for example, when the analyst requests a fixed number of clusters from a hierarchical procedure. The algorithm must assign samples somewhere, but the existence of an output is not evidence that the data support distinct groups. A diffuse matrix, extensive overlap between within- and between-cluster frequencies, or many intermediate pairwise values should prompt caution.

Practical Checks, Alternatives, and Limitations

Before interpreting a consensus partition, verify that the runs are comparable. The same samples must refer to the same observational units; label alignment is unnecessary for pairwise co-clustering, but consistent sample identity is essential. The perturbation scheme and analysis pipeline should be specified, and preprocessing should be performed in a way consistent with the intended stability assessment. If runs use different feature sets, transformations, or sampling designs, those differences affect what the frequencies summarize. These considerations connect to the earlier material on resampling units, preprocessing within resamples, and co-clustering matrices as stability summaries.

Check coverage as well as values. A pair with \(N_{ij}=1\) and \(C_{ij}=1\) was co-clustered in its single eligible run; that is not equivalent evidence to a pair with \(N_{ij}=100\) and \(C_{ij}=1\). If many entries are undefined because pairs rarely overlap in the runs, a consensus matrix may be too incomplete to support an interpretable partition. Likewise, a hierarchical method may need a complete dissimilarity matrix. Any imputation or restriction to a common set of pairs is an additional analysis choice and should be stated, not hidden.

Consensus clustering differs from choosing one representative run: it uses repeated pairwise relationships rather than allowing one run to stand for all perturbations. It also differs from reporting a global run-to-run agreement score, because the matrix preserves which pairs are consistently together or apart. The earlier material on the adjusted Rand index, cluster-wise Jaccard stability, and variation of information addresses complementary summaries of partition agreement. None of these approaches turns stability into proof of biological validity.

A final limitation is that consensus depends on the runs included. Changing the resampling scheme, feature perturbation, preprocessing, clustering algorithm, or number of clusters can change \(C\). Consensus frequencies should therefore be reported with their design and interpreted as stability under that design. They can identify robust pairwise relationships and flag a poorly separated solution; they cannot establish causality, clinical usefulness, or biological meaning on their own.