Bootstrap Stability of Model-Based Clusters

Why assess cluster stability?

A fitted clustering can look clear in one data set and change substantially when a few cases are perturbed. This matters because the component labels are estimated rather than observed: a cluster that appears only under a small change in the cases may be a fragile feature of the fitted analysis. Bootstrap stability asks a focused question: if the observed cases were perturbed by case resampling and the same clustering model were refitted, would the resulting assignments be similar?

Here the model is a selected Gaussian mixture with a fixed number \(K\) of components and a fixed covariance constraint or covariance family. The stability exercise holds those choices fixed so that it assesses variation in the fitted clusters, not variation in model selection. It is a sensitivity analysis under resampling, not a test that clusters are real or useful.

Definition and resampling protocol

Let the observed vectors be \(x_1,\ldots,x_n\), and let the selected model have \(K\) Gaussian components. Its fitted density has the form

where the covariance matrices obey the selected covariance structure. A reference fit on the original data gives a hard assignment \(z_i^{(0)}\) for each case, usually the component with largest fitted posterior membership probability. For bootstrap replicate \(b\), draw \(n\) indices independently with replacement, refit the same model, and evaluate its fitted posterior memberships on every original case \(x_i\). This gives bootstrap-fit labels \(z_i^{(b)}\) on the common evaluation set, even for cases absent from that replicate's resample.

This common set is important. Bootstrap samples contain repeated rows and omit some original rows; comparing label sequences in resample order does not compare the same cases. Evaluating each fit on the original cases avoids that mismatch. The reference fit and bootstrap fit may give the same partition but use different numerical labels, so labels must also be aligned.

Matching labels across fits

Mixture-component labels have no intrinsic order. A fit that calls a particular group “component 1” and another fit that calls it “component 3” may represent the same partition. Matching removes this arbitrary relabeling before any agreement score is calculated.

For one replicate, define the \(K\)-by-\(K\) overlap table \(N\) by

Rows refer to reference clusters and columns to bootstrap-fit clusters. A permutation \(\pi\) maps each reference label \(k\) to one bootstrap label \(\pi(k)\). Choose the permutation maximizing the number of cases whose labels agree after this mapping. This is a linear assignment problem; for moderate \(K\), the Hungarian algorithm finds the optimum efficiently.

To verify the result, fix any permutation \(\pi\). A case contributes one to the agreement count precisely when its reference label is \(k\) and its bootstrap label is \(\pi(k)\) for some \(k\). Those cases are exactly the cells \(N_{k,\pi(k)}\), so the agreement count is \(\sum_k N_{k,\pi(k)}\). Every relabeling is a permutation, and maximizing this count over all permutations therefore gives the best possible agreement. Dividing by \(n\) gives the fraction of cases that agree. \(\square\)

Cluster-specific stability

Overall matched agreement can conceal an unstable small cluster: many cases in large clusters may agree even when the small cluster changes substantially. A cluster-specific measure compares the cases in one reference cluster with the cases assigned to its matched bootstrap cluster. For reference cluster \(k\), let \(A_k\) be its case set and let \(B_{\pi(k)}\) be the matched bootstrap cluster's case set, both on the common evaluation data. Their Jaccard similarity is

The numerator counts cases assigned to both matched clusters. The union counts cases assigned to at least one of them, so \(J_k\) is between zero and one; a value of one means the two case sets are identical. Averaging this score over bootstrap replicates gives a useful summary for each reference cluster. It should be accompanied by cluster sizes: a score based on a very small cluster can be especially sensitive to individual cases.

What case resampling does to the likelihood

A bootstrap sample can be represented by its case multiplicities. Let \(W_i^{(b)}\) be the number of times original case \(i\) is drawn in replicate \(b\). These counts are nonnegative integers and satisfy \(\sum_i W_i^{(b)}=n\). For parameter \(\theta\), the observed-data log-likelihood of a Gaussian mixture on that resample is

This follows directly from the mixture density: each occurrence of case \(x_i\) contributes its log density once, and \(W_i^{(b)}\) occurrences contribute that term \(W_i^{(b)}\) times. Summing over distinct observed cases gives the displayed expression. Thus fitting to a case-resampled data set is equivalent, for the likelihood, to fitting the original rows with bootstrap multiplicities as weights. Cases with multiplicity zero do not contribute to that replicate's fitting objective, but their assignments can still be evaluated afterward under its fitted model.

Summarizing replicates and interpreting results

Repeat the procedure for \(B\) bootstrap replicates. A practical report includes the distribution or mean and spread of matched agreement, the distribution of Jaccard similarity for each reference cluster, and the number of successful fits. For example, if four replicates give matched agreements \(0.90,\ 0.85,\ 0.80,\ 0.95\), their mean is \((0.90+0.85+0.80+0.95)/4=3.50/4=0.875\). The range, 0.80 to 0.95, also communicates that stability varies across resamples.

This summary describes sensitivity of the fitted partition to case resampling, conditional on the chosen model and fitting procedure. It is not automatically a confidence interval for a population quantity: finite-mixture estimation can be nonregular, and stability scores are themselves functions of fitted labels. Nor does high stability establish that the clusters are scientifically meaningful. A misspecified model can repeatedly produce the same partition.

Fitting details are part of the sensitivity analysis. Mixture likelihoods can have local optima, so a replicate that converges to a different solution because of initialization may appear unstable even when the data perturbation is not the cause. Use a consistent, sufficiently robust initialization policy; record convergence failures; and do not silently discard failed replicates. Report whether the same preprocessing and constraints were used in every fit.

This conditional procedure keeps \(K\) and the covariance family fixed. If the goal is instead to assess the entire analysis pipeline, model selection must also be repeated within each resample, and the number of components may change. That question requires explicit handling of unmatched, split, or merged clusters; the equal-\(K\) permutation score defined here is not adequate by itself.

Common pitfall