Stability of Cluster-Derived Predictions

Why Stability Requires Two Checks

A clustering method can produce different assignments when its initialization changes, when the training sample changes, or when a small number of observations are added or removed. If cluster membership is then used to create a predictor, those changes may carry through to the predictions. But different cluster labels do not automatically mean the clusters changed: labels such as “1” and “2” are arbitrary names. Stability therefore has two distinct questions: did the partition change, and did predictions change materially?

This distinction matters in actuarial work. A hypothetical insurer might use clusters of policyholders as a feature in a model of expected claim cost. If a rerun exchanges the labels “Cluster 1” and “Cluster 2” but assigns the same policyholders together, the partition is unchanged. If the model is refitted consistently, predictions need not change either. In contrast, modest movement in cluster boundaries might alter predictions for records near those boundaries, even when most assignments remain stable.

Definition: Partition stability describes how consistently records are grouped together across clustering runs. Predictive stability describes how consistently the downstream model predicts for the same records across those runs. A stable partition does not guarantee stable predictions, and an unstable partition does not necessarily imply materially unstable predictions.

Comparing Partitions Without Relying on Label Names

For two clustering runs on the same $n$ records, form an overlap table. Let $n_{ij}$ be the number of records assigned to cluster $i$ in the first run and cluster $j$ in the second. Let $a_i=\sum_j n_{ij}$ be row totals and $b_j=\sum_i n_{ij}$ be column totals. The quantity $C$ counts record pairs grouped together in both runs; $A$ and $B$ count pairs grouped together in the first and second runs, respectively.

C=\sum_{i,j}\binom{n_{ij}}{2},\qquad A=\sum_i\binom{a_i}{2},\qquad B=\sum_j\binom{b_j}{2},\qquad T=\binom{n}{2}.

The adjusted Rand index (ARI) compares the observed shared-pair count with a chance reference based on the two partitions’ margins:

\operatorname{ARI} = \frac{C-\dfrac{AB}{T}} {\dfrac{A+B}{2}-\dfrac{AB}{T}}.

The adjustment $AB/T$ is the expected value of $C$ when partitions are randomized subject to their fixed cluster sizes. The denominator uses $(A+B)/2$ as the normalization reference. This average is not necessarily the largest attainable value of $C$; because $C\leq\min(A,B)$, the average can even exceed every attainable shared-pair count when $A\ne B$. The ARI is 1 for identical partitions, is near 0 at its chance reference, and can be negative when agreement is below that reference. It is unchanged by relabeling clusters.

Condition: The displayed ARI formula requires $T>0$ and a nonzero denominator. If there are too few records or both partitions have degenerate pair counts that make the denominator zero, the formula is undefined; report that the comparison is degenerate rather than silently assigning a value.

Why the Chance Adjustment Has This Form

There are $T=\binom{n}{2}$ unordered record pairs. In the first partition, exactly $A$ pairs are together. In the second, exactly $B$ pairs are together. Under random rearrangement of the second partition with its cluster sizes fixed, each pair has probability $B/T$ of being together in that partition: all pairs are exchangeable under this randomization, and exactly $B$ of the $T$ pairs are together in every arrangement.

Consider only the $A$ pairs already together in the first partition. For each such pair, the probability that it is also together in the randomized second partition is $B/T$. Summing the expected indicator over those $A$ pairs gives:

E[C]=\sum_{r=1}^{A}P(\text{pair }r\text{ is together in the second partition}) =A\frac{B}{T}=\frac{AB}{T}.

Thus $C-AB/T$ measures shared grouping beyond this fixed-margin chance reference. The denominator scales that excess using the average pair total from the two partitions, after subtracting the same reference. This construction explains why the ARI is not simply the fraction of matching labels or the fraction of pairs that agree.

Worked Example: Calculating Agreement From an Overlap Table

A hypothetical analyst clusters 10 commercial vehicles using two different initializations. The overlap table has entries $\begin{pmatrix}4&1\\1&4\end{pmatrix}$. Calculate the ARI and interpret it.

Method. Count pairs in shared cells, then calculate the pair totals from each partition’s margins. This avoids treating arbitrary cluster names as meaningful.

Step 1: Margins. Each row totals $4+1=5$, and each column totals $4+1=5$. Thus $a_1=a_2=b_1=b_2=5$, and $n=10$.

Step 2: Shared pairs. The two diagonal cells each contribute $\binom{4}{2}=6$; the off-diagonal cells each contribute $\binom{1}{2}=0$. Therefore $C=6+0+0+6=12$.

Step 3: Pair totals. $A=\binom{5}{2}+\binom{5}{2}=10+10=20$, and likewise $B=20$. Also, $T=\binom{10}{2}=45$.

Step 4: Chance reference and index. $AB/T=(20)(20)/45=400/45=80/9$. The numerator is $12-80/9=108/9-80/9=28/9$. The denominator is $20-80/9=180/9-80/9=100/9$. Hence $\operatorname{ARI}=(28/9)/(100/9)=0.28$.

Check and interpretation. The value is between 0 and 1, and it indicates some agreement beyond the fixed-margin chance reference, but far from identical partitions. It does not tell us whether predictions based on these clusters are stable.

Compare Predictions on the Same Records

To assess predictive stability, refit the clustering-and-prediction pipeline under a specified perturbation, then predict for a common set of records. The common records are essential: comparing predictions on different sets confounds pipeline changes with changes in the records being evaluated. For a numeric prediction, write $\hat{y}^{(r)}_q$ for the prediction on record $q$ from run $r$. For two runs, useful summaries include the mean absolute prediction difference and the root mean square prediction difference:

\operatorname{MAD}_{\mathrm{pred}} =\frac{1}{m}\sum_{q=1}^{m} \left|\hat{y}^{(1)}_q-\hat{y}^{(2)}_q\right|, \qquad \operatorname{RMSD}_{\mathrm{pred}} =\sqrt{\frac{1}{m}\sum_{q=1}^{m} \left(\hat{y}^{(1)}_q-\hat{y}^{(2)}_q\right)^2}.

Here $m$ is the number of common evaluation records. The mean absolute difference is easy to interpret in the prediction’s units; the root mean square difference gives larger discrepancies more influence. Also report a maximum absolute difference or the fraction exceeding a decision-relevant threshold when isolated large changes matter. For predicted probabilities, apply the same comparisons to probabilities and interpret the result in probability units.

A stability diagnostic is not a replacement for predictive validation. It describes sensitivity to specified perturbations; it does not establish that predictions are accurate for future observations. The records used for the comparison should be handled consistently with the modeling design, and transformations and cluster mappings should be estimated from the relevant training data rather than from evaluation outcomes.

Worked Example: Separating Label Switching From a Changed Partition

A model assigns hypothetical repair shops to two clusters and uses the cluster category in a prediction model. In run 1, four shops have assignments $A,A,B,B$, with fitted predictions $\$100,\$100,\$300,\$300$. In run 2, the assignments are recorded as $2,2,1,1$, and the refitted category-specific predictions are $\$100,\$100,\$300,\$300$. Determine what changed.

Method. Compare the grouping of records, not the literal labels, and then compare paired predictions.

Step 1: Check grouping. In run 1, the first two shops are together and the last two are together. In run 2, labels 2 and 1 identify those same two groups, respectively. The partition is identical; only its names have been exchanged. A label-based match rate would misleadingly report zero matching labels.

Step 2: Check predictions. The paired differences are $100-100=0$, $100-100=0$, $300-300=0$, and $300-300=0$ dollars. Thus the mean absolute difference and root mean square difference are both $\$0.00$, and the maximum absolute difference is also $\$0.00$.

Check and interpretation. Relabeling alone has not changed either the partition or the predictions. If the category-to-prediction mapping had not been refitted consistently, the apparent prediction change would be an implementation error, not evidence of clustering instability.

Worked Example: Measuring Prediction Change After a Modest Perturbation

A hypothetical claim-cost model is refitted after one training record is removed. Two runs produce the following predictions, in dollars, for the same six policyholders. Their cluster assignments agree for five of the six records.

Run 1: $120, 140, 160, 300, 320, 340$.
Run 2: $125, 145, 170, 295, 325, 350$.

Method. Calculate paired prediction differences on the fixed evaluation records, then summarize their magnitude. Assignment agreement and prediction stability are separate results.

Step 1: Absolute differences. The absolute differences are $|120-125|=5$, $|140-145|=5$, $|160-170|=10$, $|300-295|=5$, $|320-325|=5$, and $|340-350|=10$ dollars. Their sum is $5+5+10+5+5+10=40$.

Step 2: Mean absolute difference. There are six records, so $\operatorname{MAD}_{\mathrm{pred}}=40/6=\$6.666\ldots$, or $\$6.67$ to the nearest cent.

Step 3: Root mean square difference. The squared differences sum to $25+25+100+25+25+100=300$. Therefore $\operatorname{RMSD}_{\mathrm{pred}}=\sqrt{300/6}=\sqrt{50}=\$7.071\ldots$, or $\$7.07$ to the nearest cent. The maximum absolute difference is $\$10.00$.

Check and interpretation. Five of six assignments agree, and all prediction differences are at most $\$10.00$. Whether that is materially stable depends on the intended use and an explicitly chosen tolerance; the calculations alone do not establish an acceptable threshold.

Designing a Useful Stability Assessment

A useful assessment identifies what is being perturbed and keeps other sources of variation controlled where possible. Different initializations test sensitivity to the clustering algorithm’s starting values. Resampling or modest changes to training records test sensitivity to the available data. If the clustering and downstream model are both stochastic, repeating both may measure end-to-end sensitivity; repeating one component while holding the other fixed can help diagnose where the instability arises.

Specify the perturbation.
State whether runs differ by initialization, training sample, or a defined data change. Avoid describing an unspecified mixture of changes as a single stability result.

Refit the intended pipeline.
When assessing end-to-end behavior, refit the clustering and downstream model in each run using that run’s training data. Apply each fitted mapping consistently to the same evaluation records.

Compare partitions and predictions separately.
Use a label-invariant partition measure such as ARI, then calculate paired prediction differences. Do not use one result as a substitute for the other.

Interpret against the application.
Report the perturbation, the comparison population, the metrics, and a justified materiality threshold. A small average difference can coexist with a consequential change for a small subset.

For repeated runs, summarize the distribution of pairwise ARI values and prediction differences rather than reporting only the most favorable run. If cluster labels need to be described across runs, align them for presentation using their overlap; the ARI itself does not require that alignment. If the number or sizes of clusters change, report those changes as well, since a single summary index cannot explain which records or groups drove the result.

Common Mistakes and Exam Trap

Trap: treating the average pair total as the maximum shared-pair count. Consider an overlap table $\begin{pmatrix}2&1\\2&1\end{pmatrix}$ for six records. The row totals are $3,3$, the column totals are $4,2$, so $A=6$, $B=7$, and $T=15$. The shared-pair count is $C=1+0+1+0=2$, and $AB/T=42/15=2.8$. The ARI is:

\operatorname{ARI} =\frac{2-2.8}{(6+7)/2-2.8} =\frac{-0.8}{6.5-2.8} =\frac{-0.8}{3.7} \approx -0.2162.

A mistaken approach might call $(A+B)/2=6.5$ the largest attainable $C$. But $C\leq\min(A,B)=6$, so 6.5 cannot be attained here. The average is the ARI’s normalization reference, not a claimed maximum. This example also shows that a negative ARI is possible.

Comparing numeric labels directly. Cluster numbers are arbitrary. Use a label-invariant partition measure or compare which records are grouped together.
Concluding prediction instability from assignment changes alone. A changed assignment can have little or no effect on the downstream model’s predictions.
Reporting only an average prediction difference. Also inspect a maximum, a distribution, or the share exceeding a relevant threshold.
Changing the evaluation records between runs. Paired differences require predictions for the same records.

Key takeaway: A complete stability assessment reports both how the partition changes and how predictions change on common records. Interpret each measure in context, and distinguish arbitrary label switching from changes that matter for the intended prediction use.

Check Your Understanding

For each problem, calculate the requested quantity and keep partition agreement distinct from prediction agreement.

Two eight-record partitions have overlap table $\begin{pmatrix}3&1\\1&3\end{pmatrix}$. Find the row and column totals, $C,A,B,T$, the chance reference $AB/T$, and the ARI. (Answer: Row and column totals are $4,4$; $C=6$, $A=B=12$, $T=28$, $AB/T=36/7\approx5.1429$, and $\operatorname{ARI}=0.1250$.)
In one run, six records have cluster labels $A,A,A,B,B,B$; in another, the same records have labels $2,2,2,1,1,1$. The second run refits the category mapping, and paired predictions are identical. State the partition relationship and prediction difference. (Answer: The partitions are identical up to relabeling, so $\operatorname{ARI}=1$; all paired prediction differences are zero.)
For four common records, numeric predictions are $80,100,120,200$ in one run and $82,95,126,197$ in another. Calculate the mean absolute and root mean square prediction differences. (Answer: Absolute differences are $2,5,6,3$; mean absolute difference $=4$, and root mean square difference $=\sqrt{18.5}\approx4.3012$, in the prediction’s units.)
For the overlap table $\begin{pmatrix}2&1\\2&1\end{pmatrix}$ on six records, calculate the margins, shared-pair count, chance reference, and ARI. Explain why the ARI normalization reference is not the maximum possible shared-pair count. (Answer: Row totals $3,3$, column totals $4,2$, $C=2$, $A=6$, $B=7$, $T=15$, $AB/T=2.8$, and $\operatorname{ARI}\approx-0.2162$; $(A+B)/2=6.5$ exceeds the bound $C\leq\min(A,B)=6$.)

What You'll Learn