Allocation Entropy as a Clustering Diagnostic

Why Measure Allocation Uncertainty?

A fitted mixture model gives each observation a posterior probability of belonging to each component. The largest of these probabilities determines a maximum-posterior assignment, but that label alone does not say how decisive the assignment was. An observation with probabilities \(0.51\) and \(0.49\) receives the same assigned label as one with probabilities \(0.99\) and \(0.01\), even though the evidence for the two labels is very different.

Allocation entropy summarizes the full posterior probability vector for one observation. It is small when nearly all posterior probability is concentrated on one component and large when probability is spread across several components. Its value can be examined observation by observation and then summarized within each fitted cluster. It is a diagnostic of classification uncertainty, not by itself a test of whether the clustering model is adequate or whether its components represent meaningful groups.

Definition and Scale

Suppose a fitted model has \(G\) components. For observation \(i\), let \(\tau_{ig}\) be its posterior probability of allocation to component \(g\), conditional on its observed data and the fitted model. Thus \(\tau_{ig}\geq 0\) and \(\sum_{g=1}^G\tau_{ig}=1\). We use the convention \(0\log 0=0\) and natural logarithms.

The unnormalized scale runs from \(0\) to \(\log G\). Normalization places the value between \(0\) and \(1\), which can make results easier to compare across models with different numbers of components. A normalized entropy is still a measure of uncertainty in the fitted posterior allocations; it does not remove differences in model specification or data.

Bounds and Their Meaning

Proof. Since \(0\leq p_g\leq1\), each nonzero term \(-p_g\log p_g\) is nonnegative. Hence \(H\geq0\). A term is zero for \(p_g=1\), and for \(p_g=0\) by convention. The sum is zero exactly when every positive probability equals \(1\), which is precisely a point mass.

For the upper bound, let \(S=\{g:p_g>0\}\), and compare \(p\) with the uniform probability vector \(q_g=1/G\). The inequality \(\log x\leq x-1\) for \(x>0\) gives $$ \sum_{g\in S}p_g\log\left(\frac{q_g}{p_g}\right) \leq \sum_{g\in S}p_g\left(\frac{q_g}{p_g}-1\right) =\frac{|S|}{G}-1\leq0. $$ Therefore \(\sum_{g\in S}p_g\log(p_g/q_g)\geq0\). Since \(q_g=1/G\), this is \(\log G-H\geq0\), so \(H\leq\log G\). Equality requires equality in both inequalities: all \(G\) probabilities must be positive, and \(q_g/p_g=1\) for every \(g\). Thus \(p_g=1/G\) for all \(g\). This proves the bounds and equality cases. \(\square\)

Summarizing Entropy Within Clusters

Observation-level entropies can be inspected directly or summarized for each component. A useful posterior-weighted summary assigns each observation's entropy fractionally to every component in proportion to its posterior membership. Define the fitted posterior weight of component \(g\) by \(\hat{\pi}_g=n^{-1}\sum_{i=1}^n\tau_{ig}\), and its posterior-weighted mean entropy by $$ \bar e_g=\frac{\sum_{i=1}^n\tau_{ig}e_i}{\sum_{i=1}^n\tau_{ig}}, $$ provided \(\sum_i\tau_{ig}>0\). These are descriptive summaries of the fitted allocations. The weighting avoids making a hard assignment before summarizing uncertainty; an observation can contribute to more than one component.

One may also report the mean entropy among observations assigned to component \(g\) by the maximum-posterior rule. That summary answers a different question: it describes uncertainty in the subset given a particular hard label. State which convention is being used, especially when observations near assignment boundaries are important.

Derivation. By definition, \(\hat{\pi}_g=(1/n)\sum_i\tau_{ig}\), so $$ \sum_{g=1}^G\hat{\pi}_g\bar e_g =\sum_{g=1}^G\left(\frac{1}{n}\sum_{i=1}^n\tau_{ig}\right) \frac{\sum_{j=1}^n\tau_{jg}e_j}{\sum_{j=1}^n\tau_{jg}} =\frac{1}{n}\sum_{g=1}^G\sum_{j=1}^n\tau_{jg}e_j =\frac{1}{n}\sum_{j=1}^n e_j\sum_{g=1}^G\tau_{jg} =\frac{1}{n}\sum_{j=1}^n e_j. $$ The final equality uses \(\sum_g\tau_{jg}=1\) for every observation \(j\). \(\square\)

Reading the Distribution, Not Just the Mean

A mean entropy can hide the pattern that matters for classification. A modest overall mean might arise because almost all observations have decisive allocations while a small group is highly uncertain. Conversely, a moderately high mean could reflect diffuse uncertainty across most observations. Examine the range or quantiles, a histogram or other distributional display, and the identities or characteristics of observations with the largest entropies. Within-cluster summaries can help locate where uncertainty is concentrated.

For a fitted model, the distribution of \(e_i\) describes the observed pattern of uncertainty under that model. A high value signals that the posterior allocation is spread across components for that observation; a low value signals concentration. Neither value alone establishes that the fitted component assignments are correct. The calculation is conditional on the fitted model and its estimated parameters.

Common Mistakes

• Interpreting the maximum-posterior label as a certainty measure. Two observations can have the same assigned label but very different entropies. Retain the full posterior vector when quantifying uncertainty.
• Comparing raw entropies across different values of \(G\) without context. The maximum is \(\log G\), so report the number of components and consider normalized entropy when a relative scale is useful.
• Calling a high entropy a wrong assignment. Entropy indicates diffuse fitted allocation probabilities, not whether a label is objectively incorrect or whether the model fits well.
• Taking entropy after averaging posterior probabilities. Entropy of the average vector is generally not the average observation entropy; the two summaries answer different questions.
• Reporting a cluster summary without defining its membership rule. Posterior-weighted and hard-assignment summaries need not agree, so specify the weighting convention.