AIC, BIC, and ICL for Cluster Selection

Why Cluster Selection Depends on the Goal

A clustering model can fit the observed data well yet leave many observations with ambiguous memberships. A different model might fit slightly less well but assign observations much more decisively. AIC, BIC, and the integrated completed likelihood (ICL) put different emphasis on these features. Understanding that difference matters because selecting a model for describing the data distribution is not always the same task as selecting a model for producing distinct clusters.

Consider a fitted finite mixture with \(K\) components and parameter vector \(\theta\). For observations \(x_1,\ldots,x_n\), let \(\ell(\hat{\theta})\) be its maximized observed-data log-likelihood, and let \(d\) be the number of free parameters under that model. These summaries record fit and nominal complexity. The posterior membership probabilities add information about how clearly the model separates the observations into groups.

Three Criteria and What They Reward

AIC penalizes each additional free parameter by \(2\). Its motivation is prediction: under regularity conditions, the penalty corrects the optimism of evaluating a fitted density on the same data used to estimate it. BIC penalizes each parameter by \(\log n\). Its usual motivation is approximation to a model’s marginal likelihood, with a complexity penalty that grows with sample size. The standard derivations of these interpretations require regularity conditions; finite mixtures can violate them, for example when a component weight approaches zero or covariance parameters become singular.

ICL adds a further penalty for uncertain allocations. If an observation has posterior membership probabilities close to \(0\) and \(1\), its entropy contribution is small. If it has substantial probability of belonging to several components, its contribution is larger. Thus, the entropy term makes ICL favor solutions whose components act more like distinct clusters. The displayed ICL score is a widely used approximation based on BIC and fitted posterior probabilities; it should not be mistaken for an exact identity for every possible definition or computation of integrated completed likelihood.

Worked Score Comparisons

Why the Entropy Penalty Works This Way

To verify the lower bound, each probability lies in \([0,1]\), so \(-\tau_{ik}\log\tau_{ik}\geq0\), with the stated convention at zero. For the upper bound, compare the probability vector for observation \(i\) with the uniform vector. Nonnegativity of Kullback–Leibler divergence gives $$ \sum_{k=1}^K \tau_{ik}\log\left(\frac{\tau_{ik}}{1/K}\right)\geq0. $$ Expanding the left side gives \(\sum_k\tau_{ik}\log\tau_{ik}+\log K\geq0\), or \(H_i\leq\log K\). Equality holds for the uniform vector; the lower bound is attained when one probability is one and all the others are zero. Summing these bounds over observations proves \(0\leq H\leq n\log K\). \(\square\)

This result explains why the size of the ICL adjustment depends on both the number of observations and the number of components. It also clarifies what the adjustment does not measure: entropy is uncertainty in the fitted posterior allocations, not a direct measure of whether the mixture density is adequate or whether the clusters are scientifically meaningful.

When a More Complex Model Wins

The score differences give a direct rule for deciding whether an added component or a less constrained covariance structure is worth its complexity. Let model 1 be more complex than model 0, with parameter-count difference \(\Delta d=d_1-d_0>0\) and log-likelihood gain \(\Delta\ell=\ell_1-\ell_0\). Subtracting the scores yields $$ \operatorname{AIC}_1-\operatorname{AIC}_0=-2\Delta\ell+2\Delta d, $$ $$ \operatorname{BIC}_1-\operatorname{BIC}_0=-2\Delta\ell+\Delta d\log n. $$

Each condition follows by requiring the corresponding score difference to be negative. For AIC, \(-2\Delta\ell+2\Delta d<0\) is equivalent to \(\Delta\ell>\Delta d\). For BIC, \(-2\Delta\ell+\Delta d\log n<0\) is equivalent to \(\Delta\ell>\tfrac12\Delta d\log n\). For ICL, subtracting the two entropy-augmented scores gives $$ -2\Delta\ell+\Delta d\log n+2(H_1-H_0)<0, $$ which rearranges to the stated condition. This derivation shows precisely how the entropy difference changes the likelihood gain required by ICL. If the more complex model reduces entropy, then \(H_1-H_0<0\), and the required gain is lower than the BIC threshold; if its assignments are more uncertain, the required gain is higher. \(\square\)

For example, with \(n=200\) and \(\Delta d=6\), BIC requires a log-likelihood gain exceeding \(\tfrac12(6)\log(200)\approx15.895\). AIC requires only a gain exceeding \(6\). The gain of \(10\) in the first worked example therefore clears the AIC threshold but not the BIC threshold. The difference between the criteria is not a disagreement about the fitted likelihood; it is a difference in the penalty applied to added complexity.

Choosing the Criterion for the Clustering Task

The appropriate criterion depends on what the fitted mixture is meant to provide. AIC is oriented toward predictive density performance. BIC is a stronger complexity penalty and is often used when the aim is to compare candidate statistical models. ICL is useful when the aim includes obtaining clusters with relatively clear memberships, rather than merely finding a mixture density that represents the observed distribution well.

These aims need not agree. A mixture may approximate a skewed or otherwise non-Gaussian density using several overlapping components. Those components can improve density fit while producing uncertain memberships. AIC or BIC may favor that representation, whereas ICL may favor fewer, more separated groups. Conversely, adding a component that produces a meaningful reduction in allocation uncertainty can make ICL favor a more complex solution even when BIC alone does not.

In Gaussian mixture applications, likelihood and criterion values also depend on the fitted model being well-defined. For example, an unconstrained covariance matrix can collapse around an observation, making the likelihood unbounded. Such a fit does not provide a meaningful AIC, BIC, or ICL comparison. Parameter counts must match the constraints actually imposed, and candidate fits should be checked for valid estimates before their scores are compared.

Common Mistake: Treating the Scores as Interchangeable

• Wrong reasoning: “All three criteria are just likelihood with a penalty, so the smallest score identifies the best clustering for every purpose.” Correction: AIC, BIC, and ICL encode different aims and penalties. ICL additionally penalizes uncertain fitted memberships.
• Wrong reasoning: “A lower ICL proves that the clusters are real.” Correction: ICL favors a particular combination of fit, complexity, and allocation certainty. It does not establish that a cluster interpretation is scientifically valid.
• Wrong reasoning: “A larger component count must increase ICL because it increases the maximum possible entropy.” Correction: The actual fitted entropy can increase or decrease. Compare the observed \(H\) values, not just their upper bound.
• Wrong reasoning: “The standard AIC and BIC penalties have exact classical interpretations in every mixture.” Correction: Mixture models can violate regularity assumptions behind their usual derivations. Treat the criteria as useful selection tools, while checking fit stability and the modeling assumptions.