Choosing Component Count with AIC, BIC, and ICL

Why Component-Count Criteria Can Disagree

A Gaussian mixture with more components can usually fit the observed data at least as well as a smaller mixture, but improved fit alone does not determine how many components to retain. A selection criterion balances fit against a complexity penalty. For mixtures, the choice also depends on the purpose: describing the data distribution and obtaining well-separated groups are not identical goals.

AIC and BIC penalize the number of fitted parameters, with BIC’s penalty increasing with sample size. The integrated completed likelihood (ICL) adds a further penalty when observations have uncertain component assignments. Thus, ICL may prefer fewer components even when AIC or BIC prefers a model with greater observed-data likelihood.

Definition: For a candidate model with maximized log-likelihood \(\ell(\hat{\theta})\), sample size \(n\), and \(p\) free parameters, the minimization forms of the criteria are

\operatorname{AIC}=-2\ell(\hat{\theta})+2p, \qquad \operatorname{BIC}=-2\ell(\hat{\theta})+p\log n.

The commonly used soft-responsibility ICL approximation is

\operatorname{ICL} = -2\ell(\hat{\theta})+p\log n+2H, \qquad H=-\sum_{i=1}^{n}\sum_{k=1}^{K}\tau_{ik}\log\tau_{ik},

where \(\tau_{ik}\) is the fitted posterior responsibility of component \(k\) for observation \(i\), and \(0\log 0\) is defined as \(0\). Smaller criterion values are preferred.

The entropy \(H\) is nonnegative: each observation contributes zero when its responsibility is entirely concentrated on one component, and a larger amount when responsibility is spread across components. The ICL expression above is a standard approximation, not a claim that every implementation computes an identical exact integrated completed likelihood. Implementations may use hard assignments or other approximations, so the convention should be reported when results are presented.

Counting Parameters and Fitting the Candidate Models

For a \(d\)-dimensional Gaussian mixture with \(K\) components, freely estimated mixing proportions contribute \(K-1\) parameters because they sum to one. Component means contribute \(Kd\) parameters. If each component has its own unrestricted positive-definite covariance matrix, the covariance contribution is \(K d(d+1)/2\), since a symmetric \(d\times d\) matrix has \(d(d+1)/2\) distinct entries. The total is therefore

p_K=(K-1)+Kd+K\frac{d(d+1)}{2}.

This count changes when parameters are shared or constrained. Count the free parameters of the model actually fitted; do not count a covariance entry as free if the fitting model fixes it or ties it to another component. Also, use the same observations and likelihood convention across candidate models.

A practical issue is especially important for Gaussian mixtures: with freely varying component covariance matrices, the likelihood can be unbounded. A component can concentrate around one observation as its covariance shrinks. In that setting, there is no finite global maximum log-likelihood to insert into AIC or BIC. A calculation based on finite values must therefore identify a constraint or regularization, or describe the values as local-fit results rather than unrestricted global maxima.

Worked Example: AIC and BIC From Constrained Fits

Suppose \(n=200\) observations in two dimensions are fitted with \(K=1,2,3\) mixtures. Each component has its own covariance matrix, and the fitting procedure constrains covariance eigenvalues to lie between \(0.05\) and \(100\), with means restricted to a sufficiently broad compact region. Suppose the resulting constrained maximized log-likelihoods are \(-620\), \(-580\), and \(-575\), respectively. These are constrained-fit maxima, not unrestricted global maxima.

In two dimensions, each component has two mean parameters and three covariance parameters, so \(p_K=(K-1)+5K=6K-1\). Thus \(p_1=5\), \(p_2=11\), and \(p_3=17\). For AIC, the three values are

\begin{aligned} \operatorname{AIC}_1&=-2(-620)+2(5)=1240+10=1250,\\ \operatorname{AIC}_2&=-2(-580)+2(11)=1160+22=1182,\\ \operatorname{AIC}_3&=-2(-575)+2(17)=1150+34=1184. \end{aligned}

AIC selects \(K=2\), since \(1182\) is the smallest value. Since \(\log(200)\approx5.2983\), BIC gives

\begin{aligned} \operatorname{BIC}_1&=1240+5\log(200)\approx1266.49,\\ \operatorname{BIC}_2&=1160+11\log(200)\approx1218.28,\\ \operatorname{BIC}_3&=1150+17\log(200)\approx1240.07. \end{aligned}

BIC also selects \(K=2\). The fit improves from \(K=2\) to \(K=3\), but the increase in parameter penalty is greater than the improvement in fit for both criteria.

Why ICL Penalizes Assignment Uncertainty

For an observation \(x_i\), let \(a_{ik}=\hat{\pi}_k f_k(x_i;\hat{\theta}_k)\), where \(\hat{\pi}_k\) is the fitted mixing proportion and \(f_k\) is the component density. The observed-data contribution to the log-likelihood is \(\log\sum_k a_{ik}\), while the responsibility is \(\tau_{ik}=a_{ik}/\sum_j a_{ij}\). The following identity explains the entropy term in the soft ICL approximation.

Result: At the fitted responsibilities, the responsibility-weighted complete-data log-likelihood contribution is the observed-data log-likelihood contribution minus the assignment entropy:

\sum_{k=1}^{K}\tau_{ik}\log a_{ik} = \log\left(\sum_{k=1}^{K}a_{ik}\right) + \sum_{k=1}^{K}\tau_{ik}\log\tau_{ik}.

Proof. Because \(\tau_{ik}=a_{ik}/\sum_j a_{ij}\), we have \(a_{ik}=\tau_{ik}\sum_j a_{ij}\). Taking logarithms gives \(\log a_{ik}=\log\tau_{ik}+\log\sum_j a_{ij}\). Multiply by \(\tau_{ik}\), sum over \(k\), and use \(\sum_k\tau_{ik}=1\):

\sum_k\tau_{ik}\log a_{ik} = \sum_k\tau_{ik}\log\tau_{ik} + \left(\sum_k\tau_{ik}\right)\log\left(\sum_j a_{ij}\right) = \sum_k\tau_{ik}\log\tau_{ik} + \log\left(\sum_j a_{ij}\right).

Summing over observations, the expected complete-data log-likelihood is \(\ell(\hat{\theta})-H\). Applying the BIC-style penalty to this lower, classification-oriented fit gives the stated soft ICL approximation: its negative twice-log-likelihood part contributes \(-2\ell(\hat{\theta})+2H\), in addition to the parameter penalty \(p\log n\). \(\square\)

The derivation also clarifies the interpretation. ICL does not merely ask whether the mixture describes the data well. It favors models for which observations can be assigned clearly to components. This can be suitable when the components are intended to define groups, but it can be too conservative if the goal is to represent a density whose overlapping components are scientifically meaningful.

Worked Example: Entropy Changes the Selected Order

Suppose \(n=60\) one-dimensional observations are fitted with \(K=2\) and \(K=3\) mixtures. Variances are constrained away from zero, and the fitted log-likelihoods are \(-100\) and \(-93\). The total responsibility entropies are \(H_2=1.95\) and \(H_3=8.00\), computed as \(-\sum_{i,k}\tau_{ik}\log\tau_{ik}\). In one dimension with component-specific variances, \(p_K=(K-1)+2K=3K-1\), so \(p_2=5\) and \(p_3=8\). Also, \(\log(60)\approx4.0943\).

AIC gives \(\operatorname{AIC}_2=200+10=210\) and \(\operatorname{AIC}_3=186+16=202\), so it selects three components. BIC gives

\operatorname{BIC}_2=200+5(4.0943)\approx220.47, \qquad \operatorname{BIC}_3=186+8(4.0943)\approx218.75.

BIC also selects three components. For ICL, add twice the entropy:

\operatorname{ICL}_2\approx220.47+2(1.95)=224.37, \qquad \operatorname{ICL}_3\approx218.75+2(8.00)=234.75.

ICL selects two components. The three-component fit has better observed-data likelihood, but its assignments are sufficiently more uncertain that the added entropy penalty reverses the choice.

Comparing the Penalties Directly

The criteria’s different penalties can be made explicit when comparing two fitted models. Write \(\Delta\ell=\ell_{\mathrm{new}}-\ell_{\mathrm{old}}\), \(\Delta p=p_{\mathrm{new}}-p_{\mathrm{old}}\), and \(\Delta H=H_{\mathrm{new}}-H_{\mathrm{old}}\). Subtracting the old criterion from the new one gives the exact comparison rules below for the formulas defined here.

Result: The new model has a smaller criterion precisely when

\begin{aligned} \text{AIC:}\quad &2\Delta\ell>2\Delta p,\\ \text{BIC:}\quad &2\Delta\ell>(\log n)\Delta p,\\ \text{ICL:}\quad &2\Delta\ell>(\log n)\Delta p+2\Delta H. \end{aligned}

Proof. For AIC, subtracting gives \(\operatorname{AIC}_{\mathrm{new}}-\operatorname{AIC}_{\mathrm{old}}=-2\Delta\ell+2\Delta p\). This difference is negative exactly when \(2\Delta\ell>2\Delta p\). For BIC the difference is \(-2\Delta\ell+(\log n)\Delta p\), which is negative exactly when \(2\Delta\ell>(\log n)\Delta p\). For ICL, the difference is \(-2\Delta\ell+(\log n)\Delta p+2\Delta H\), and it is negative exactly when \(2\Delta\ell>(\log n)\Delta p+2\Delta H\). \(\square\)

Worked Example: The Cost of Adding One Component

Suppose \(n=500\) two-dimensional observations are fitted using the same covariance constraints as in the first example. Moving from \(K=2\) to \(K=3\) adds six parameters because \(p_K=6K-1\), so \(\Delta p=6\). Assume the constrained log-likelihood improves by \(\Delta\ell=8\), and total responsibility entropy rises by \(\Delta H=3.5\).

For AIC, the improvement threshold is \(2\Delta p=12\), while the likelihood gain on the doubled scale is \(2\Delta\ell=16\). Since \(16>12\), AIC prefers three components. For BIC, the threshold is \(6\log(500)\approx6(6.2146)=37.29\); since \(16<37.29\), BIC prefers two. For ICL the threshold is \(37.29+2(3.5)=44.29\), so ICL also prefers two. The increased assignment uncertainty makes ICL’s case for retaining the smaller model stronger still.

Interpretation, Limitations, and a Common Mistake

AIC is commonly motivated as an estimate of predictive information loss and uses a fixed penalty of two per parameter. BIC uses a penalty that grows as \(\log n\), and in regular, identifiable models it arises as a large-sample approximation to a log marginal likelihood. Those standard justifications require care for Gaussian mixtures. Mixture models have label symmetries and singularities, and the likelihood can be unbounded without constraints. Consequently, regular-model AIC and BIC theory does not automatically give exact risk or evidence interpretations for every mixture fit, nor does ordinary BIC’s familiar consistency claim automatically apply. Treat these criteria as selection tools whose assumptions and fitting procedure matter, not as guarantees.

Common mistake: “The three-component model has a higher log-likelihood, so it must be selected.” The correction is that likelihood alone does not compare model complexity or assignment certainty. Calculate the parameter penalty for AIC or BIC, and, when classification certainty is relevant, calculate the entropy term used by ICL. Also verify that every likelihood value is a legitimate finite result from the stated fitting procedure; unrestricted global maxima may not exist for Gaussian mixtures.

Key takeaway: AIC emphasizes fit with a fixed parameter penalty, BIC applies a sample-size-dependent parameter penalty, and ICL adds a penalty for uncertain assignments. State the covariance and likelihood constraints, parameter count, and ICL convention when reporting a selected mixture order.

Check Your Understanding

Use the definitions and comparison rules above to answer the following.

For an unrestricted-covariance Gaussian mixture in dimension \(d\), explain the contributions to \(p_K\) from weights, means, and covariance matrices.
Why can’t finite unrestricted global maximum log-likelihood values always be assumed for Gaussian-mixture AIC or BIC calculations?
If two fitted models have equal parameter counts and the new model has higher log-likelihood, how do AIC and BIC compare them?
What does a large total responsibility entropy indicate, and how does it affect the soft ICL approximation?
Give one reason BIC’s regular-model interpretation requires caution when applied to Gaussian mixtures.