Using BIC to Select Cluster and Covariance Models

Why BIC compares both fit and complexity

A Gaussian clustering analysis can produce several plausible fitted models: different numbers of clusters, and different restrictions on cluster covariance matrices. A model with more freedom can often attain a larger maximized likelihood, but that improvement alone does not establish that the extra flexibility is worthwhile. The Bayesian information criterion (BIC) provides a common score that balances fit against the number of estimated parameters.

The goal here is to calculate and interpret BIC across these candidates. The central comparison is not simply “which model fits best?” but “does its improvement in fit compensate for its additional estimated parameters?” BIC is a selection criterion, not a test that certifies a clustering as true.

Definition: For a fitted model with maximized log-likelihood $\ell(\hat{\theta})$, $p$ freely estimated parameters, and $n$ observations, the Bayesian information criterion is $$ \operatorname{BIC}=-2\ell(\hat{\theta})+p\log n. $$ When comparing candidates fitted to the same observations using the same likelihood convention, the model with the smaller BIC is preferred by this criterion.

The first term rewards fit: larger log-likelihood makes $-2\ell(\hat{\theta})$ smaller. The second term penalizes complexity, and its size depends on both the parameter count and sample size. The parameter count must describe the model actually fitted, including its covariance constraints; it is not enough to count only cluster means.

Counting parameters for clustering candidates

For a $K$-component mixture in $d$ dimensions, the component means contribute $Kd$ parameters. The mixing proportions sum to one, so they contribute $K-1$ free parameters. If the covariance specification contributes $q$ free parameters in total, then

$$ p=Kd+(K-1)+q. $$

This expression assumes the component means and mixing proportions are freely estimated and that $q$ counts only the covariance parameters allowed by the chosen structure. For example, a single shared unrestricted covariance matrix contributes $d(d+1)/2$ parameters. Separate unrestricted covariance matrices contribute $K d(d+1)/2$. A shared spherical covariance, constrained to be $\lambda I_d$ for one positive scalar $\lambda$, contributes just one.

Parameter-count check: A symmetric $d\times d$ covariance matrix has $d$ diagonal entries and $d(d-1)/2$ distinct off-diagonal entries, for $d(d+1)/2$ free entries in total. Apply the covariance constraints before adding the means and mixing proportions.

For instance, in two dimensions, an unrestricted covariance matrix has $2(3)/2=3$ free parameters. With $K=2$, two component means contribute $2\times2=4$, and the mixing proportions contribute one. Thus a two-component model with one shared unrestricted covariance has $p=4+1+3=8$, while one with two unrestricted component-specific covariances has $p=4+1+2(3)=11$.

Worked Example: Counting Parameters Under Two Covariance Structures

Consider $K=3$ clusters in $d=2$ dimensions. Compare a model with one shared unrestricted covariance matrix to one with a separate unrestricted covariance matrix for each cluster.

Shared covariance. The means contribute $Kd=3\times2=6$ parameters. The mixing proportions contribute $K-1=2$. The shared covariance contributes $2(3)/2=3$. Therefore $p=6+2+3=11$.

Separate covariances. The means and proportions still contribute $6+2=8$. There are three covariance matrices, each with three free entries, so the covariance contribution is $3\times3=9$. Therefore $p=8+9=17$.

The separate-covariance model has six additional parameters. Its larger parameter count is not a reason by itself to reject it: BIC compares that cost with the fitted likelihood gain.

The exact trade-off between fit and penalty

The BIC comparison can be reduced to a simple threshold. This is useful because it isolates the likelihood improvement required to justify a more complex candidate.

Result: Let candidates $A$ and $B$ be fitted to the same $n$ observations, with maximized log-likelihoods $\ell_A,\ell_B$ and parameter counts $p_A,p_B$. BIC prefers $B$ over $A$ exactly when $$ 2(\ell_B-\ell_A)>(p_B-p_A)\log n. $$

Proof. By definition, BIC prefers $B$ exactly when \[ -2\ell_B+p_B\log n<-2\ell_A+p_A\log n. \] Add $2\ell_B+2\ell_A$ to both sides and rearrange: \[ (p_B-p_A)\log n<2(\ell_B-\ell_A). \] This is the stated condition. If the two candidates have the same parameter count, it reduces to preferring whichever has the larger maximized log-likelihood. If $B$ has more parameters, its fit improvement must exceed the penalty threshold. $\square$

Worked Example: The Likelihood Gain Needed for More Parameters

Suppose a candidate with $p_A=8$ parameters is compared with one having $p_B=11$, using $n=120$ observations. The more complex candidate must satisfy

2(\ell_B-\ell_A)>(11-8)\log(120) =3(4.7875)\mathrel{\approx}14.36.

Equivalently, its maximized log-likelihood must improve by more than $14.36/2=7.18$. If the increase is $6.9$, the candidate fits better, but BIC does not prefer it. If the increase is $7.3$, the inequality holds and BIC prefers it. Equality would give equal BIC scores.

A model with the best fit can lose

The following comparison illustrates why likelihood ranking and BIC ranking can differ. Suppose $n=120$ observations in two dimensions have been fitted under four candidate specifications. Assume the displayed log-likelihoods are the values used for scoring and the stated parameter counts match the candidate constraints.

Worked Example: The Highest Likelihood Does Not Win

Calculate BIC for the following candidates. The one-cluster model has an unrestricted covariance. The two-cluster models use a shared spherical, shared unrestricted, or component-specific unrestricted covariance, respectively.

Candidate	$K$	Covariance structure	$p$	$\ell(\hat{\theta})$
A	1	Unrestricted	5	-350
B	2	Shared spherical	6	-333
C	2	Shared unrestricted	8	-326
D	2	Component-specific unrestricted	11	-321

The counts can be checked directly. Candidate A has two mean parameters and three covariance parameters, for $p=5$. Candidate B has four mean parameters, one mixing-proportion parameter, and one shared spherical variance, for $p=6$. Candidate C has $4+1+3=8$. Candidate D has $4+1+2(3)=11$.

Since $\log(120)\approx4.7875$, the four BIC values are:

\begin{aligned} \operatorname{BIC}_A&=-2(-350)+5(4.7875)=723.94,\\ \operatorname{BIC}_B&=-2(-333)+6(4.7875)=694.73,\\ \operatorname{BIC}_C&=-2(-326)+8(4.7875)=690.30,\\ \operatorname{BIC}_D&=-2(-321)+11(4.7875)=694.66. \end{aligned}

Candidate D has the largest log-likelihood, $-321$, so it is the best-fitting candidate by that measure. But candidate C has the smallest BIC, approximately $690.30$, and is selected by BIC. From C to D, the log-likelihood improves by $5$, so the improvement in twice the log-likelihood is $10$. The parameter penalty increases by $3\log(120)\approx14.36$, which is larger than the fit gain. Thus D loses despite fitting better.

The result is a preference among the candidates provided, under the specified likelihood and parameter counts. It does not show that the selected covariance structure is scientifically correct, nor that the clusters are substantively meaningful. BIC’s penalty grows with $\log n$, while the improvement in fit depends on the data and the candidate models.

Interpreting the score responsibly

Several practical details matter when applying this calculation. First, compare models fitted to the same observations and with comparable likelihoods. If one candidate omits observations or uses a different likelihood convention, its raw BIC is not directly comparable without accounting for that difference. Second, use the maximized log-likelihood, not a likelihood value, an average log-likelihood, or a likelihood from a different fitting stage. Third, count every free parameter once and only once. In particular, do not count all entries of a covariance matrix as free when symmetry or a shared-structure constraint ties some entries together.

There is also an important mixture-model caveat. An unrestricted Gaussian mixture likelihood can become unbounded when a component covariance collapses around an observation. In that situation, an ordinary finite maximum-likelihood estimate need not exist, so a routine BIC calculation based on an alleged unrestricted maximum is not well-defined. Practical fitting may impose eigenvalue constraints or other regularization, or may return a local fitted solution. If so, report the fitting convention and interpret the resulting BIC as a conventional score for those fitted candidates, rather than as an automatic guarantee of the usual large-sample approximation.

For regular, identifiable models under suitable large-sample conditions, BIC is motivated as an approximation to a marginal-likelihood comparison. Finite mixture models have nonstandard features, including label switching and possible weak identification, so this motivation requires care. In routine clustering work, BIC is useful as a consistent bookkeeping rule for fit and complexity, but the score should be considered alongside the modeling assumptions and the quality of the fitted solutions.

Common mistake

Wrong reasoning: “Candidate D has the largest log-likelihood, so it must be the model BIC selects.” This overlooks the penalty term. In the worked comparison, D improves twice the log-likelihood over C by $10$, but it pays an additional penalty of about $14.36$.

Correction: Calculate both terms for every candidate and compare the resulting BIC values. A larger likelihood is guaranteed for a nested, more flexible fit under suitable optimization, but a smaller BIC is not. Parameter counts must also reflect the actual covariance constraints; miscounting them can reverse the ranking.

Key takeaway: For each candidate cluster count and covariance structure, compute $-2\ell(\hat{\theta})+p\log n$ using the correct free-parameter count. BIC selects the smallest score, not necessarily the largest likelihood.

Check Your Understanding

Use the definitions and comparisons above to check your reasoning.

For a two-component model in three dimensions with one shared unrestricted covariance matrix, calculate the total number of free parameters.
Two candidates have the same parameter count and are fitted to the same data. Which one has the smaller BIC: the candidate with the larger or smaller maximized log-likelihood?
A candidate adds four free parameters at sample size $n=200$. How much must its log-likelihood improve, at minimum, for BIC to prefer it over the simpler candidate?
Why can a component-specific unrestricted covariance model have a larger log-likelihood but a larger BIC than a shared-covariance model?
What problem arises if an unrestricted Gaussian mixture likelihood is unbounded, and why does that matter for calculating BIC?