From Candidate Fits to a Defensible Clustering
Selecting a Gaussian clustering model is not just choosing the smallest information criterion in a table. The decision depends on which cluster counts and covariance structures were considered, whether the fits are credible, how clearly observations are assigned, and whether the solution persists under resampling. A useful analysis makes these steps explicit, so that the selected model is a supported working description rather than a claim that the data contain one uniquely correct partition.
For \(n\) observations \(x_1,\ldots,x_n\) in \(p\) dimensions, a \(K\)-component Gaussian mixture has density
where \(\phi_p(\,\cdot\,;\mu_k,\Sigma_k)\) is the \(p\)-variate Gaussian density with mean \(\mu_k\) and positive-definite covariance matrix \(\Sigma_k\). A candidate model specifies both \(K\) and constraints on the covariance matrices, such as a common full covariance matrix across components or separate full covariance matrices. In practice, candidate fits should be computed with suitable safeguards against degenerate covariance estimates and with multiple starting values, because mixture-model likelihood optimization can have local maxima.
Counting Parameters and Comparing Candidates
For a fitted candidate with maximized log-likelihood \(\ell\) and \(d\) free parameters, use the convention
so lower BIC is preferred among the candidates being compared. The parameter count includes the component weights, means, and the free covariance parameters. The \(K\) weights sum to one, so they contribute \(K-1\) parameters. The means contribute \(Kp\). A full symmetric covariance matrix contributes \(p(p+1)/2\) parameters; diagonal and spherical covariance constraints reduce that count.
To verify the count, each component mean has \(p\) unrestricted coordinates, giving \(Kp\). A symmetric \(p\times p\) matrix has \(p\) diagonal entries and \(p(p-1)/2\) distinct off-diagonal entries, for \(p+p(p-1)/2=p(p+1)/2\) parameters. Covariance equality across components leaves one such matrix; separate matrices multiply the count by \(K\). A diagonal matrix has only its \(p\) diagonal entries, while a common spherical matrix has the form \(\sigma^2 I_p\) and contributes one variance parameter. Adding the \(K-1\) independent weights gives the stated formula. \(\square\)
This count makes the BIC trade-off concrete. If candidate \(A\) has log-likelihood \(\ell_A\) and \(d_A\) parameters, and candidate \(B\) has \(\ell_B\) and \(d_B\), then
Thus, if \(B\) has more parameters, its improvement in log-likelihood must exceed \((d_B-d_A)\log(n)/2\) to lower BIC. This is an algebraic comparison of the stated scores, not a guarantee that the winning mixture model is scientifically correct.
Worked Example: Comparing Cluster Counts and Covariance Structures
Suppose \(n=120\) observations have \(p=2\) measured variables. Five safeguarded, multiple-start fits give the following maximized log-likelihoods. The candidate set includes one through four components and several covariance structures.
| Candidate | \(K\) | Covariance constraint | \(\ell\) | \(d\) | BIC |
|---|---|---|---|---|---|
| A | 1 | One common full matrix | -455 | 5 | 933.9 |
| B | 2 | One common full matrix | -410 | 8 | 858.3 |
| C | 2 | Separate full matrices | -409 | 11 | 870.7 |
| D | 3 | One common full matrix | -404 | 11 | 860.7 |
| E | 3 | Separate diagonal matrices | -400 | 14 | 867.0 |
For candidate B, the count is \(d=(2-1)+2(2)+2(3)/2=1+4+3=8\). Since \(\log(120)\approx4.7875\), its BIC is
For candidate D, \(d=(3-1)+3(2)+3=11\), so its BIC is \(808+11(4.7875)=860.66\), or 860.7. Candidate B has the smallest BIC in this table. Candidate D gains \(6\) log-likelihood units over B, but its three additional parameters impose a penalty of \(3\log(120)\approx14.36\) BIC units. The net BIC difference is \( -2(6)+14.36=2.36\), in favor of B. The result supports two components with a common full covariance matrix within this candidate set; the relatively small BIC separation from D counsels against presenting the choice as decisive.
Check What the Selected Fit Says About Assignments
A fitted mixture does not assign every observation to a component with certainty. For an observation \(x\), the posterior probability of component \(k\), conditional on the fitted parameters, is
This follows from Bayes' rule: the joint density for \(X=x\) and \(Z=k\) is \(\pi_k\phi_p(x;\mu_k,\Sigma_k)\), and the marginal density of \(X=x\) is the sum of those joint densities over all components. Dividing the joint density by that marginal density gives the displayed posterior probability. The maximum-posterior assignment is the component with the largest \(\tau_k(x)\), but the probability vector preserves uncertainty that a hard label would hide. \(\square\)
In a full analysis, inspect the distribution of the largest posterior probabilities and identify observations with diffuse membership. Also check component-specific fit, for example with the Mahalanobis diagnostics established in the earlier material. A good overall criterion score does not rule out a poorly fitting component, an outlier-driven component, or a near-duplicate component.
Worked Example: Calculating an Uncertain Membership
Consider two fitted components with equal identity covariance matrices, means \(\mu_1=(0,0)\) and \(\mu_2=(3,0)\), and weights \(\pi_1=0.55\) and \(\pi_2=0.45\). For \(x=(1,0)\), the squared distances from the means are \(1\) and \(4\). Because the covariance matrices are equal, the common Gaussian normalizing factor cancels in the posterior ratio; the component-specific density factors are proportional to \(\exp(-1/2)\) and \(\exp(-4/2)\). Therefore
Thus \(\tau_1(x)\approx0.846\), and the maximum-posterior assignment is component 1. It is not a virtually certain assignment: the fitted model gives about 15.4% membership probability to component 2. The point should therefore be described as assigned to component 1 with appreciable, but not overwhelming, model-based uncertainty.
Assess Stability Before Making the Selection
A model-selection score and a membership diagnostic describe the fitted data; neither alone shows whether the partition is sensitive to sampling variation. Bootstrap or subsampling refits provide a way to assess that sensitivity. Compare assignments for the same observations across refits, using a label-invariant summary such as the proportion of refits in which each pair is assigned together. If a refit does not contain an observation, compare only refits in which both members of the pair can be classified, or use a clearly specified prediction procedure to classify the original observations.
Stability is evidence about reproducibility under the chosen resampling and fitting procedure, not proof that clusters are natural entities. It is useful to examine whether the broad grouping persists and whether particular observations repeatedly change partners. A stable result can still reflect a continuous gradient, misspecified Gaussian components, or an unrepresentative sample.
Worked Example: Reading a Small Co-Clustering Summary
Suppose five bootstrap refits classify four common observations, A, B, C, and D. In four refits, the partition is \(\{A,B\},\{C,D\}\); in the remaining refit, it is \(\{A,C\},\{B,D\}\). The co-clustering proportion for a pair is the number of refits placing the pair together divided by five.
A and B are together in the first four refits, so their proportion is \(4/5=0.8\). C and D are also together in those four, giving \(0.8\). A and C are together only in the fifth refit, giving \(1/5=0.2\); B and D likewise have proportion \(0.2\). A and D are never together, so their proportion is \(0/5=0\). The high proportions for A–B and C–D and the low proportions across those pairs indicate a recurring two-group pattern in this small illustration. In an actual analysis, report the number of usable refits and inspect the full co-clustering pattern rather than relying on a few selected pairs.
A Practical End-to-End Analysis
The workflow below keeps model comparison, diagnostics, and interpretation connected while making clear what each part contributes.
Choose a scientifically plausible range of \(K\) and covariance constraints before interpreting the winning fit. State any covariance regularization or other restrictions that define the fitted candidates.
Use multiple starting values and check convergence. Compare the best attained likelihoods across starts, and investigate fits with degenerate or numerically unstable covariance estimates rather than accepting their scores uncritically.
Compute the same information criterion for every candidate, using the appropriate parameter count. Record the score differences and avoid implying that a small difference establishes a unique answer.
Review posterior membership uncertainty, component sizes, component-level fit diagnostics, and indications of redundant or outlier-driven components.
Compare plausible alternative covariance models and resampling refits. Report the chosen candidate, the main supporting evidence, important alternatives, and limitations.
Common Pitfalls and Interpretation
Pitfall: “The minimum BIC proves that the selected number of clusters is true.” This reasoning treats a relative score as a test of whether clusters exist and ignores the candidate set and model assumptions. The correction is to say that the model has the lowest BIC among the fitted candidates, then qualify that choice using score separation, assignment uncertainty, component diagnostics, and stability.
A second mistake is to interpret every maximum-posterior label as a certain classification. A label is only the component with the largest fitted membership probability; a case with probabilities \(0.51\) and \(0.49\) is very different from one with probabilities \(0.99\) and \(0.01\). Finally, high bootstrap stability does not validate the Gaussian assumption or establish that the groups are substantively meaningful. These checks answer different questions and should be reported as complementary evidence.
A concise report might state: “Among the evaluated one- to three-component Gaussian models, the two-component model with a common full covariance matrix had the lowest BIC (858.3); the three-component common-covariance fit was close (860.7). We retained the two-component fit as a parsimonious working solution, while noting that assignment uncertainty, component diagnostics, and resampling stability determine how strongly individual memberships and the overall partition should be interpreted.” A complete report should replace these illustrative values with the actual candidate set and observed diagnostics, and should state the population and measurements to which the analysis applies.
Check Your Understanding
Use these questions to check whether you can connect the selection calculation to the interpretation of a fitted clustering.
- For \(K=3\), \(p=4\), and one common full covariance matrix, calculate the number of free parameters.
- A more complex candidate adds four parameters and improves the maximized log-likelihood by 5. If \(n=100\), does its BIC decrease or increase relative to the simpler candidate?
- Why can a maximum-posterior assignment be misleading when the two largest membership probabilities are close?
- What does a high pairwise co-clustering proportion across bootstrap refits support, and what does it not establish?
- Name two checks, beyond the minimum BIC, that should inform a final clustering report.