Why Compare Component Counts Carefully?
A two-component Gaussian mixture can fit data better than a single Gaussian because the larger model includes more ways to describe variation. That improvement alone does not establish that the data support two components: an added component can capture a real secondary concentration, or merely exploit a few observations. A defensible comparison therefore considers the likelihood gain, a complexity penalty such as AIC or BIC, and evidence calibrated under the one-component model.
Mixture comparisons also require attention to the covariance estimates. An unconstrained Gaussian-mixture likelihood can become unbounded when a component covariance collapses around an observation. Here, when fitting two components, impose the explicit constraint that the shared variance is at least \(0.25\). The constraint makes the numerical comparison well-defined for the examples; it does not make every model-selection criterion automatically reliable.
For an univariate Gaussian, the one-component model estimates a mean and variance, so \(k_1=2\). In the examples below, the two-component model estimates two means, one shared variance, and one mixing weight, so \(k_2=4\). The mixing weight is constrained to lie strictly between zero and one. A fitted model with a covariance floor still has the same nominal parameter count when the estimate is above the floor; if an estimate is on a constraint boundary, ordinary large-sample arguments need additional care.
Worked Comparison on Eight Observations
Worked Example: Likelihood Improvement and Information Criteria
Consider the eight observations \(-4,-4,-3,-3,3,3,4,4\). Compare a one-component normal model with a two-component mixture having equal component variances, constrained to be at least \(0.25\). The data are symmetric, with sample mean \(0\). Their sum of squared deviations from zero is \(4(3^2)+4(4^2)=100\), so the one-component maximum-likelihood variance is \(100/8=12.5\).
The one-component log-likelihood is
A two-component fit places one mean near \(-3.5\), the other near \(3.5\), and the mixing weight near \(0.5\). At the displayed precision, the shared variance is \(0.25\). This is a numerical description, not a claim that the exact unapproximated likelihood has its maximum precisely on the variance boundary: the distant component contributes a tiny but positive density, so the exact fitted variance can be slightly above \(0.25\). The fitted log-likelihood is approximately \(\ell_2=-11.351508266\), giving
For the criteria, use unrounded likelihoods until the final reported values. The one-component model has two parameters and the two-component model has four:
Both criteria prefer the two-component fit because its score is smaller. In this example, the BIC score for the one-component model exceeds the two-component score by approximately \(47.067729-31.020783=16.046946\), or \(16.047\). This is the BIC score difference for these particular data and parameter counts.
What the Criteria Do—and Do Not—Say
The likelihood improvement \(T\) measures how much better the larger model fits before accounting for its additional parameters. AIC and BIC subtract different amounts for that added flexibility. For the same two candidate models, the exact score differences satisfy
These identities follow by subtracting the definitions. For example, \(\mathrm{BIC}_2-\mathrm{BIC}_1=(-2\ell_2+ k_2\log n)-(-2\ell_1+k_1\log n)\), which equals \(-2(\ell_2-\ell_1)+(k_2-k_1)\log n\), or \(-T+(k_2-k_1)\log n\). This also makes the sign convention transparent: a negative difference favors the two-component model.
AIC and BIC are useful summaries, not hypothesis tests and not guarantees that the selected model is literally true. In mixture models, the one-component null corresponds to a boundary or singular situation: under that null, the second component’s parameters are not identified in the ordinary way. Consequently, the usual likelihood-ratio chi-squared reference distribution for regular nested models should not be assumed. A parametric bootstrap provides a practical way to assess the likelihood gain under the fitted one-component model.
Worked Example: Reading a Parametric Bootstrap
Continue with the eight observations above and observed statistic \(T_{\mathrm{obs}}\approx20.205829\). Fit the one-component null model, here \(N(0,12.5)\). To approximate the null distribution of the statistic, generate \(B=999\) bootstrap samples of size eight from that fitted normal model. For each sample, fit both models using the same variance constraint and comparable initialization and restart procedures, then calculate its statistic \(T_b\).
Suppose a particular bootstrap run produces \(b=19\) statistics at least as large as the observed value. The customary plus-one estimate of the upper-tail probability is
This calculation illustrates how to interpret a bootstrap output; the value \(0.020\) is conditional on the stated run and is not implied by the data or by the likelihood calculation alone. A small estimated tail probability means that improvements at least this large were uncommon in the simulated samples under the fitted one-component null. It is evidence against that null model, subject to the adequacy of the bootstrap fitting procedure and the plug-in null estimate.
Covariance Collapse and a Separate BIC Illustration
A covariance estimate near its imposed floor is a warning to inspect the fit, not a reason by itself to declare the larger model superior. Check whether a component is supported by several observations, whether it has effectively isolated one point, and whether the likelihood improvement persists under a reasonable constraint or penalty. The numerical optimizer should use the same lower bound, convergence checks, and restart strategy in the observed data and every bootstrap replicate. Otherwise, the comparison can partly reflect different optimization behavior rather than different evidence.
The BIC difference depends on the data as well as the parameter penalty. In particular, one cannot transfer the BIC difference from one dataset to another merely because both comparisons use eight observations and the same parameter counts.
Worked Example: A Larger BIC Gap for More Separated Data
To see how separation changes the comparison, take four observations at \(-a\) and four at \(a\), with \(a^2=\exp(4.92274)\), approximately \(137.38\). The one-component normal fit has mean zero and variance \(a^2\). In the two-component fit, place means at \(-a\) and \(a\), use equal weights, and use the variance floor \(0.25\). At this separation, the opposite component’s contribution to each observed density is negligible at the displayed precision.
The one-component log-likelihood is \(\ell_1=-4\log(2\pi a^2)-4\). At each observed point, the near component has density \(2/\sqrt{2\pi}\) and weight \(1/2\), so the mixture density is \(1/\sqrt{2\pi}\). Thus \(\ell_2\approx8\log(1/\sqrt{2\pi})=-4\log(2\pi)\). Subtracting the BIC scores gives
The positive difference means BIC is lower for the two-component fit by about \(43.22\). The calculation uses the stated approximate fit and effectively neglects the extremely small opposite-component densities; a numerical optimizer would report the corresponding finite-mixture values. The example is not a universal cutoff: it shows that stronger separation can produce a substantially larger fit gain, while the BIC penalty remains fixed for these parameter counts and sample size.
A Practical Comparison Sequence
A careful analysis keeps model fitting, scoring, and interpretation distinct. The following sequence makes the comparison easier to audit.
State the component counts, covariance structure, mixing-weight constraints, and any covariance floor before fitting.
Use appropriate starts and restarts, check convergence, and inspect fitted weights and covariance estimates for collapse or near-collapse.
Use the maximized log-likelihoods at full precision, count parameters under the specified models, and use the same sample size in BIC.
When a likelihood-ratio reference distribution is nonstandard, simulate under the fitted null and repeat the full fitting procedure in each bootstrap sample.
Report which criterion favors which model and whether the result is sensitive to covariance constraints, initialization, or bootstrap uncertainty.
Common Mistake
A frequent error is to treat any numerical estimate equal to the covariance floor as proof that the exact mixture likelihood is maximized at the boundary. For a finite Gaussian mixture, even a very distant component contributes a positive density. That contribution can change the derivative of the likelihood, so rounding an estimate to the floor does not establish exact boundary contact. Report the estimate as “at the floor to the displayed precision,” or explicitly state that a calculation approximates negligible component contributions as zero.
A second check is arithmetic and sign discipline. For the eight-observation example, the BIC difference is \(47.067729-31.020783\approx16.047\), not a value obtained by adding the penalty terms with the wrong signs. Showing the two BIC scores before taking their difference makes the preferred model and the magnitude auditable.
Check Your Understanding
Use the definitions and calculations above to answer the following questions.
- For a two-component fit with \(\ell_2-\ell_1=6\), \(k_1=2\), and \(k_2=4\), what is the likelihood-improvement statistic?
- Why should the usual regular-model chi-squared likelihood-ratio calibration not be assumed automatically for adding a mixture component?
- In the eight-observation worked example, which model has the smaller BIC, and what are the two displayed BIC values?
- A fitted shared variance is reported as \(0.25\) to two decimal places under a \(0.25\) lower bound. What additional caution is needed before claiming exact boundary contact?
- If 9 of 499 bootstrap statistics are at least as large as the observed statistic, calculate the plus-one bootstrap tail estimate.