Why Gaussian Mixture Likelihoods Are Unbounded

Why a Finite Mixture Can Have No Finite Maximum

A likelihood can become arbitrarily large even when every parameter value in the model gives valid, nonsingular Gaussian distributions. In an unrestricted Gaussian mixture, the key is to let one component become extremely concentrated around an observed point, while another component continues to assign positive density to the rest of the data. The covariance matrices along this sequence remain positive definite; it is their limiting determinant that approaches zero.

This is a boundary phenomenon. A stationary point in the interior of the parameter space does not rule it out: the likelihood may increase along a sequence approaching a boundary without attaining its supremum at any admissible parameter value.

Model and Likelihood

Fix observations \(x_1,\ldots,x_n\) in \(\mathbb{R}^d\), where \(n\geq 1\). A \(K\)-component Gaussian mixture has weights \(\pi_k\) that sum to one and component distributions \(N(\mu_k,\Sigma_k)\), with each covariance matrix \(\Sigma_k\) positive definite. Its density at \(x\) is the weighted sum of the component densities, and its likelihood for the fixed observations is the product of those mixture densities.

Here

The unrestricted model allows each component to have its own positive-definite covariance, with no fixed lower bound on its eigenvalues. The following result uses at least two components, so one can collapse while another supplies a positive background density.

The construction also works if the parameter space permits zero weights, but that is not needed. To keep every weight positive for any \(K\geq2\), choose a collapsing component with weight \(a\in(0,1)\), and give each of the other \(K-1\) components weight \((1-a)/(K-1)\). Set all those other components equal to the same fixed Gaussian \(N(\mu_0,\Sigma_0)\), where \(\Sigma_0\) is positive definite. Their combined contribution is the background density \((1-a)q(x)\), with \(q(x)=\phi_d(x;\mu_0,\Sigma_0)>0\) at every finite \(x\).

Proof by Collapsing One Component

Choose an observed point, say \(x_1\), and set the first component mean equal to it. For \(\varepsilon>0\), let its covariance be \(\varepsilon^2 I_d\), where \(I_d\) is the \(d\)-dimensional identity matrix. Every covariance on this path is positive definite, and

Write \(f_\varepsilon\) for the resulting mixture density. At the selected observation, its density is at least the contribution of the collapsing component. At each other observation, it is at least the contribution of the background mixture. Thus

Multiplying these \(n\) lower bounds gives

where the empty product is one when \(n=1\), and \(C>0\) is fixed independently of \(\varepsilon\). Since \(\varepsilon^{-d}\to\infty\) as \(\varepsilon\downarrow0\), the likelihood is unbounded above. Each \(\theta_\varepsilon\) is admissible; the singular covariance occurs only in the limit and is not itself a parameter value in this model. \(\square\)

Taking logarithms of the lower bound also shows that the log-likelihood diverges:

The argument needs no assumption that the observations are distinct, that they come from the mixture model, or that a particular clustering is correct. It is a property of the unrestricted finite-sample likelihood.

How the Divergence Depends on Repeated Observations

The lower bound above uses one collapsing observation and guarantees a divergence of at least order \(\varepsilon^{-d}\). If several observations coincide at the collapse point, the same component raises the density for each of them. In fact, along the stated covariance path, the likelihood then has an exact leading order.

To verify this, let \(c=(2\pi)^{-d/2}\). At any observation equal to \(x_*\), the mixture density is \(ac\varepsilon^{-d}+b(x_*)\), where \(b\) is the fixed background density. Dividing by \(ac\varepsilon^{-d}\) gives a ratio tending to one. For an observation \(x_i\ne x_*\), its collapsing-component contribution is

Indeed, with \(t=\varepsilon^{-2}\) and \(r=\|x_i-x_*\|^2/2>0\), the logarithm of the variable part is \((d/2)\log t-rt\), which tends to \(-\infty\). Hence the density at \(x_i\) tends to the positive background value \(b(x_i)\). There are finitely many observations, so multiplying these limits gives the claimed equivalent, with

This argument establishes the exact rate for this particular path. It does not assert that the likelihood has no other paths to divergence.

Worked Examples

Interpretation and Common Pitfalls

An unbounded likelihood means that some admissible parameter values have likelihoods larger than any chosen finite threshold. It does not mean that a parameter with zero covariance is an admissible maximizer. In the construction, every \(\varepsilon>0\) gives a valid Gaussian component, but the limiting singular covariance is outside the model. Thus the likelihood supremum is infinite and no finite parameter value attains it.

This boundary behavior matters when interpreting likelihood geometry and numerical fitting. An interior stationary point can exist without being a global maximum, and a sequence of fitted parameters may move toward a covariance boundary. An algorithm's convergence to a stationary point alone does not establish that the unrestricted likelihood has a finite global maximum.

Common mistake: “The collapsing component has density approaching zero at observations other than its mean, so the product likelihood must go to zero.” This overlooks that the mixture density is a sum. At every non-collapsing observation, the fixed background component contributes \((1-a)q(x_i)>0\). The full likelihood multiplies these positive mixture densities; it does not multiply the collapsing component's densities alone.

A second calculation error is to confuse covariance determinant with density growth. If \(\Sigma_\varepsilon=\varepsilon^2I_d\), then \(|\Sigma_\varepsilon|=\varepsilon^{2d}\), and the Gaussian normalization factor is \(|\Sigma_\varepsilon|^{-1/2}=\varepsilon^{-d}\). The exponent depends on dimension; it is not generally \(\varepsilon^{-1}\).