Nonstandard Asymptotics at Mixture Singularities

Why Mixture Parameters Can Have Nonstandard Limits

Likelihood-based estimates are often expected to fluctuate at rate \(n^{-1/2}\) and to have an approximately normal distribution. Those conclusions rely on a locally identifiable parameter in the interior of its parameter space, a sufficiently smooth likelihood, and nonsingular Fisher information. Finite mixtures can violate these conditions in distinct ways: a weight can be zero, component parameters can become indistinguishable, or a parameter can lie on a boundary. The resulting likelihood may have flat directions or higher-order curvature, so the usual quadratic approximation need not describe estimation.

For a \(K\)-component mixture with observation density \(f_\theta\), the density has the form

Here \(\eta_k\) denotes the parameters of component \(k\). Even when each component density is smooth, the full model can be nonregular at particular parameter values. The issue is not simply that a mixture has many parameters; it is that different parameter values can describe the same distribution, or that local changes in parameters affect the density only at a higher-than-first order.

Three Different Sources of Singularity

Label switching is the invariance of a mixture density to a permutation of component labels. For distinct components, this is a finite global ambiguity: a sufficiently small neighborhood around one labeling can exclude the permuted labeling. After choosing such a local labeling, the likelihood can still have nonsingular local curvature. Thus label switching alone does not imply singular local information at every parameter point.

A vanishing component weight creates another kind of failure. At \(\pi_k=0\), the density does not depend on \(\eta_k\), so the parameters of that absent component cannot be identified from the distribution. Also, the weight lies on the boundary of the simplex rather than in an open neighborhood. A coincident-component point occurs when two or more components have the same distribution. Their relative weights then cannot be determined from the observed density, and some directions that separate the components may change the density only at second or higher order.

A Vanishing Weight: A One-Sided Limit

A simple model isolates the effect of a boundary weight while keeping the two component distributions known. Let \(X_1,\ldots,X_n\) be independent with density

Suppose the true value is \(\varepsilon=0\). The first component is present and the second has zero weight. Relative to the true density, the likelihood ratio for one observation is \(1+\varepsilon s(X)\), where

Under \(X\sim N(0,1)\), \(E[r(X)]=1\) and \(E[r(X)^2]=e\). Therefore \(E[s(X)]=0\) and \(I=E[s(X)^2]=e-1>0\). In this example, the score for the weight has positive finite variance; nonregularity remains because \(\varepsilon=0\) is a boundary point.

Coincident Components: Curvature Appears at Higher Order

At a coincident-component point, even the first derivative of the density with respect to a separation parameter can vanish. Then the likelihood may respond to separation at a higher order, producing a slower estimation rate. The following model makes this mechanism explicit without estimating additional nuisance parameters.

Deriving the Coincident-Component Limit

To verify the local expansion, put \(\delta=n^{-1/4}\sqrt h\), for bounded \(h\geq0\). Then \(\delta^2=h/\sqrt n\). Summing the displayed log-density expansion over the observations gives

The remainder is uniform for bounded \(h\): the sum of its absolute values is bounded by a constant times \(n^{-3/2}\sum_i |X_i|^6\), which is \(O_p(n^{-1/2})\) because \(E|X|^6<\infty\). By the central limit theorem, \(\Delta_n=n^{-1/2}\sum_i a(X_i)\xrightarrow{d}\Delta\sim N(0,1/2)\). Also \(n^{-1}\sum_iX_i^4\xrightarrow{p}E[X^4]=3\). The limiting local log-likelihood is therefore

This strictly concave quadratic has the unique maximizer \(H=\max(0,\Delta/I)\). The assumed tightness of \(n^{1/2}\hat\delta^2\), together with the local argmax conditions, gives \(n^{1/2}\hat\delta^2\xrightarrow{d}H\). Since the square-root function is continuous on \([0,\infty)\), the continuous mapping theorem gives \(n^{1/4}\hat\delta\xrightarrow{d}\sqrt H\). This proves the stated localized limits. In particular, \(H=\max(0,\sqrt2 Z)\) for \(Z\sim N(0,1)\), so the separation estimate has a fourth-root rate and a limit with positive probability at zero. \(\square\)

Implications for Mixture Inference

These examples show two different departures from ordinary likelihood theory. A boundary weight can retain positive score information while still producing a one-sided, nonnormal limit because its parameter space has an edge. At a coincident-component point, the score for separation can vanish, so the density changes first through a squared separation and estimation proceeds at a slower rate. When a component weight is zero, its mean and covariance are additionally unidentified because they do not enter the distribution at all.

For comparison, the familiar interior approximation requires a locally identifiable parameter, an open parameter neighborhood, differentiability and suitable integrability of the likelihood, and finite positive-definite Fisher information. Under the usual regularity and consistency conditions, these assumptions yield a quadratic local likelihood and an \(n^{-1/2}\) asymptotically normal maximum-likelihood estimator. At a mixture singularity, one or more assumptions fail, so standard errors, Wald intervals, and likelihood approximations cannot be accepted just because the component densities themselves are Gaussian and smooth. The appropriate limit depends on the local geometry and on which parameters remain identifiable.