Weak Identification Under Nearly Coincident Components

Why Nearly Coincident Components Are Difficult to Estimate

A fitted mixture can have sharply reported component means or covariances while the observed data provide little evidence for those particular values. This happens when two components are close enough that substantially different parameter choices produce almost the same overall density. The components may be difficult to distinguish even though the mixture as a whole fits well.

This is weak identification: the distribution changes so little across a range of parameter values that finite data cannot pin those values down precisely. It is different from exact nonidentifiability, where distinct parameter values produce exactly the same distribution. Mixture models also have label switching: permuting component labels leaves the density unchanged. Weak identification is an additional problem, not a remedy for label switching.

Definition: A parameter is weakly identified in a region when changes in that parameter over a meaningful range produce only small changes in the probability distribution, so the likelihood contains little information for distinguishing those values. A local diagnostic is low curvature of the log-likelihood in the parameter direction; a density diagnostic is small change in the fitted density across the same range.

The issue is especially acute when component means or covariance matrices nearly coincide. A useful approach is to examine both the parameter profile and the fitted densities. Parameter estimates alone can exaggerate how much the data have taught us: component labels, means, and covariances may shift considerably across fits while the overall density remains nearly unchanged.

A Symmetric Mean Split

Consider a univariate, equally weighted mixture with known component variance 1 and means \(-\delta\) and \(+\delta\). Write \(\phi(x)\) for the \(N(0,1)\) density. Its mixture density is

f_\delta(x)=\frac{1}{2}\phi(x-\delta)+\frac{1}{2}\phi(x+\delta) =\phi(x)e^{-\delta^2/2}\cosh(\delta x).

The parameter \(\delta\) measures the half-separation between the means. The sign of \(\delta\) only exchanges the labels of the two components, so comparisons of separation ordinarily use \(|\delta|\). At \(\delta=0\), both components coincide and the mixture reduces to \(\phi(x)\).

Worked Example: Comparing a Small Mean Split with One Gaussian

For \(\delta=0.1\), compare the mixture density with the \(N(0,1)\) density at \(x=2\). The density ratio is

\frac{f_{0.1}(2)}{\phi(2)} =e^{-0.1^2/2}\cosh(0.1\cdot 2) =e^{-0.005}\cosh(0.2).

Using \(e^{-0.005}\approx 0.9950125\) and \(\cosh(0.2)\approx 1.0200668\), the ratio is approximately \(1.0149792\), or \(1.0150\) to four decimal places. Thus, at this point the mixture density is only about 1.50% higher than the single-normal density, despite having two distinct component means. A small pointwise difference does not by itself establish that two entire distributions are close, but it illustrates why near-coincidence can be difficult to detect.

Why the Mean Separation Is Locally Hard to Estimate

The density formula gives a precise local explanation. Expanding its ratio to the coincident density around \(\delta=0\) yields

e^{-\delta^2/2}\cosh(\delta x) =1+\frac{\delta^2}{2}(x^2-1)+O(\delta^4).

For each fixed \(x\), the first change in the density is of order \(\delta^2\), not order \(\delta\). The first derivative with respect to separation vanishes at coincidence.

Result: In the equally weighted \(N(-\delta,1)\) and \(N(\delta,1)\) mixture, the score for \(\delta\) at \(\delta=0\) is zero for every observation \(x\). Consequently, the Fisher information for \(\delta\), defined as the expectation of the squared score under the model at \(\delta=0\), is zero in this parameterization.

Differentiate the density ratio.
Let \(r_\delta(x)=f_\delta(x)/\phi(x)=e^{-\delta^2/2}\cosh(\delta x)\). The product rule gives \(r'_\delta(x)=e^{-\delta^2/2}\{-\delta\cosh(\delta x)+x\sinh(\delta x)\}\).

Evaluate at coincidence.
Since \(\sinh(0)=0\), substitution gives \(r'_0(x)=0\). Because \(f_\delta(x)=\phi(x)r_\delta(x)\), it follows that \(\left.\partial f_\delta(x)/\partial\delta\right|_{\delta=0}=0\).

Compute the score and information.
At \(\delta=0\), \(f_0(x)=\phi(x)>0\), so the score \(\left.\partial\log f_\delta(x)/\partial\delta\right|_{\delta=0}\) is zero. Its square is therefore zero for every \(x\), and its expectation is zero. \(\square\)

This zero information is a local statement at coincidence, not a claim that the separation can never be learned when the components are distinct. It does mean that ordinary first-order, regular approximations can fail near \(\delta=0\). For a sample, the log-likelihood may change mainly at second or higher order in the separation, producing a broad or unusually shaped profile.

Worked Example: A Nearly Flat Separation Profile

Take five observations \(x=(-\sqrt{1.25},-\sqrt{1.25},0,\sqrt{1.25},\sqrt{1.25})\), and suppose the variance is known to be 1 and the component weights are fixed at one half. Compare the log-likelihood at separation \(\delta\) with its value at \(\delta=0\). The observations’ common \(N(0,1)\) density factors cancel, giving

\ell(\delta)-\ell(0) =-\frac{5\delta^2}{2} +4\log\cosh(\sqrt{1.25}\,\delta).

At \(\delta=0.2\), the first term is \(-5(0.2)^2/2=-0.1\). Also, \(\sqrt{1.25}(0.2)=\sqrt{0.05}\approx0.223607\), and \(\log\cosh(0.223607)\approx0.0247944\). Therefore,

\ell(0.2)-\ell(0) \approx -0.1+4(0.0247944) =-0.0008224.

The log-likelihood at separation \(0.2\) is only about \(0.00082\) below its value at coincidence in this restricted model. The shallow profile means that this sample gives little likelihood-based separation between those two choices; it does not mean the fitted component means are known accurately.

The local curvature confirms the flatness. Differentiating the profile twice gives \(\ell''(0)=-5+\sum_i x_i^2\). Here \(\sum_i x_i^2=4(1.25)=5\), so \(\ell''(0)=0\). In fact, expanding \(\log\cosh z=z^2/2-z^4/12+O(z^6)\) gives \(\ell(\delta)-\ell(0)=-\frac{6.25}{12}\delta^4+O(\delta^6)\). The leading change is fourth order for this particular data set.

Nearly Equal Component Variances

The same cancellation can arise in a variance direction. Let \(g_v(x)\) be the \(N(0,v)\) density, where \(v>0\), and consider the equally weighted mixture with component variances \(1+\eta\) and \(1-\eta\), for \(|\eta|<1\). Its density is \(h_\eta(x)=\{g_{1+\eta}(x)+g_{1-\eta}(x)\}/2\). At \(\eta=0\), both components have variance 1.

Result: For the symmetric variance split just defined, \(\left.\partial h_\eta(x)/\partial\eta\right|_{\eta=0}=0\) for every \(x\). The density has no first-order change in the variance-splitting direction at coincidence.

Proof. The normal density \(g_v(x)=(2\pi v)^{-1/2}\exp\{-x^2/(2v)\}\) is differentiable in \(v>0\). By the chain rule,

\frac{\partial h_\eta(x)}{\partial\eta} =\frac{1}{2}g'_{1+\eta}(x)-\frac{1}{2}g'_{1-\eta}(x).

At \(\eta=0\), the two terms are the same derivative with opposite signs, so their sum is zero. Thus the density has zero first derivative in this direction at equal variances. \(\square\)

Worked Example: Variance-Split Component Densities

Compare \(N(0,1.1)\) and \(N(0,0.9)\) with \(N(0,1)\) at \(x=2\). For variance \(v\), the density ratio to \(N(0,1)\) is \(v^{-1/2}\exp\{(x^2/2)(1-1/v)\}\). Substituting \(x=2\) gives

\frac{g_{1.1}(2)}{g_1(2)} =\frac{1}{\sqrt{1.1}}\exp\{2(1-1/1.1)\} \approx 1.1436, \qquad \frac{g_{0.9}(2)}{g_1(2)} =\frac{1}{\sqrt{0.9}}\exp\{2(1-1/0.9)\} \approx 0.8441.

The equally weighted mixture’s ratio is the average: \((1.1436+0.8441)/2\approx0.99385\), or approximately \(0.9938\) to four decimal places when calculated before rounding the component ratios. The mixture density at \(x=2\) is therefore close to the \(N(0,1)\) density there, even though its component variances differ by \(0.2\). As with the mean-split example, this is a pointwise comparison, not a complete measure of distributional similarity.

Practical Diagnostics

A useful diagnostic compares the stability of component parameters with the stability of the overall fit. A likelihood profile fixes a parameter of interest at a sequence of values and, at each value, maximizes over the remaining parameters allowed by the model. A nearly level profile indicates that many values have similar likelihood. A fitted-density comparison asks a complementary question: do those values actually produce noticeably different probability densities over the data’s relevant range?

For a mean split, one can plot the profile against \(|\delta|\), or compare fitted density curves at several values along it. For covariance differences, one can similarly vary a covariance contrast and compare density contours or density values over a grid. In either case, profile and density comparisons should use the same constraints and likelihood being interpreted. A profile with a clear minimum or maximum is not itself a proof of practical stability; the range of plausible parameter values and the corresponding fitted densities matter.

These diagnostics are particularly useful when different initializations or small data perturbations produce visibly different component means or covariances. If the total fitted densities remain nearly identical, the component-specific estimates are sensitive to the data even though the overall model fit may be stable. Reporting that distinction is more informative than presenting one set of component estimates without qualification.

Common Pitfall

Confusing a stable fitted density with stable component estimates. Two fits can assign quite different means or covariances to the components yet produce almost the same mixture density. The correction is to report the parameter instability and assess the likelihood profile or compare fitted densities across the plausible parameter range.
Treating a small profile difference as proof that two component values are exactly equivalent. A finite sample may simply be uninformative for distinguishing them. Weak identification is about limited sensitivity, not an assertion of exact equality of distributions.
Reading a component’s standard error as if the model were regular at coincidence. First-order information can vanish in symmetric splitting directions. The correction is to examine the local likelihood shape and use inferential methods appropriate to the mixture setting rather than assuming a regular quadratic approximation.

Key takeaway: When mixture components nearly coincide, component-specific parameters can be unstable while the overall fitted density changes very little. Check both likelihood profiles and fitted densities; neither a single optimizer’s parameter estimates nor a good overall fit alone establishes that the components are precisely identified.

Check Your Understanding

Use the definitions, derivations, and examples above to answer the following.

In the symmetric two-component mean-split model, why is the density’s first derivative with respect to \(\delta\) zero at coincidence?
What does a nearly level likelihood profile for component separation indicate, and what does it not establish?
For the five observations in the profile example, what condition makes the second derivative at \(\delta=0\) vanish?
Explain how a variance split can have no first-order effect on the mixture density when its two component variances coincide.
Why should fitted-density comparisons accompany reports of unstable component means or covariances?