Likelihood-Ratio Tests for an Additional Component

Why Adding a Component Is Not an Ordinary Nested-Model Test

A likelihood-ratio statistic compares the best fit under a larger model with the best fit under a restricted model. In many familiar regular problems, its null distribution is approximately chi-square, with degrees of freedom equal to the difference in parameter dimensions. Adding a component to a mixture looks like such a comparison, but it fails key regularity conditions: the null lies on a boundary, and parameters describing the extra component cease to be identifiable under the null.

These failures are not minor adjustments to the degrees of freedom. They change the asymptotic problem itself. A chi-square reference distribution obtained by counting the extra means, covariances, and weights is generally unjustified.

Definition: For nested models with parameter spaces $\Theta_0\subseteq\Theta_1$, the likelihood-ratio statistic is $$ \Lambda_n=2\left\{\sup_{\theta\in\Theta_1}\ell_n(\theta)-\sup_{\theta\in\Theta_0}\ell_n(\theta)\right\}, $$ where $\ell_n$ is the log-likelihood based on $n$ observations. The statistic is nonnegative because the larger model contains the smaller one.

Wilks’ theorem gives $\Lambda_n\xrightarrow{d}\chi^2_r$ under regularity conditions, where $r$ is the difference in model dimensions. In its usual form, the observations are independent and identically distributed; the null parameter is an interior point of a smooth, identifiable model; the likelihood is suitably differentiable; and the Fisher information is finite and nonsingular. The dimension difference is meaningful only within this regular framework.

How a K-Component Null Sits Inside the Larger Model

Write a finite mixture density as

f_K(x)=\sum_{j=1}^{K}\pi_j f(x;\eta_j),\qquad \pi_j\geq 0,\qquad \sum_{j=1}^{K}\pi_j=1.

A $K+1$-component model contains the $K$-component model by setting the added component’s weight to zero. If the new weight is $\pi_{K+1}$, the null restriction is $\pi_{K+1}=0$, which is a boundary of the allowed weight space. If the model instead requires strictly positive weights, the null is not in that parameter space itself; it is in its closure. Either way, the ordinary interior-point condition does not hold.

There is a second problem. When $\pi_{K+1}=0$, the density no longer depends on $\eta_{K+1}$, the extra component’s parameters. Its mean, covariance, or other shape parameters cannot be learned from data generated under that null representation. In addition, the same $K$-component density can sometimes be represented by splitting a component into two identical components. Thus the null may have multiple representations in the larger model.

Label switching creates equivalent parameter labels, but resolving labels does not remove the zero-weight boundary or the unidentified parameters. Nor does counting the nominal number of parameters fix the problem: the added parameters do not behave like ordinary, estimable directions under the null.

Worked Example: A Boundary Test With the Component Shapes Fixed

Let $X_1,\ldots,X_{10}$ be independent Bernoulli observations. Compare a null distribution with success probability $0.2$ to mixtures of two fixed Bernoulli distributions, with success probabilities $0.2$ and $0.8$. If $p$ is the weight on the second component, the mixture is Bernoulli with probability

q(p)=(1-p)(0.2)+p(0.8)=0.2+0.6p,\qquad 0\leq p\leq1.

Suppose four of the ten observations are successes. Under the alternative, the sample proportion $0.4$ is allowed, so it maximizes the likelihood. It corresponds to

\hat p=\frac{0.4-0.2}{0.6}=\frac13.

The binomial coefficient cancels in the likelihood ratio. Therefore

\Lambda_{10} =2\left\{4\log\left(\frac{0.4}{0.2}\right) +6\log\left(\frac{0.6}{0.8}\right)\right\} =8\log 2+12\log(0.75) \approx 2.093.

This example isolates the boundary issue: the component distributions are fixed, so there are no unknown extra shape parameters. Even so, $p=0$ is an endpoint, not an interior value. This simpler test has a one-sided boundary reference law under the null, as derived below; it is not an ordinary two-sided one-parameter Wilks test.

What the Boundary Alone Does to the Reference Law

The fixed-shape Bernoulli example has a particularly simple asymptotic limit. It is useful because it shows that a boundary can change the likelihood-ratio distribution even before introducing the unidentified parameters that occur in a general mixture-order test.

Theorem: Suppose $X_1,\ldots,X_n$ are independent Bernoulli observations with null success probability $q_0$, where $0<q_0<1$. Test $q=q_0$ against $q\geq q_0$, with the upper endpoint bounded away from $q_0$. Then, under the null, $$ \Lambda_n\xrightarrow{d}\tfrac12\delta_0+\tfrac12\chi^2_1, $$ where $\delta_0$ denotes a point mass at zero.

Proof. Let $\hat q_n$ be the sample proportion and define

Z_n=\frac{\sqrt n(\hat q_n-q_0)}{\sqrt{q_0(1-q_0)}}.

The central limit theorem gives $Z_n\xrightarrow{d}N(0,1)$. The unrestricted maximum likelihood estimate is $\hat q_n$; under the one-sided alternative the constrained estimate is $\max(\hat q_n,q_0)$, with probability tending to one the upper endpoint does not bind. A second-order Taylor expansion of the Bernoulli log-likelihood about $q_0$ gives, for $q-q_0=O(n^{-1/2})$,

2\{\ell_n(q)-\ell_n(q_0)\} =2Z_n t-t^2+o_p(1), \qquad t=\frac{\sqrt n(q-q_0)}{\sqrt{q_0(1-q_0)}}\geq0.

The quadratic expression is maximized over $t\geq0$ at $t=\max(Z_n,0)$. Consequently,

\Lambda_n=(\max(Z_n,0))^2+o_p(1).

For a standard normal variable $Z$, $P(Z\leq0)=1/2$, producing a point mass at zero. Conditional on $Z>0$, $Z^2$ has the $\chi^2_1$ distribution. This proves the stated limit. $\square$

This result does not supply a general mixture-order reference distribution. In a genuine additional-component test, the shape of that component is also unknown and unidentified under the null. The likelihood can search across those possible shapes, so the simple one-sided quadratic argument no longer describes the whole limit.

The Extra Component’s Parameters Are Unidentified

The identification failure can be seen directly from a two-part mixture

f_{p,\eta}(x)=(1-p)f_0(x)+p\,g(x;\eta),\qquad 0\leq p\leq1.

Here $f_0$ is the null density, $g(\cdot;\eta)$ is the additional component, and $\eta$ is its parameter. Assume the densities are differentiable in the relevant parameters and the displayed derivatives exist. At $p=0$, the density equals $f_0$ for every value of $\eta$. Hence the null distribution contains no information about $\eta$.

Derivation: At $p=0$, differentiation of the density gives $$ \frac{\partial f_{p,\eta}(x)}{\partial\eta} =p\frac{\partial g(x;\eta)}{\partial\eta}=0, \qquad \left.\frac{\partial f_{p,\eta}(x)}{\partial p}\right|_{p=0} =g(x;\eta)-f_0(x). $$ Thus the score for $\eta$ vanishes under the null, while the score for $p$ depends on $\eta$. The information matrix has no ordinary nonsingular information for $\eta$ at the null, and even the direction in which the weight first increases varies with $\eta$.

This calculation explains why the null likelihood cannot estimate the extra component’s parameters and why the alternative effectively searches over them. Standard Wilks theory requires a locally identifiable parameterization with nonsingular information; that requirement fails here. A nominal parameter-count difference therefore does not determine the limiting degrees of freedom. In some special mixture testing problems a nonstandard limit can be characterized under additional assumptions, but it must be derived for that problem rather than inferred from parameter counting.

Worked Example: A Gaussian Component With an Unidentified Mean

Let the null density be $\phi(x)$, the $N(0,1)$ density, and consider

f_{p,\mu}(x)=(1-p)\phi(x)+p\phi(x-\mu).

Under the null $p=0$, every value of $\mu$ gives exactly the same density $\phi(x)$. For a single observation $x=2$, the likelihood ratio for the particular alternative values $p=1/2,\mu=2$, compared with the null, is

\frac{f_{1/2,2}(2)}{\phi(2)} =\frac12+\frac12\frac{\phi(0)}{\phi(2)} =\frac12+\frac12 e^2 \approx 4.195.

The corresponding twice-log likelihood ratio for this candidate is $2\log(4.195)\approx2.867$. The fully maximized statistic can be no smaller, because the alternative includes this candidate and the null fit is fixed here. This number is an illustration of likelihood improvement, not a calibrated significance test.

The dependence on the unidentifiable mean is also visible in the score for $p$ at zero:

\left.\frac{\partial}{\partial p}\log f_{p,\mu}(x)\right|_{p=0} =\frac{\phi(x-\mu)}{\phi(x)}-1 =e^{\mu x-\mu^2/2}-1.

Under the null $X\sim N(0,1)$, this score has mean zero, but its variance is $e^{\mu^2}-1$, which depends on $\mu$. Indeed, $E[e^{\mu X-\mu^2/2}]=1$ and $E[e^{2\mu X-\mu^2}]=e^{\mu^2}$. The null data do not select a value of $\mu$, yet the alternative’s score behavior varies with it.

Worked Example: A Candidate Two-Gaussian Fit

Take observations $-2$ and $2$. Compare a one-component model $N(\mu,1)$ with a two-component model whose component variances are fixed at $1$. The null maximum likelihood estimate is $\hat\mu=0$. As a particular two-component candidate, use equal weights and component means $-1$ and $1$.

For any mean $m$, the density ratio is

\frac{\phi(x-m)}{\phi(x)}=e^{mx-m^2/2}.

At $x=2$, the candidate mixture’s density relative to the null density is

\frac12e^{1.5}+\frac12e^{-2.5} =\frac{e^{1.5}+e^{-2.5}}{2} \approx 2.282.

By symmetry, the same ratio holds at $x=-2$. The candidate’s likelihood ratio for both observations is therefore approximately $2.282^2$, giving

2\log\left(\frac{L_{\mathrm{candidate}}}{L_{\mathrm{null}}}\right) =4\log(2.282)\approx3.30.

This is only a lower bound on the maximized likelihood-ratio statistic: the two-component fit may improve further when its weight and means are optimized. The calculation shows how an added component can improve the fit, but it does not make a $\chi^2$ calibration valid. Under the null, the added weight can be zero and the added mean is then unidentified.

Practical Implications and Common Pitfalls

A likelihood-ratio statistic remains a useful measure of relative fit, but its reference distribution must be justified separately from its definition. In particular, the usual $\chi^2$ approximation with degrees of freedom equal to the nominal parameter-count difference is not generally justified for testing $K$ against $K+1$ mixture components.

There is also a distinct likelihood issue for unconstrained Gaussian mixtures: covariance collapse can make the likelihood unbounded. In that setting, a finite likelihood-ratio statistic may not even exist without appropriate restrictions, such as covariance constraints. A well-defined constrained likelihood does not, by itself, restore Wilks’ regularity conditions; the boundary and unidentified-parameter problems remain.

Pitfall: counting parameters and applying Wilks’ theorem. The extra component may nominally add a weight, mean, and covariance, so one might use that number as the chi-square degrees of freedom. Correction: dimension counting supplies the usual degrees of freedom only when the null is regular, interior, identifiable, and has nonsingular information. These conditions fail for an additional mixture component.
Pitfall: treating the weight boundary as the only issue. A one-sided boundary can yield a simple mixture of chi-square distributions in a regular one-parameter problem. Correction: in a mixture-order test, the extra component’s parameters are also unidentified under the null, so the boundary-only result need not apply.
Pitfall: assuming relabeling resolves the singularity. Component labels can be permuted without changing the density. Correction: label conventions address that symmetry, but they do not make a zero-weight component identifiable.
Pitfall: interpreting a large likelihood improvement as a calibrated test result. A maximized improvement is not itself a p-value. Correction: the null distribution must be obtained using a method and assumptions appropriate to the particular mixture test.

Key takeaway: Testing for an additional mixture component is nonregular because the null can set its weight to zero, a boundary value, while leaving the added component’s parameters unidentified. The ordinary chi-square likelihood-ratio rule cannot be justified by parameter counting alone.

Check Your Understanding

Use the definitions and derivations above to answer these questions.

Which regularity conditions required by the usual form of Wilks’ theorem fail when testing $K$ versus $K+1$ mixture components?
In a two-part mixture with added weight $p$, why does the null $p=0$ contain no information about the added component parameter $\eta$?
Why does the fixed-shape Bernoulli mixture have a simpler boundary limit than a Gaussian mixture with an unknown added mean?
In the Gaussian candidate-fit example, why is the calculated statistic a lower bound on the fully maximized likelihood-ratio statistic?
Why does correcting for component label switching not resolve the additional-component testing problem?