Finite Gaussian Mixture Likelihood

Why the Mixture Likelihood Has a Particular Form

A finite Gaussian mixture represents a distribution whose density is assembled from a fixed number of multivariate Gaussian densities. Its parameters include both the parameters of those Gaussian components and the weights that determine their contributions. For observed vectors, the likelihood must account for the fact that each observation is described by the mixture density as a whole.

That last point determines the order of the calculation: first add the weighted component densities for each observation, then multiply those mixture densities across independent observations. Reversing these operations generally gives a different quantity. This tutorial defines the model and likelihood precisely and works through both the component calculations and the resulting product.

Finite Gaussian Mixture Density and Constraints

Definition: Fix positive integers $K$ and $d$. A $K$-component finite Gaussian mixture on $\mathbb{R}^d$ has density $$ f(x;\theta)=\sum_{k=1}^{K}\pi_k\,\phi_d(x;\mu_k,\Sigma_k), $$ where $$ \phi_d(x;\mu_k,\Sigma_k) =\frac{1}{(2\pi)^{d/2}|\Sigma_k|^{1/2}} \exp\!\left[-\frac{1}{2}(x-\mu_k)^\mathsf{T}\Sigma_k^{-1}(x-\mu_k)\right]. $$ Here $\mu_k\in\mathbb{R}^d$, $\Sigma_k$ is a $d\times d$ covariance matrix, and $\theta$ denotes the collection of all mixture parameters.

For this formula to define a mixture density with ordinary, nonsingular Gaussian components, the constraints are

\pi_k\geq 0,\qquad \sum_{k=1}^{K}\pi_k=1,\qquad \Sigma_k=\Sigma_k^\mathsf{T}\succ 0\quad (k=1,\ldots,K).

The weights lie in the probability simplex; they are not arbitrary coefficients. A zero weight is permitted on the boundary, although that component then contributes nothing to the density. The notation $\Sigma_k\succ0$ means that $v^\mathsf{T}\Sigma_kv>0$ for every nonzero $v\in\mathbb{R}^d$. This ensures that $\Sigma_k^{-1}$ and $|\Sigma_k|^{1/2}$ in the Gaussian density are well-defined and that the component is nondegenerate.

The mixture formula is a weighted sum of densities, not a weighted sum of covariance matrices or a Gaussian with averaged parameters. Its components may have different means and covariance matrices. The index $k$ identifies a component in the formula; the density itself is a function of the observed vector $x$ and the full parameter value $\theta$.

Worked Example: Checking the Mixture Constraints

Consider $K=2$ components in dimension $d=2$, with $\pi_1=0.3$, $\pi_2=0.7$, $\mu_1=(0,0)^\mathsf{T}$, $\mu_2=(1,-1)^\mathsf{T}$, $\Sigma_1=I_2$, and $\Sigma_2=2I_2$. Check that these values define an admissible mixture.

The weights are nonnegative and sum to one: $0.3+0.7=1$. Both covariance matrices are symmetric. For any nonzero $v\in\mathbb{R}^2$, $v^\mathsf{T}I_2v=v^\mathsf{T}v>0$, and $v^\mathsf{T}(2I_2)v=2v^\mathsf{T}v>0$. Thus both are positive definite, so the constraints hold.

For example, at $x=(0,0)^\mathsf{T}$, the first component density is $\phi_2(x;0,I_2)=(2\pi)^{-1}\approx0.159155$. For the second component, the determinant is $|2I_2|=4$, and the quadratic form is $(x-\mu_2)^\mathsf{T}(2I_2)^{-1}(x-\mu_2)=1$. Therefore

\phi_2(x;\mu_2,2I_2) =\frac{1}{4\pi}e^{-1/2} \approx 0.048267.

Combining the two contributions gives $f(x;\theta)=0.3(0.159155)+0.7(0.048267)\approx0.081533$. This value is the mixture density at $x$, not a probability that a continuous observation equals exactly $x$.

Why the Mixture Formula Is a Density

Each $\phi_d(\,\cdot\,;\mu_k,\Sigma_k)$ is a normalized multivariate Gaussian density. Because the weights are nonnegative and sum to one, their weighted sum is also normalized. The argument is short but important: it shows how the weight constraint enters the definition.

Result: If every component covariance matrix is positive definite, every $\pi_k\geq0$, and $\sum_{k=1}^{K}\pi_k=1$, then $f(x;\theta)$ is nonnegative and integrates to one over $\mathbb{R}^d$.

Proof. Every Gaussian density is nonnegative, so $f(x;\theta)\geq0$. To integrate, write $\Sigma_k=A_kA_k^\mathsf{T}$ for an invertible Cholesky factor $A_k$, and use the change of variables $u=A_k^{-1}(x-\mu_k)$. Its Jacobian gives $dx=|\det A_k|\,du=|\Sigma_k|^{1/2}du$. Consequently, the integral of the $k$th Gaussian component is the integral of the standard $d$-variate Gaussian density, which equals one. Since the sum has finitely many terms, its integral is the sum of the integrals:

\int_{\mathbb{R}^d} f(x;\theta)\,dx =\sum_{k=1}^{K}\pi_k \int_{\mathbb{R}^d}\phi_d(x;\mu_k,\Sigma_k)\,dx =\sum_{k=1}^{K}\pi_k=1.

Thus $f$ is a probability density. $\square$

Observed-Data Likelihood for Independent Observations

Suppose the observed data are $x_1,\ldots,x_n\in\mathbb{R}^d$, independent draws from the same mixture density $f(\,\cdot\,;\theta)$. The likelihood is the joint density of these observations, viewed as a function of the unknown parameter $\theta$ with the observed vectors held fixed.

Definition: The observed-data likelihood for $n$ independent observations from a finite Gaussian mixture is $$ L(\theta;x_1,\ldots,x_n) =\prod_{i=1}^{n}f(x_i;\theta) =\prod_{i=1}^{n}\left\{\sum_{k=1}^{K} \pi_k\phi_d(x_i;\mu_k,\Sigma_k)\right\}, $$ evaluated over parameter values satisfying the mixture constraints.

This product form follows directly from independence: the joint density of independent observations is the product of their individual densities, and each individual density is the mixture sum. The log-likelihood, useful for expressing this product as a sum, is therefore

\ell(\theta;x_1,\ldots,x_n) =\log L(\theta;x_1,\ldots,x_n) =\sum_{i=1}^{n}\log\!\left\{ \sum_{k=1}^{K}\pi_k\phi_d(x_i;\mu_k,\Sigma_k) \right\}.

The logarithm applies to the sum of weighted component densities for each observation. In general, it cannot be moved inside that sum: $\log(a+b)$ is not $\log(a)+\log(b)$. This observed-data likelihood is the target defined here; the expression does not require specifying or observing component labels.

Worked Example: A Two-Observation Likelihood in Two Dimensions

Use two components with weights $0.4$ and $0.6$, means $\mu_1=(0,0)^\mathsf{T}$, $\mu_2=(1,0)^\mathsf{T}$, and covariance matrices $\Sigma_1=I_2$, $\Sigma_2=2I_2$. The independent observations are $x_1=(0,0)^\mathsf{T}$ and $x_2=(1,0)^\mathsf{T}$. Evaluate their observed-data likelihood.

For $x_1$, the first component density is $\phi_2(x_1;\mu_1,I_2)=1/(2\pi)\approx0.159155$. For the second component, $|2I_2|=4$, and its quadratic form is $1/2$. The exponent is therefore $-\tfrac12(1/2)=-1/4$, giving

\phi_2(x_1;\mu_2,2I_2) =\frac{1}{4\pi}e^{-1/4} \approx0.061975.

The mixture density at the first observation is $f(x_1;\theta)=0.4(0.159155)+0.6(0.061975)\approx0.100847$.

For $x_2$, the first component has quadratic form $1$, so $\phi_2(x_2;\mu_1,I_2)=(2\pi)^{-1}e^{-1/2}\approx0.096532$. The second component is evaluated at its mean, so $\phi_2(x_2;\mu_2,2I_2)=1/(4\pi)\approx0.079577$. Thus $f(x_2;\theta)=0.4(0.096532)+0.6(0.079577)\approx0.086359$.

Independence now gives the likelihood:

L(\theta;x_1,x_2) =f(x_1;\theta)f(x_2;\theta) \approx(0.100847)(0.086359) \approx0.008709.

The component densities are evaluated first at each observation, combined using the weights, and only then multiplied across the two observations. The displayed likelihood is a joint density value for the observed pair at this parameter value.

Deriving the Product Form

The likelihood identity can also be read as a direct derivation from the joint density. Under the stated common-parameter model, each $X_i$ has density $f(\,\cdot\,;\theta)$, and independence means their joint density factors. Substituting the finite mixture formula into every factor yields the product of sums, with no further assumption about the observations.

Result: For independent $X_1,\ldots,X_n$ with common density $f(\,\cdot\,;\theta)$, the joint density evaluated at $x_1,\ldots,x_n$ equals $\prod_i f(x_i;\theta)$. If that common density is a finite Gaussian mixture, this is exactly the observed-data likelihood above.

Derivation. Independence gives $p_{X_1,\ldots,X_n}(x_1,\ldots,x_n)=\prod_{i=1}^n p_{X_i}(x_i)$. Since each marginal density is $p_{X_i}(x_i)=\sum_{k=1}^K\pi_k\phi_d(x_i;\mu_k,\Sigma_k)$, substitution gives

p_{X_1,\ldots,X_n}(x_1,\ldots,x_n) =\prod_{i=1}^{n}\left\{\sum_{k=1}^{K} \pi_k\phi_d(x_i;\mu_k,\Sigma_k)\right\}.

For fixed observed data, this joint density as a function of $\theta$ is $L(\theta;x_1,\ldots,x_n)$, by definition of likelihood. Taking its logarithm changes the product into a sum of logarithms, while leaving the sum over components inside each logarithm. $\square$

Worked Example: A One-Dimensional Mixture Likelihood

Let $K=2$, with weights $1/2$ and $1/2$, component distributions $N(0,1)$ and $N(2,1)$, and independent observations $x_1=0$ and $x_2=2$. Compute the likelihood in exact form and approximately.

At $x_1=0$, the component densities are $\phi(0;0,1)=1/\sqrt{2\pi}$ and $\phi(0;2,1)=e^{-2}/\sqrt{2\pi}$, since the squared displacement from the second mean is $4$. Hence

f(0;\theta)=\frac{1}{2\sqrt{2\pi}}(1+e^{-2}) \approx 0.226467.

At $x_2=2$, the component densities appear in the opposite order: $\phi(2;0,1)=e^{-2}/\sqrt{2\pi}$ and $\phi(2;2,1)=1/\sqrt{2\pi}$. Their equally weighted sum is therefore the same: $f(2;\theta)=\frac{1}{2\sqrt{2\pi}}(1+e^{-2})\approx0.226467$.

Multiplying the two observation densities gives

L(\theta;0,2) =\left\{\frac{1+e^{-2}}{2\sqrt{2\pi}}\right\}^{2} =\frac{(1+e^{-2})^2}{8\pi} \approx 0.051287.

Equivalently, the log-likelihood is $2\log\{\frac{1+e^{-2}}{2\sqrt{2\pi}}\}$, approximately $-2.970$. This illustrates that the log-likelihood adds one log mixture density per observation, rather than adding separate component log-densities.

Common Mistake: Summing Componentwise Products

A tempting but incorrect shortcut is to multiply the density values for each component across observations and then sum those products. That operation does not produce the observed-data likelihood for independent observations. The correct operation is a mixture sum separately for each observation, followed by the product of those sums.

In the two-dimensional example, the correct calculation was $f(x_1;\theta)f(x_2;\theta)\approx0.008709$. The tempting alternative is

0.4\{0.159155(0.096532)\} +0.6\{0.061975(0.079577)\} \approx0.4(0.015364)+0.6(0.004932) \approx0.009105.

This is not the likelihood just derived. It uses one weighted componentwise product across the entire data set, instead of allowing each observation's density to be the mixture sum and then multiplying those independent-observation densities. In general, $\prod_i\sum_k a_{ik}$ is not equal to $\sum_k\prod_i a_{ik}$. A reliable check is to write the required operation in words: for each observation, sum the weighted component densities; across observations, multiply.

Key takeaway: A finite Gaussian mixture is a convex combination of valid multivariate Gaussian densities. For independent observations from that mixture, the observed-data likelihood is a product over observations of the weighted component-density sum, subject to nonnegative weights summing to one and positive-definite component covariance matrices.

Check Your Understanding

Use the definitions and likelihood construction above to answer these questions.

For a mixture with $K$ components, what restrictions must the weights satisfy, and what matrix condition must each component covariance satisfy?
Explain why a weighted sum of normalized component densities integrates to one under the mixture-weight constraint.
For independent observations, which operation comes first in the observed-data likelihood: summing component densities or multiplying across observations?
Why is $\log\{\sum_k \pi_k\phi_d(x_i;\mu_k,\Sigma_k)\}$ generally not equal to a sum of component log-densities?
In one sentence, identify the specific error in replacing $\prod_i\sum_k a_{ik}$ with $\sum_k\prod_i a_{ik}$.