Normal Mixtures With Random Means and Variances

Normal Components, One Mixture Distribution

A normal model can describe variation within a relatively homogeneous group, but a population may combine groups with different centers or different spreads. The resulting distribution is a normal mixture: each component is normal, while the observed value comes from one component selected according to its mixing weight. The overall distribution need not be normal, and its shape depends on both the component distributions and their weights.

Represent the selected component by a latent label $Z$. Its possible values are $1,\ldots,k$, with $P(Z=j)=p_j$, where $p_j>0$ and $\sum_{j=1}^k p_j=1$. Given $Z=j$, suppose $X\sim N(\mu_j,\sigma_j^2)$, with $\sigma_j^2>0$. The component label determines a pair of parameters: a mean and a variance. In particular, the means and variances need not be constant across components, nor need the two lists of parameters be unrelated.

Definition: A finite normal mixture with weights $p_1,\ldots,p_k$, means $\mu_1,\ldots,\mu_k$, and variances $\sigma_1^2,\ldots,\sigma_k^2$ is the distribution of $X$ such that $P(Z=j)=p_j$ and $X\mid Z=j\sim N(\mu_j,\sigma_j^2)$. Its density is the weighted sum of its component densities.

Write $\phi_{\mu,\sigma^2}(x)$ for the $N(\mu,\sigma^2)$ density. Conditioning on the label gives the mixture density and distribution function:

f_X(x)=\sum_{j=1}^k p_j\phi_{\mu_j,\sigma_j^2}(x) =\sum_{j=1}^k p_j\frac{1}{\sqrt{2\pi\sigma_j^2}} \exp\!\left(-\frac{(x-\mu_j)^2}{2\sigma_j^2}\right), \qquad F_X(x)=\sum_{j=1}^k p_j\Phi\!\left(\frac{x-\mu_j}{\sigma_j}\right).

Here $\Phi$ is the standard normal distribution function. The density formula is valid because each component has a density and there are finitely many components. Its integral is $\sum_j p_j=1$, as required. Notice that the formula retains each mean and variance with its own weight; averaging parameters first does not in general give the mixture density.

Mean and Variance

The mean is the weighted average of the component means. The variance has two contributions: variation within components and variation among their means. These formulas require only finite component means and variances, which hold automatically for a finite mixture of nondegenerate normal distributions.

Theorem: For the finite normal mixture above, define $\bar{\mu}=\sum_{j=1}^k p_j\mu_j$. Then $$ E[X]=\bar{\mu}, \qquad \operatorname{Var}(X) =\sum_{j=1}^k p_j\sigma_j^2 +\sum_{j=1}^k p_j(\mu_j-\bar{\mu})^2. $$ The first variance term is the weighted average of component variances; the second is the weighted variance of component means.

Proof. Since the mixture has finitely many components and each normal component has finite first and second moments, expectations can be computed by conditioning on $Z$. Thus

E[X]=\sum_{j=1}^k P(Z=j)E[X\mid Z=j] =\sum_{j=1}^k p_j\mu_j =\bar{\mu}.

For the variance, condition on each component and write $X-\bar{\mu}=(X-\mu_j)+(\mu_j-\bar{\mu})$ when $Z=j$. Squaring and taking the conditional expectation yields

\begin{aligned} E[(X-\bar{\mu})^2\mid Z=j] &=E[(X-\mu_j)^2\mid Z=j] +2(\mu_j-\bar{\mu})E[X-\mu_j\mid Z=j] +(\mu_j-\bar{\mu})^2\\ &=\sigma_j^2+(\mu_j-\bar{\mu})^2, \end{aligned}

because $E[X-\mu_j\mid Z=j]=0$. Averaging over $Z$ gives

\operatorname{Var}(X) =E[(X-\bar{\mu})^2] =\sum_{j=1}^k p_j\{\sigma_j^2+(\mu_j-\bar{\mu})^2\}.

This is the stated decomposition. $\square$

Worked Example: Three Components With Different Means and Variances

Suppose a measurement comes from one of three normal components. The weights are $1/2,1/3,1/6$, the component means are $-2,1,5$, and the component variances are $1,4,9$, respectively. Find the density, mean, and variance.

The density is the weighted sum of the three normal densities:

f_X(x)= \frac12\frac{1}{\sqrt{2\pi}}\exp\!\left(-\frac{(x+2)^2}{2}\right) +\frac13\frac{1}{\sqrt{8\pi}}\exp\!\left(-\frac{(x-1)^2}{8}\right) +\frac16\frac{1}{\sqrt{18\pi}}\exp\!\left(-\frac{(x-5)^2}{18}\right).

The weights sum to $1/2+1/3+1/6=1$. The mixture mean is

\bar{\mu} =\frac12(-2)+\frac13(1)+\frac16(5) =-1+\frac13+\frac56 =\frac16.

The within-component contribution to the variance is

\frac12(1)+\frac13(4)+\frac16(9) =\frac12+\frac43+\frac32 =\frac{10}{3}.

For the between-means contribution, the deviations from $1/6$ are $-13/6$, $5/6$, and $29/6$. Therefore,

\begin{aligned} \sum_{j=1}^3 p_j(\mu_j-\bar{\mu})^2 &=\frac12\left(-\frac{13}{6}\right)^2 +\frac13\left(\frac56\right)^2 +\frac16\left(\frac{29}{6}\right)^2\\ &=\frac{169}{72}+\frac{25}{108}+\frac{841}{216}\\ &=\frac{507}{216}+\frac{50}{216}+\frac{841}{216} =\frac{233}{36}. \end{aligned}

Combining the two contributions gives

\operatorname{Var}(X)=\frac{10}{3}+\frac{233}{36} =\frac{120}{36}+\frac{233}{36} =\frac{353}{36}.

As a check, $E[X^2]$ is the weighted average of component second moments $E[X^2\mid Z=j]=\sigma_j^2+\mu_j^2$:

E[X^2]=\frac12(1+4)+\frac13(4+1)+\frac16(9+25) =\frac52+\frac53+\frac{17}{3} =\frac{59}{6}.

Hence $E[X^2]-E[X]^2=59/6-1/36=353/36$, agreeing with the decomposition.

Random Variances and Distributional Shape

Even if all components have the same mean, differing variances change the shape of the density. The following calculation also illustrates why replacing a random variance by its average does not usually preserve the distribution.

Worked Example: A Mixture With Random Variance

Let $P(Z=1)=3/4$, $P(Z=2)=1/4$, and suppose $X\mid Z=1\sim N(0,1)$, while $X\mid Z=2\sim N(0,9)$. Then the density is

f_X(x)=\frac34\frac{1}{\sqrt{2\pi}}e^{-x^2/2} +\frac14\frac{1}{\sqrt{18\pi}}e^{-x^2/18}.

Both component means are zero, so $E[X]=0$. The variance is

\operatorname{Var}(X)=\frac34(1)+\frac14(9)=3.

This does not make the mixture $N(0,3)$. At zero, the mixture density is

f_X(0)=\frac{3}{4\sqrt{2\pi}}+\frac{1}{12\sqrt{2\pi}} =\frac{5}{6\sqrt{2\pi}},

whereas the $N(0,3)$ density at zero is $1/\sqrt{6\pi}=1/(\sqrt{3}\sqrt{2\pi})$. These values differ because $5/6\ne 1/\sqrt{3}$, as is confirmed by squaring the positive quantities: $25/36\ne 1/3$. Thus the matching mean and variance do not make the distributions identical.

A central feature of normal mixtures is that combining normal components need not preserve the single-peaked shape of an individual normal density. A simple curvature calculation gives a useful sufficient condition for two modes.

Theorem: Consider the equally weighted mixture $\frac12N(-a,s^2)+\frac12N(a,s^2)$, where $a>0$ and $s>0$. If $a>s$, its density has at least two local maxima.

Proof. Let $\phi_s(u)=(s\sqrt{2\pi})^{-1}\exp(-u^2/(2s^2))$. The mixture density is

f(x)=\frac12\phi_s(x-a)+\frac12\phi_s(x+a).

It is symmetric about zero, so $f'(0)=0$. Differentiating the normal density twice gives

\phi_s''(u)=\left(\frac{u^2}{s^4}-\frac{1}{s^2}\right)\phi_s(u).

Since $\phi_s$ is even, $\phi_s''$ is even too. Consequently,

f''(0)=\phi_s''(a) =\left(\frac{a^2}{s^4}-\frac{1}{s^2}\right)\phi_s(a)>0

when $a>s$. Thus zero is a strict local minimum. To see that there is a local maximum on the positive side, choose a sufficiently small $\epsilon>0$ so that $f(\epsilon)>f(0)$, which is possible because zero is a strict local minimum. Since $f(x)\to0$ as $x\to\infty$, choose $R>\epsilon$ with $f(R)<f(0)$. The continuous function $f$ attains a maximum on $[0,R]$; neither endpoint can be a maximizer because $f(\epsilon)>f(0)$ and $f(R)<f(0)$. A maximizer in $(0,R)$ is therefore a local maximum. By symmetry there is also a local maximum at its negative. Hence the density has at least two local maxima. $\square$

Worked Example: Separated Means Produce Two Modes

Take an equally weighted mixture of $N(-2,1)$ and $N(2,1)$. Here $a=2$ and $s=1$, so $a>s$ and the theorem applies. Explicitly,

f(x)=\frac12\phi(x+2)+\frac12\phi(x-2),

where $\phi$ is the standard normal density. The mixture is symmetric and has mean zero. Its variance is $1+4=5$: the within-component variance is $1$, and the variance of the two means $-2$ and $2$ is $4$. Yet the density has a local minimum at zero, since

f''(0)=\phi''(2)=(2^2-1)\phi(2)=3\phi(2)>0.

So the mean lies between the two modes rather than at a mode. Each component is a single-peaked normal density, but their separated centers give the mixture two peaks.

Interpretation and a Common Pitfall

The mean and variance summarize location and overall spread, but they do not describe the full shape of a mixture. The within-component term reflects variation among observations assigned to the same component. The between-means term reflects separation among component centers. When component means are far apart relative to their standard deviations, a density can display distinct peaks; when variances differ, a narrow component can contribute a sharp peak while a wider component spreads mass over a broader range. The component weights also matter: a small-weight component can still have a visible influence on the density.

Common mistake: using only the weighted average of the component variances as the mixture variance. That calculation omits variation among the component means. In the three-component example, the within-component contribution is $10/3$, but the full variance is $353/36$, because the between-means contribution $233/36$ is substantial. The correction is to include both terms in the variance decomposition. A related error is to assume that a mixture is normal with the weighted-average mean and variance. Those moments are valid summaries, but they do not determine the mixture's density or modality.

Key takeaway: A finite normal mixture keeps the component densities distinct: its density is their weighted sum, its mean is the weighted mean of component means, and its variance combines average within-component variance with between-component variation in means.

Check Your Understanding

Use the component weights and parameters in each question, and distinguish mixture moments from the full shape of the distribution.

A mixture has weights $1/4$ and $3/4$, component means $0$ and $4$, and component variances $1$ and $1$. Find its mean and variance.
Write the density for weights $2/3$ and $1/3$ with components $N(1,1)$ and $N(1,4)$.
Why does a mixture with all component means equal have no between-means contribution to its variance?
For the equally weighted mixture $\frac12N(-3,1)+\frac12N(3,1)$, determine whether the curvature criterion guarantees at least two local maxima.
Explain why knowing the mean and variance of a normal mixture does not, by itself, establish that the mixture is normal.

Normal Mixtures With Random Means and Variances

What You'll Learn