Continuous Mixing and Mixture Densities

Why Mix Over a Continuous Parameter?

A distribution may describe observations whose behavior depends on an unobserved or varying parameter. If that parameter is itself random, the distribution of the observation averages the conditional laws over the parameter’s distribution. Unlike a finite mixture, this averaging uses a continuum of possible parameter values. The result can have a different density, support, or tail behavior from any single conditional law.

The key operation is integration with respect to the mixing distribution. It applies whether the conditional laws are continuous, discrete, or contain both continuous and discrete parts. The resulting distribution need not have a density just because the mixing parameter does.

Definition and Basic Construction

Definition: Let $\Theta$ be a random mixing parameter with distribution $G$. Given $\Theta=\theta$, let $X$ have conditional distribution $K(\theta,\cdot)$, where $K$ is a probability kernel: for each $\theta$, $K(\theta,\cdot)$ is a probability distribution, and for each measurable set $A$, $K(\cdot,A)$ is measurable. The resulting marginal distribution of $X$ is the continuous mixture obtained by averaging the conditional laws over $G$: $$ P(X\in A)=\int K(\theta,A)\,G(d\theta). $$ If $G$ has density $g$, this becomes $P(X\in A)=\int K(\theta,A)g(\theta)\,d\theta$.

The word “continuous” describes the mixing over a continuously distributed parameter; the mixture itself is not necessarily continuous. For the conditional CDF $F_\theta(x)=P(X\leq x\mid\Theta=\theta)$, the marginal CDF is $$ F_X(x)=\int F_\theta(x)\,G(d\theta). $$

Theorem (Mixture CDF and density): For any measurable conditional probability kernel, the CDF of the mixture is $F_X(x)=\int F_\theta(x)\,G(d\theta)$. If, in addition, every conditional law has a density $f_\theta$ with respect to Lebesgue measure, and $(\theta,x)\mapsto f_\theta(x)$ is jointly measurable, then the mixture has density $$ f_X(x)=\int f_\theta(x)\,G(d\theta) $$ for almost every $x$.

To prove the CDF formula, apply the definition of the mixture to the measurable set $(-\infty,x]$: $$ P(X\leq x)=\int K(\theta,(-\infty,x])\,G(d\theta) =\int F_\theta(x)\,G(d\theta). $$ For the density claim, let $A$ be any Borel set. Since $f_\theta$ is a conditional density, $$ P(X\in A)=\int\left(\int_A f_\theta(x)\,dx\right)G(d\theta). $$ The integrand is nonnegative, so Tonelli’s theorem permits reversing the order of integration: $$ P(X\in A)=\int_A\left(\int f_\theta(x)\,G(d\theta)\right)dx. $$ Thus $x\mapsto\int f_\theta(x)\,G(d\theta)$ is a density of the marginal law. It integrates to one because another application of Tonelli gives $$ \int\left(\int f_\theta(x)\,G(d\theta)\right)dx =\int\left(\int f_\theta(x)\,dx\right)G(d\theta) =\int 1\,G(d\theta)=1. $$ This proves both formulas under the stated conditions. $\square$

Worked Examples

Worked Example: Mixing Uniform Distributions Extends the Support

Let $\Theta\sim\operatorname{Uniform}(1,3)$, and conditionally on $\Theta=\theta$, let $X\sim\operatorname{Uniform}(0,\theta)$. Find the marginal density and CDF.

The mixing density is $g(\theta)=1/2$ for $1<\theta<3$. Given $\theta$, the conditional density is $f_\theta(x)=1/\theta$ for $0<x<\theta$, and zero otherwise. For $x$ to be in the conditional support, $\theta$ must exceed $x$. Therefore $$ f_X(x)=\frac12\int_{\max(1,x)}^3\frac{1}{\theta}\,d\theta $$ when $0<x<3$, and the density is zero outside that range. Splitting at $x=1$ gives $$ f_X(x)= \begin{cases} \frac12\log 3,&0<x<1,\\[2pt] \frac12\log(3/x),&1\leq x<3,\\[2pt] 0,&\text{otherwise}. \end{cases} $$ For a normalization check, its integral is $$ \frac12\log 3+\frac12\int_1^3\log(3/x)\,dx =\frac12\log 3+\frac12(2-\log 3)=1. $$ Integrating the density gives $F_X(x)=0$ for $x<0$, $F_X(x)=\tfrac12x\log 3$ for $0\leq x<1$, and $F_X(x)=\tfrac12[x\log(3/x)+x-1]$ for $1\leq x<3$. For $x\geq3$, $F_X(x)=1$. The support reaches 3 because component distributions with $\theta$ arbitrarily close to 3 assign positive probability to values near 3; smaller values of $\theta$ have shorter supports.

Worked Example: A Mixture Can Have an Atom

Let $\Theta\sim\operatorname{Uniform}(0,1)$. Given $\Theta=\theta$, let $X=0$ with probability $\theta$; with the remaining probability $1-\theta$, let $X\sim\operatorname{Uniform}(0,1)$. Find the marginal distribution.

The probability of the atom at zero is the average of its conditional probability: $$ P(X=0)=\int_0^1\theta\,d\theta=\frac12. $$ For $0<x<1$, the conditional density contributed by the uniform part is $1-\theta$. Mixing gives $$ f_X(x)=\int_0^1(1-\theta)\,d\theta=\frac12,\qquad 0<x<1. $$ This density accounts for total probability $\int_0^1(1/2)\,dx=1/2$, while the atom accounts for the other half. In particular, the CDF is $0$ for $x<0$, equals $\tfrac12+\tfrac12x$ for $0\leq x<1$, and is $1$ for $x\geq1$. It jumps by $1/2$ at zero. Thus a continuously distributed mixing parameter does not guarantee that the mixture has a density: here, the conditional point mass at zero survives averaging.

Atom formula: For any fixed point $a$, the mixture’s probability at $a$ is $$ P(X=a)=\int P(X=a\mid\Theta=\theta)\,G(d\theta). $$ This follows by applying the mixture definition to the singleton set $\{a\}$. In particular, if every conditional law is absolutely continuous, then $P(X=a\mid\Theta=\theta)=0$ for every $a$ and almost every $\theta$, so the mixture has no atoms.

Worked Example: Mixing Exponential Rates Produces a Heavy Tail

Let $\Theta\sim\operatorname{Exponential}(\lambda)$, with $\lambda>0$, and conditionally on $\Theta=\theta$, let $X\sim\operatorname{Exponential}(\theta)$. Find the marginal density and survival probability.

The mixing density is $\lambda e^{-\lambda\theta}$ for $\theta>0$, and the conditional density is $\theta e^{-\theta x}$ for $x>0$. Hence, for $x>0$, $$ f_X(x)=\int_0^\infty \theta e^{-\theta x}\lambda e^{-\lambda\theta}\,d\theta =\lambda\int_0^\infty\theta e^{-(\lambda+x)\theta}\,d\theta =\frac{\lambda}{(\lambda+x)^2}. $$ The last integral equals $1/(\lambda+x)^2$, as follows by integrating $\theta e^{-c\theta}$ for $c=\lambda+x>0$. The density is normalized: $$ \int_0^\infty\frac{\lambda}{(\lambda+x)^2}\,dx=1. $$ Alternatively, mix the conditional survival probabilities: $$ P(X>x)=\int_0^\infty e^{-\theta x}\lambda e^{-\lambda\theta}\,d\theta =\frac{\lambda}{\lambda+x},\qquad x\geq0. $$ Thus $F_X(x)=x/(\lambda+x)$ for $x\geq0$, and $F_X(x)=0$ for $x<0$. Each conditional distribution has an exponential tail, but the mixture has survival probability asymptotic to $\lambda/x$ as $x$ grows. Small values of the random rate $\Theta$, which allow large values of $X$, produce this heavier tail.

Support and Tail Behavior

The mixture’s support reflects the conditional supports that receive mixing probability. A useful precise criterion is this: a point $x$ belongs to the support of the marginal law if every open neighborhood $U$ of $x$ has $$ P(X\in U)=\int K(\theta,U)\,G(d\theta)>0. $$ This need not be the union of every conditional support, because parameter values in a set of mixing probability zero do not contribute probability. In the uniform example, values approaching 3 occur for mixing parameters near 3, a set with positive probability, so the marginal support extends to 3 even though no single conditional distribution has $\theta=3$ with positive probability.

Tail behavior also depends on how the conditional distributions vary across the mixing distribution. A mixture can assign non-negligible probability to parameter values associated with much slower decay, as in the exponential-rate example. It is therefore unsafe to infer the marginal tail by examining a typical or average component alone. The CDF or density integral gives the relevant marginal probability directly.

Common Mistake: Assuming the Mixture Must Have a Density

A tempting but incorrect argument is: “The parameter has a density and the conditional distributions are being averaged, so the marginal distribution must have a density.” The atom example disproves this: a continuum of conditional laws each places some probability at exactly zero, and their integrated point mass is still $1/2$. A density formula obtained by integrating only the continuous parts describes only that part of the mixture.

The correction is to begin with the conditional probability kernel, or equivalently the conditional CDF, which handles continuous, discrete, and mixed conditional laws. Use the density-integral formula only after verifying that the conditional laws are absolutely continuous. If conditional atoms are present, calculate their marginal masses separately using the atom formula.

Key takeaway: A mixture distribution is formed by integrating conditional probabilities over the mixing law. The CDF formula always applies; the density formula requires conditional absolute continuity. Mixing can extend support, preserve or create atoms, and substantially change tail decay.

Check Your Understanding

Use the mixture formulas to answer the following questions.

If $F_\theta(x)$ is the conditional CDF and $G$ is the mixing distribution, write the marginal CDF of $X$.
What additional condition allows the marginal density to be written as an integral of conditional densities?
Suppose $P(X=a\mid\Theta=\theta)=q(\theta)$. Express $P(X=a)$ in terms of $q$ and $G$.
In the uniform-mixture example, why does the marginal support extend beyond the support of many individual components?
For the exponential-rate mixture, what is $P(X>x)$, and how does its tail decay compare with a fixed-rate exponential survival probability?