Integrating Out Latent Variables in Likelihoods

Why Latent Variables Create Likelihood Integrals

A likelihood is built from the probability mass or density of the data that were actually observed. In many econometric models, however, the distribution of an observed outcome depends on an unobserved quantity: an individual-specific effect, a latent state, or an unrecorded shock. The likelihood for the observed data must account for that quantity without pretending it was observed. This requires summing or integrating over its possible values.

Let $y=(y_1,\ldots,y_n)$ denote observed outcomes, let $x$ collect observed covariates, and let $\theta$ denote the model parameters. Write $w$ for the latent variable or collection of latent variables. Suppose the model specifies a conditional mass or density $f(y\mid w,x;\theta)$ and a latent-variable mass or density $g(w\mid x;\theta)$. The observed-data likelihood is obtained by averaging the conditional likelihood over the model distribution of $w$.

Definition: The observed-data likelihood is the joint probability mass or density of the observed outcomes, conditional on the observed covariates, as a function of $\theta$. When the model includes a latent variable $W$, it is obtained by marginalizing the complete-data likelihood over the distribution of $W$. For a continuous $W$, this is an integral; for a discrete $W$, it is a sum.

For a continuous latent variable, the general expression is

L(\theta;y\mid x) = \int f(y\mid w,x;\theta)\,g(w\mid x;\theta)\,dw.

For a discrete latent variable with possible values $w_1,\ldots,w_K$, the corresponding expression is

L(\theta;y\mid x) = \sum_{k=1}^{K} f(y\mid w_k,x;\theta)\,P(W=w_k\mid x;\theta).

These formulas follow from the law of total probability: the observed-data probability or density is the conditional probability or density given each possible latent value, averaged using the latent variable’s probability distribution. The result is a likelihood for the observed data alone. It is not the likelihood conditional on one guessed or estimated value of $w$.

From a Complete-Data Model to the Observed Likelihood

The complete-data likelihood treats both $y$ and $w$ as available. Its joint density or mass factors as $f(y\mid w,x;\theta)g(w\mid x;\theta)$. Integrating this expression over $w$ removes the latent quantity and gives the observed-data likelihood. The factor $g$ must be included: omitting it would average conditional likelihoods without using the model’s probabilities for the latent values.

Even when the conditional density and latent-variable distribution are simple, their product may not have an integral with a convenient closed form. With many latent variables, the integral can also be high-dimensional. In those cases, a direct likelihood evaluation may be computationally difficult. A simulated approximation replaces the integral with an average over draws from the latent distribution.

Definition: Given draws $w^{(1)},\ldots,w^{(R)}$ from $g(\cdot\mid x;\theta)$, a basic simulated approximation to the observed-data likelihood is $$ \widetilde L_R(\theta;y\mid x) = \frac{1}{R}\sum_{r=1}^{R} f(y\mid w^{(r)},x;\theta). $$ Each summand is the joint conditional likelihood of the observed data at one latent draw. When the latent distribution depends on $\theta$, the draws must represent that distribution at the parameter value being evaluated, or be adjusted appropriately.

This expression approximates the likelihood integral; it does not change which likelihood the model defines. Its accuracy depends on the draws and their number, issues considered in detail in Monte Carlo integration. One important distinction is that the average is formed on the likelihood scale. In general, the logarithm of this average is not the average of the conditional log likelihoods.

Worked Example: Marginalizing a Discrete Latent Type

Suppose one binary outcome $Y$ depends on an unobserved type $W$, which takes values 0 and 1 with probability one-half each. The conditional success probabilities are $P(Y=1\mid W=0)=0.2$ and $P(Y=1\mid W=1)=0.8$. A hypothetical observation is $y=1$. Find its observed-data likelihood under these specified probabilities.

The conditional likelihood for the observed success is 0.2 for type 0 and 0.8 for type 1. The observed-data likelihood sums these conditional likelihoods weighted by the type probabilities:

L = P(Y=1\mid W=0)P(W=0) + P(Y=1\mid W=1)P(W=1) = (0.2)(0.5)+(0.8)(0.5) = 0.10+0.40 = 0.50.

Thus, the probability of observing a success after accounting for the unknown type is 0.50. Using only the conditional likelihood at $W=0$, for instance, would give 0.2, but that would answer a different question: the likelihood conditional on knowing the latent type was 0.

Independent Latent Variables and Shared Latent Variables

The structure of the latent variables determines the form of the observed likelihood. If each observation has its own latent variable and the model makes observations independent conditional on their respective latent variables, the integration can separate into a product of individual integrals. For independent $W_i$ with densities $g_i$, and conditional densities $f_i$, the model gives

L(\theta;y\mid x) = \prod_{i=1}^{n} \int f_i(y_i\mid w_i,x_i;\theta)\, g_i(w_i\mid x_i;\theta)\,dw_i.

This factorization depends on the stated independence structure: the $W_i$ are independent across observations, and the conditional outcome model factors across observations given their latent variables. If the latent variables are dependent, their joint distribution and the joint conditional likelihood must be used instead.

A shared latent variable $W$, such as a common market-level shock, produces a different structure. In general, the observed-data likelihood is

L(\theta;y\mid x) = \int f(y_1,\ldots,y_n\mid w,x_1,\ldots,x_n;\theta)\, g(w\mid x_1,\ldots,x_n;\theta)\,dw.

The joint conditional likelihood inside the integral is essential. If the observations are conditionally independent given $W$, then and only then may it be written as the product of individual conditional likelihoods:

f(y_1,\ldots,y_n\mid w,x_1,\ldots,x_n;\theta) = \prod_{i=1}^{n} f_i(y_i\mid w,x_i;\theta).

Under that assumption, a simulated approximation uses the product at each shared draw, then averages those products. Averaging separate marginal likelihoods and multiplying them instead would generally treat the shared latent variable as if it were independently redrawn for every observation.

Worked Example: A Continuous Latent Variable and a Simulated Likelihood

Consider a hypothetical model with one observed outcome $Y$. Conditional on a latent variable $U=u$, suppose $Y\sim N(u,1)$, while $U\sim N(0,1)$. The observed value is $y=0.5$. The conditional density is

f(0.5\mid u) = \frac{1}{\sqrt{2\pi}} \exp\!\left[-\frac{(0.5-u)^2}{2}\right].

The observed-data likelihood is the integral of this conditional density against the standard normal density of $U$:

L = \int_{-\infty}^{\infty} \frac{1}{\sqrt{2\pi}}\exp\!\left[-\frac{(0.5-u)^2}{2}\right] \frac{1}{\sqrt{2\pi}}\exp\!\left[-\frac{u^2}{2}\right]\,du.

For this particular normal model, the sum $Y=U+\varepsilon$, with independent $U\sim N(0,1)$ and $\varepsilon\sim N(0,1)$, has distribution $N(0,2)$. Therefore the exact marginal density at 0.5 is

L = \frac{1}{\sqrt{4\pi}}\exp\!\left(-\frac{0.5^2}{4}\right) \approx 0.2650.

To illustrate the simulated approximation, suppose three draws from the latent distribution, rounded to one decimal place, are $-1.0,\ 0.0,\ 1.0$. The conditional density contributions are

f(0.5\mid -1.0)\approx 0.1295,\qquad f(0.5\mid 0.0)\approx 0.3521,\qquad f(0.5\mid 1.0)\approx 0.3521.

Averaging those contributions gives

\widetilde L_3 = \frac{0.1295+0.3521+0.3521}{3} = \frac{0.8337}{3} \approx 0.2779.

The simulated value, 0.2779, is an approximation to the exact likelihood, 0.2650; the discrepancy is possible with only three draws. The likelihood is a density here, so its numerical value is not itself a probability that $Y$ equals exactly 0.5.

Worked Example: Simulating a Likelihood with a Shared Latent Variable

Suppose two binary outcomes $Y_1,Y_2$ share a latent variable $W$, with $P(W=0)=P(W=1)=0.5$. Given $W$, assume the outcomes are conditionally independent. Their success probabilities are 0.2 when $W=0$ and 0.8 when $W=1$. The observed pair is $(y_1,y_2)=(1,0)$. Compute the observed-data likelihood and a simulated approximation from four draws.

Conditional independence gives the joint conditional likelihood of the observed pair. If $W=0$, the probability of success then failure is $(0.2)(1-0.2)=0.16$. If $W=1$, it is $(0.8)(1-0.8)=0.16$. Marginalizing over the shared type gives

$$ L = (0.5)(0.2)(0.8)+(0.5)(0.8)(0.2) = 0.08+0.08 = 0.16. $$

Now suppose four draws from the shared latent distribution are $0,1,0,1$. The joint conditional likelihood is 0.16 at every draw, so the simulated likelihood is

\widetilde L_4 = \frac{1}{4}(0.16+0.16+0.16+0.16) = 0.16.

For comparison, the marginal success probability for either individual outcome is 0.5, and its marginal failure probability is also 0.5. Multiplying marginal probabilities would give $(0.5)(0.5)=0.25$, not 0.16. That product incorrectly treats the outcomes as independent after marginalizing out their shared $W$. Conditional independence given a shared latent variable does not generally imply marginal independence.

Why the Order of Operations Matters

The likelihood must be marginalized before taking its logarithm. For a given draw set, the simulated log likelihood contribution is

\log \widetilde L_R(\theta;y\mid x) = \log\!\left[ \frac{1}{R}\sum_{r=1}^{R} f(y\mid w^{(r)},x;\theta) \right].

This is generally not equal to the average of the conditional log likelihoods, $R^{-1}\sum_{r=1}^{R}\log f(y\mid w^{(r)},x;\theta)$. The latter averages logs of conditional contributions rather than approximating the log of the marginalized likelihood. The distinction matters because maximizing these different expressions can produce different parameter estimates.

A second practical issue is that a likelihood can be very small, especially when it is a product across many observations. Computations may then use log scales, but the operation represented must still correspond to the marginalized likelihood. In particular, one should preserve the sum or average over latent draws inside the logarithm.

Common Mistakes

Conditioning on a latent value as if it were observed. Substituting one value of $w$ gives a conditional likelihood. The observed-data likelihood instead sums or integrates over all possible values using $g$.
Dropping the latent distribution. An integral of $f(y\mid w,x;\theta)$ alone is not generally the observed likelihood. The integrand must be weighted by $g(w\mid x;\theta)$.
Multiplying conditional densities without checking dependence. For a shared latent variable, the valid general expression uses the joint conditional likelihood. A product of individual terms is justified only when the observations are conditionally independent given that shared variable.
Marginalizing shared effects separately for each observation. This replaces one common latent draw with separate draws and changes the model’s dependence structure.
Taking logs before averaging. Averaging conditional log likelihoods does not in general give the log of the observed-data likelihood.

The central modeling task is to identify precisely what is latent and how it is shared across observations. Once that structure is specified, the observed-data likelihood follows by marginalization. Simulation can approximate the resulting sum or integral, but it must approximate the correct joint likelihood under the stated model.

Check Your Understanding

Use the definitions and distinctions above to check your understanding.

In one sentence, explain why an observed-data likelihood averages conditional likelihoods over the latent-variable distribution.
A latent variable takes three possible values. What replaces an integral in the observed-data likelihood, and what weights each conditional likelihood?
For a shared continuous latent variable, write the general simulated likelihood using the joint conditional density. What additional assumption permits replacing that joint density by a product?
Why can multiplying separately marginalized likelihood contributions give the wrong answer when observations share a latent variable?
Explain why the log of an average of conditional likelihoods is generally different from the average of their logarithms.