Importance Sampling for Likelihood Integrals

Why Reweight Simulated Draws?

A likelihood that depends on an unobserved variable often requires integrating that variable out. For example, a model may specify the distribution of a latent effect and the probability of the observed outcome conditional on that effect. The resulting likelihood is an average over the latent-effect distribution. Direct simulation from that distribution can be inefficient when only a small part of its range contributes appreciably to the integral.

Importance sampling addresses this problem by drawing from a different distribution, called a proposal distribution, and correcting each draw with a weight. A useful proposal concentrates draws in regions that matter to the integral. But if the proposal misses important regions or places too little probability there, a few large weights can dominate the estimate. The weighting identity makes the correction exact in expectation; diagnostics help determine whether it is useful in a particular simulation.

In a likelihood integral with latent variable \(u\), a common form is \(L(\theta)=\int p(y\mid u,\theta)f(u\mid\theta)\,du\). Here \(y\) denotes observed data, \(f(u\mid\theta)\) is the latent-variable density, and \(p(y\mid u,\theta)\) is the likelihood conditional on \(u\). The integrand is \(h(u)=p(y\mid u,\theta)f(u\mid\theta)\). A proposal \(q(u)\) then gives weight \(p(y\mid u,\theta)f(u\mid\theta)/q(u)\). The parameter \(\theta\) is held fixed while estimating this integral.

The Weighting Identity and Its Conditions

The identity follows by multiplying and dividing the integrand by the proposal density. The condition on support is essential: if \(h(z)>0\) in a region where \(q(z)=0\), no proposal draw can represent that region, and the integral cannot be recovered by reweighting.

Consequently, under the support condition and a finite integral, the sample average of the weights is unbiased: its expectation under repeated independent draws from \(q\) equals \(I\). To have finite variance, a stronger condition is needed: \(\int h(z)^2/q(z)\,dz<\infty\). When this holds, the variance of the estimate is the variance of one weight divided by \(N\). Thus an unbiased estimator can still be highly variable if the weights are dispersed.

Importance Sampling for a Latent-Variable Likelihood

The same calculation applies directly to a likelihood integral. The target integrand includes both the probability of the observed data conditional on the latent variable and the latent variable's model density. The proposal replaces the latter as the simulation distribution, but the ratio of target density to proposal density must remain in each weight.

Diagnosing Weight Dispersion

The weights provide a direct diagnostic of proposal performance. If they are nearly equal, each draw contributes similarly to the estimate. If one weight is much larger than the others, the estimate is sensitive to whether that draw appears in the simulation. Equal weights are not required for validity, but extreme dispersion can make a formally unbiased estimate practically unreliable.

A commonly used summary is the effective sample size (ESS) of the importance weights. For nonnegative weights \(w_1,\ldots,w_N\), define

The ESS is at most \(N\), and it equals \(N\) when all weights are equal. It approaches one when nearly all the total weight is concentrated in a single draw. It is a summary of weight concentration, not a literal count of independent observations and not, by itself, a confidence interval or a guarantee of accuracy.

Choosing and Using a Proposal

A good proposal places draws where the target integrand contributes substantially, while retaining enough probability in all regions needed for support and tail coverage. In a likelihood integral, this often means concentrating draws in latent-variable regions compatible with the observed data, rather than drawing only from the unconditional latent-variable distribution. The proposal may depend on the observed data and on the parameter value at which the likelihood is evaluated; the weight must use the actual proposal density that generated each draw.

There is no universal proposal that is best for every parameter value or data set. A proposal that works well for one likelihood evaluation can yield dispersed weights for another. A practical check is to inspect maximum normalized weight, ESS, and changes in estimates across independent batches or sensible alternative proposals. A low ESS or a single dominant weight is a warning to revise the proposal or increase simulation effort, not a mechanical rule that identifies the correction.

For very small likelihood contributions, it is useful to calculate log weights, \(\log w_i=\log h(Z_i)-\log q(Z_i)\), rather than forming tiny ratios directly. To sum positive weights stably, let \(a=\max_i\log w_i\); then \(\log\sum_i w_i=a+\log\sum_i\exp(\log w_i-a)\). The log of the estimated integral is this log-sum minus \(\log N\). In general, the logarithm of an unbiased likelihood estimate is not itself unbiased: because the logarithm is concave, Jensen's inequality gives \(E[\log\hat I_{\mathrm{IS}}]\leq\log I\), when the expectations exist. This distinction matters when estimates are passed into a log-likelihood calculation.

For ratios of integrals, a related estimator uses normalized importance weights. For example, if the target expectation is \(E_f[a(Z)]\), draw from \(q\), set \(w_i=f(Z_i)/q(Z_i)\), and estimate it by \(\sum_iw_i a(Z_i)/\sum_iw_i\). This self-normalized estimator is generally biased at finite \(N\), though it is consistent under suitable conditions. It should not be confused with the unbiased sample average used to estimate a single integral.

Common Mistakes and Key Takeaway

• Leaving out the correction. Averaging \(h(Z_i)\) when draws come from \(q\) generally estimates an integral under \(q\), not \(\int h(z)\,dz\). Use \(h(Z_i)/q(Z_i)\).
• Assuming more draws fix missing support. If \(q(z)=0\) where \(h(z)>0\), no number of draws from \(q\) can recover that part of the integral. Choose a proposal with adequate support.
• Treating a large ESS as proof of accuracy. ESS measures observed weight concentration. It cannot reveal an important region that the proposal never visits.
• Calling the realized estimate unbiased. Unbiasedness is a repeated-sampling property. A single importance-sampling estimate may be above or below the integral.