Heavy-Tailed Weights and Defensive Importance Proposals

Why Proposal Tails Matter

Importance sampling can estimate an NLME marginal likelihood when the random-effects integral is difficult to evaluate directly. Its reliability depends not only on whether the proposal places draws near the important part of the integrand, but also on whether it covers the integrand’s tails. A proposal that is too concentrated can produce occasional enormous weights; in some cases, the weights have infinite variance, so the usual Monte Carlo standard error is not valid.

For one individual, fix the population parameters and write the random effect as $b$. The nonnegative integrand and marginal likelihood are

h(b)=p(y\mid b)\,p(b), \qquad L=\int h(b)\,db.

Here $p(y\mid b)$ is the conditional data density and $p(b)$ is the random-effects density. Let $q(b)$ be a proposal density. It must be positive wherever $h(b)$ is positive, apart from sets of measure zero. For a draw $b\sim q$, the importance weight is $W=h(b)/q(b)$, and $E_q(W)=L$. The average of independent weights is therefore an unbiased estimator of $L$, provided the integral is finite and the support condition holds.

Definition: The second moment of an importance weight is $E_q(W^2)=\int h(b)^2/q(b)\,db$. When this integral is finite, the weight variance is $\operatorname{Var}_q(W)=\int h(b)^2/q(b)\,db-L^2$. If it is infinite, the estimator can remain unbiased, but it does not have a finite variance for the usual Monte Carlo standard-error calculation.

The denominator $q(b)$ explains the tail risk. In regions where the integrand remains appreciable, a very small proposal density makes $h(b)/q(b)$ large. A proposal can look adequate around the mode and still fail far from it, where a few rare draws dominate the estimate.

A Tail Condition for Finite Variance

The finite-variance condition can be read directly from the second-moment integral. The relevant comparison is not simply whether $q$ has full support: the rate at which $q$ decays must be sufficient relative to $h^2$. The following Gaussian example makes this distinction explicit.

Worked Example: A Gaussian Proposal With Infinite Weight Variance

Suppose the target integrand is the standard normal density $h(b)=\phi(b)$, so its integral is $L=1$. Use a centered Gaussian proposal $q(b)=N(0,s^2)$. For this example, assume both densities are defined on the entire real line.

The densities are $\phi(b)=(2\pi)^{-1/2}\exp(-b^2/2)$ and $q(b)=(2\pi s^2)^{-1/2}\exp[-b^2/(2s^2)]$. Thus

\frac{h(b)^2}{q(b)} =\frac{s}{\sqrt{2\pi}} \exp\left[-b^2+\frac{b^2}{2s^2}\right] =\frac{s}{\sqrt{2\pi}} \exp\left[-b^2\left(1-\frac{1}{2s^2}\right)\right].

The integral over $b$ is finite only when the coefficient $1-1/(2s^2)$ is positive, or $s^2>1/2$. At $s^2=1/2$, the integrand is a positive constant and its integral over the real line diverges. If $s^2<1/2$, it grows in the tails and also has an infinite integral.

For example, take $s^2=0.25$. Then the expression is $0.25^{1/2}/\sqrt{2\pi}\,\exp(b^2)$, whose integral is infinite. The proposal has full support, and its center matches the target’s center, but its tails are too light to give finite weight variance. A proposal with $s^2=1$ matches this target exactly and gives constant weights $W=1$, hence zero variance.

This calculation illustrates a general rule: $E_q(W^2)$ is finite exactly when $\int h^2/q$ is finite. A quick visual comparison of tail widths is useful, but the integral condition is the precise criterion. Having support wherever the target is nonzero is necessary for unbiasedness; it is not sufficient for finite variance.

Defensive Mixture Proposals

A defensive proposal combines a proposal tailored to the region of high integrand values with a component chosen to protect tail coverage. Let $q_0(b)$ be the tailored proposal, $r(b)$ a protective density, and $0<\epsilon\leq 1$. Define

q_{\epsilon}(b)=(1-\epsilon)q_0(b)+\epsilon r(b).

The mixture must be sampled as a mixture and its full density $q_{\epsilon}(b)$ must be used in the weight. Using only the density of the component that generated a draw gives the wrong importance weight. The defensive component should cover the support of $h$; for a useful variance guarantee, it should also have tails appropriate to the integrand.

Theorem: If $q_{\epsilon}(b)\geq\epsilon r(b)$ and $\int h(b)^2/r(b)\,db$ is finite, then $$ E_{q_{\epsilon}}(W^2)=\int\frac{h(b)^2}{q_{\epsilon}(b)}\,db \leq \frac{1}{\epsilon}\int\frac{h(b)^2}{r(b)}\,db. $$ In particular, if $h(b)/r(b)\leq C$ wherever $h(b)>0$, then $E_{q_{\epsilon}}(W^2)\leq CL/\epsilon$, and the weight variance is finite.

The first inequality follows because $q_{\epsilon}\geq\epsilon r$, so $h^2/q_{\epsilon}\leq h^2/(\epsilon r)$; integration preserves the inequality. For the second bound, write $h^2/r=h(h/r)\leq Ch$ and integrate to obtain $\int h^2/r\leq CL$. The bound may be conservative, but it establishes a finite second moment under its stated condition.

Worked Example: Bounding Variance With a Broad Defensive Component

Continue with $h(b)=\phi(b)$, and let $q_0=N(0,0.25)$, which gave infinite variance in the previous example. Add a defensive component $r=N(0,4)$ with $\epsilon=0.10$. Assume the proposal density is the full mixture $q_{\epsilon}=0.90q_0+0.10r$.

Since $q_{\epsilon}\geq0.10r$, the second moment is bounded by $10\int\phi(b)^2/r(b)\,db$. For the stated densities,

\int\frac{\phi(b)^2}{r(b)}\,db =\frac{\sqrt{8\pi}}{2\pi}\int \exp\left(-\frac{7b^2}{8}\right)\,db =\frac{4}{\sqrt{7}} \approx 1.512.

Consequently, $E_{q_{\epsilon}}(W^2)\leq 10(1.512)=15.12$. Because $L=1$, the weight variance is at most $15.12-1=14.12$. This is a guaranteed upper bound, not an estimate of the mixture’s actual variance. The broad component makes the second moment finite even though the narrow component alone does not.

In NLME likelihood calculations, a natural defensive component is sometimes the random-effects prior itself. If $r(b)=p(b)$, then $h(b)/r(b)=p(y\mid b)$. When the conditional likelihood is bounded above by $M$, the theorem gives a finite second moment bounded by $ML/\epsilon$. For example, a normal observation density with fixed positive residual variance has a finite maximum. This guarantee is conditional on the model and fixed parameter values; it does not automatically apply if the likelihood is unbounded or the prior does not cover the target support.

Checking a Defensive Proposal in an NLME Calculation

A defensive component should be selected in relation to the integrand, not merely described as “wide.” A broad Gaussian may still have lighter tails than a heavy-tailed target. A Student-$t$ component can provide heavier tails than a Gaussian, while the prior component can be attractive when the likelihood is bounded. The density used in the weight must always be the density of the actual mixture.

Worked Example: Comparing Weights at a Tail Point

Consider a hypothetical one-dimensional NLME random effect with prior $p(b)=N(0,1)$ and conditional likelihood $p(y\mid b)=\exp[-(b-1)^2/(2(0.5)^2)]$. This likelihood is bounded above by 1. Use $q_0=N(0,0.25)$ and mix it with the prior using $\epsilon=0.10$. Compare the weights at $b=2$. All densities and weights here are for this hypothetical calculation.

At $b=2$, the likelihood is $\exp(-2)\approx0.1353$, and the prior density is $\phi(2)\approx0.05399$. Thus the integrand is $h(2)=0.05399(0.1353)\approx0.007306$. The narrow proposal density is $q_0(2)=[1/(0.5\sqrt{2\pi})]\exp(-8)\approx0.0002677$, whereas the prior density used as the defensive component is $r(2)=0.05399$.

The full mixture density and weight are therefore

q_{\epsilon}(2)=0.90(0.0002677)+0.10(0.05399)\approx0.005640, \qquad W_{\epsilon}(2)=\frac{0.007306}{0.005640}\approx1.295.

With the narrow proposal alone, the weight at the same point would be $0.007306/0.0002677\approx27.29$. The mixture reduces this tail weight substantially. Moreover, because $h(b)/r(b)=p(y\mid b)\leq1$, the general bound gives $W_{\epsilon}(b)\leq1/0.10=10$ for every $b$. This bound applies to the full weight, not just to the value at $b=2$.

Diagnostics, Alternatives, and Limitations

Useful diagnostics include the largest weights, the distribution of log weights, the effective sample size, and whether likelihood estimates change substantially across independent runs. Plotting or tabulating weights against random-effect values can reveal that large weights arise in a tail region. Repeating the calculation with a heavier-tailed defensive component is also informative. These checks can expose a problem, but a finite simulation cannot prove that the theoretical second moment is finite: an extremely rare region may not have been sampled.

The main alternatives have different trade-offs:

Proposal choice	Potential advantage	Main concern
Single Gaussian centered near a conditional mode	Can place many draws in a concentrated, influential region.	May be too narrow in the tails or miss other important regions.
Broad or heavy-tailed proposal	Can improve coverage of distant random-effect values.	May spend many draws where the integrand is negligible, increasing weight variability elsewhere.
Defensive mixture	Retains a tailored component while ensuring a chosen component contributes everywhere it has support.	Its guarantee depends on the protective density and can be loose; an unsuitable component may not control the second moment.

A large estimated effective sample size does not establish finite variance, and a small observed maximum weight does not establish that the tails are safe. Weight truncation or clipping can reduce the influence of extreme observations, but it changes the estimator and generally introduces bias. It is not a substitute for choosing a proposal with suitable tail coverage. Nor does a variance guarantee ensure a practically precise estimate: the upper bound may be large, and finite samples can still be noisy.

The defensive mixture is most useful when the tailored proposal is efficient near the main mass but its tail behavior is uncertain. The value of $\epsilon$ trades protection against focus: increasing $\epsilon$ gives more draws from the defensive component, while decreasing it leaves more draws for $q_0$. Neither the mixture nor its bound removes other challenges, such as high-dimensional random effects or a target with multiple important regions.

Common pitfall: “The proposal has full support, so the importance weights must have finite variance.” Full support only prevents omission of positive-integrand regions. The correct check is whether $\int h^2/q$ is finite. A defensive mixture helps only when its protective component makes this integral finite; otherwise, changing the mixture proportion alone does not guarantee stability.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

Why can a full-support Gaussian proposal still produce infinite-variance weights?
For a defensive mixture, why must the weight denominator be the full mixture density rather than the density of the component that generated the draw?
If $h/r\leq 3$, $L=2$, and $\epsilon=0.20$, give the theorem’s upper bound on the second moment.
Does a finite observed maximum weight establish that the theoretical weight variance is finite? Explain.
What is the statistical cost of clipping extreme importance weights?

Solutions. (1) Full support does not control the tail rate; the integral $\int h^2/q$ can diverge when $q$ decays too quickly. (2) The sampling distribution is the mixture, so the unbiased importance weight is $h/q_{\epsilon}$; using a component density instead gives a different, generally invalid weight. (3) The bound is $CL/\epsilon=3(2)/0.20=30$. (4) No. A finite sample may not include rare tail draws that determine whether the second moment is finite. (5) Clipping changes the estimator and generally introduces bias, even if it reduces the influence of extreme weights.