Importance Weights and Effective Sample Size

Why Look at the Weights?

In an NLME model, an individual likelihood contribution is an integral over that individual’s random effects. Importance sampling estimates the integral using draws from a proposal distribution, with each draw adjusted by an importance weight. The estimate can be computed even when the weights are very unequal, but then a few draws may dominate it. Looking at the weights helps reveal that concentration and assess whether the proposal is representing the important parts of the integral effectively.

The earlier material on the importance sampling identity gives the basis for the calculation: the integrand is divided by the proposal density, and its expectation under the proposal equals the desired integral when the proposal covers the target integrand. Here the focus is on the weights produced by a finite set of draws—not on proving that a particular finite-sample estimate is accurate.

Raw and Normalized Weights

For individual $i$, write the likelihood contribution as $L_i=\int g_i(b)\,db$, where $g_i(b)$ is the nonnegative integrand, such as the product of the individual’s conditional data density and random-effects density. Let $q_i(b)$ be a proposal density, and draw $b_{i1},\ldots,b_{iM}$ from it. At a draw where $q_i(b_{im})>0$, the raw importance weight is the integrand divided by the proposal density.

r_{im}=\frac{g_i(b_{im})}{q_i(b_{im})},\qquad m=1,\ldots,M.

The usual support condition is that $g_i(b)=0$ wherever $q_i(b)=0$, except possibly on a set of measure zero. This ensures the proposal does not omit a region with positive contribution to the integral. Raw weights are nonnegative. To define normalized weights for a finite draw set, however, the sampled raw weights must also have a positive sum. The support condition alone does not guarantee this: if $g_i$ is zero on part of the proposal’s support, all finite draws could land in that zero-integrand region.

Definition: Whenever $\sum_{m=1}^{M}r_{im}>0$, the normalized importance weight for draw $m$ is $$ \tilde{r}_{im}=\frac{r_{im}}{\sum_{k=1}^{M}r_{ik}}. $$ These normalized weights are nonnegative and sum to one. If the sampled raw-weight sum is zero, the normalized weights and the effective sample size below are undefined. A sufficient condition that prevents this particular problem is $g_i(b)>0$ for $q_i$-almost every $b$, with finite raw weights at the sampled points.

Normalizing removes an overall positive scale from the weights. It does not change their relative sizes. That makes normalized weights useful for describing how much each sampled point contributes relative to the other points in the same draw set.

Effective Sample Size

The effective sample size (ESS) summarizes the concentration of the normalized weights. It is defined only when the normalized weights exist. It is a descriptive weight diagnostic: it expresses how many equally weighted draws would have the same sum of squared normalized weights.

\operatorname{ESS}_i =\frac{1}{\sum_{m=1}^{M}\tilde{r}_{im}^{\,2}} =\frac{\left(\sum_{m=1}^{M}r_{im}\right)^2}{\sum_{m=1}^{M}r_{im}^{\,2}}.

The second expression follows by substituting the definition of $\tilde{r}_{im}$. The ESS is unchanged if every raw weight is multiplied by the same positive constant. It is not generally an integer, and it is not the number of genuinely independent draws from the random-effects distribution or from the conditional distribution of random effects given the data.

Why ESS Is Between One and the Number of Draws

The bounds follow directly from the normalized weights. Because they are nonnegative and sum to one, their squared sum cannot exceed one: expanding the square gives $1=(\sum_m\tilde{r}_{im})^2=\sum_m\tilde{r}_{im}^{\,2}+2\sum_{j<k}\tilde{r}_{ij}\tilde{r}_{ik}$, and the cross-products are nonnegative. Also, Cauchy–Schwarz gives $1=(\sum_m\tilde{r}_{im})^2\leq M\sum_m\tilde{r}_{im}^{\,2}$. Thus the squared sum is between $1/M$ and 1, so its reciprocal is between 1 and $M$.

The upper bound is attained when all weights are equal: each normalized weight is $1/M$, and ESS is $M$. The lower bound is attained when one normalized weight is 1 and all others are zero, giving ESS 1. These bounds describe the concentration of the observed weights. They do not guarantee that a draw set with high ESS has covered every important region of the integrand.

Worked Example: Computing ESS From Four Raw Weights

Suppose a hypothetical set of four draws for one individual has raw weights $2,4,6,8$. Their sum is $20>0$, so normalization is defined. The normalized weights are $0.10,0.20,0.30,0.40$.

The squared normalized weights sum to $0.10^2+0.20^2+0.30^2+0.40^2=0.01+0.04+0.09+0.16=0.30$. Therefore, $$ \operatorname{ESS}=\frac{1}{0.30}=3.333. $$

The raw-weight calculation gives the same answer: $(2+4+6+8)^2/(2^2+4^2+6^2+8^2)=400/(4+16+36+64)=400/120=3.333$. There were four draws, but their weight concentration corresponds to an ESS of about 3.33 equally weighted draws.

Worked Example: One Draw Dominates

Consider a second hypothetical set of four raw weights, $9,1,0,0$. Their sum is $10>0$, so the normalized weights are $0.9,0.1,0,0$. The ESS is $$ \operatorname{ESS}=\frac{1}{0.9^2+0.1^2}=\frac{1}{0.82}=1.220. $$

Although four points were sampled, the first accounts for 90% of the total normalized weight. The small ESS flags substantial concentration in this draw set. It suggests that the estimate may be sensitive to a small number of proposal draws, but it does not by itself establish the size or direction of any error in the likelihood estimate.

Using Weight Concentration as a Proposal Diagnostic

For an NLME likelihood contribution, the proposal should place draws in regions that matter to $g_i(b)$, and the ratios $g_i(b)/q_i(b)$ should not be dominated by only a few sampled points. A low ESS is a warning that the proposal and the integrand may be poorly matched in the sampled region. A proposal based on a conditional mode and local curvature can improve this match, but weight concentration remains worth checking for each individual contribution.

For example, suppose four hypothetical draws have the following integrand and proposal-density evaluations. The last column gives each raw weight. The values are illustrative, not from a real study.

Draw	$g_i(b_{im})$	$q_i(b_{im})$	$r_{im}=g_i/q_i$
1	0.080	0.010	8
2	0.040	0.020	2
3	0.010	0.010	1
4	0.020	0.020	1

Worked Example: Interpreting a Concentrated Weight Set

Using the four raw weights in the table, their sum is $8+2+1+1=12>0$. The normalized weights are $8/12=0.6667$, $2/12=0.1667$, $1/12=0.08333$, and $1/12=0.08333$. The first draw alone carries two-thirds of the total normalized weight.

The ESS calculation is $$ \operatorname{ESS}=\frac{12^2}{8^2+2^2+1^2+1^2} =\frac{144}{70}=2.057. $$

This value is close to two rather than four, reflecting the unequal contributions. It warrants checking proposal adequacy and numerical stability, especially if similar concentration appears repeatedly across draw sets. It is not a universal pass/fail result: no single ESS threshold proves that a proposal is adequate or inadequate for every model and purpose.

What to Check, and What ESS Cannot Tell You

ESS is most useful as one diagnostic alongside examination of the weights and repeated calculations. For individual likelihood contributions, report or inspect the ESS relative to $M$, the largest normalized weight, and whether a few draws repeatedly dominate. A low ESS for one individual can be obscured by reporting only an average across individuals, so the distribution of individual ESS values is informative.

When concentration is substantial, a practical response is to check that the proposal has the intended support and is centered and dispersed sensibly for the individual’s integrand. Repeating the computation with more draws or a revised proposal can help determine whether the observed concentration is persistent. These checks do not replace careful likelihood evaluation; they help identify when an importance-sampling estimate may be fragile.

Observation	Reasonable interpretation	What it does not establish
ESS is near $M$	The sampled weights are relatively even.	That the proposal covers all important regions or that the integral estimate is accurate.
ESS is much smaller than $M$	A few sampled points carry much of the total weight; investigate proposal match and sensitivity.	The exact error in the likelihood estimate or its direction.
All sampled raw weights are zero	The normalized weights and ESS are undefined for this draw set.	That the integral itself is zero; draws may have missed the region where the integrand is positive.

The main alternatives address integration in different ways. Increasing the number of importance draws can reveal whether low ESS persists, but it may not fix an unsuitable proposal efficiently. Revising a Gaussian proposal around the conditional mode changes the distribution of draws and may reduce concentration when the local approximation is useful. Numerical quadrature avoids random importance weights but has its own computational and dimensional limitations. ESS is therefore a diagnostic within importance sampling, not a universal comparison score across integration methods.

Common Mistake

A common mistake is to say that normalized weights always sum to one whenever the proposal has support wherever the integrand is positive. That support condition concerns whether the proposal can represent the target integral; it does not ensure that a finite set of draws includes any point with a positive integrand. If the sampled raw weights are all zero, dividing each by their sum is impossible. The correction is to state that normalized weights and ESS are defined only when the sampled raw-weight sum is positive, and to investigate an all-zero draw set rather than reporting an ESS.

A second mistake is to interpret ESS as a count of independent observations. It is instead an algebraic summary of the concentration of a particular set of normalized importance weights. Even a high ESS cannot reveal an important region that the proposal never sampled.

Key takeaway: Normalize raw importance weights only when their sampled sum is positive. ESS, $1/\sum_m\tilde{r}_{im}^{\,2}$, ranges from 1 to $M$ and summarizes weight concentration; use it to flag possible proposal problems, not to certify accuracy.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

For raw weights $1,1,1,1$, what are the normalized weights and ESS?
For raw weights $5,0,0$, calculate ESS and interpret it in one sentence.
Why can a support condition on the proposal fail to make normalized weights defined for every finite draw set?
If every raw weight is multiplied by 10, what happens to the normalized weights and ESS?
What does an ESS close to $M$ indicate, and what does it not prove?

Solutions. (1) The normalized weights are $1/4$ each, so ESS $=1/(4(1/4)^2)=4$. (2) The normalized weights are $1,0,0$, so ESS $=1/(1^2)=1$; one draw carries all the weight. (3) The integrand may be zero on part of the proposal’s support, and all finite draws can land there, making every sampled raw weight zero and their sum zero. (4) Neither changes: the common factor cancels in normalization and in $(\sum r_m)^2/\sum r_m^2$. (5) It indicates relatively even sampled weights; it does not prove that the proposal covers all important regions or that the integral estimate is accurate.

Observation	Reasonable interpretation	What it does not establish
ESS is near \(M\)	The sampled weights are relatively even.	That the proposal covers all important regions or that the integral estimate is accurate.
ESS is much smaller than \(M\)	A few sampled points carry much of the total weight; investigate proposal match and sensitivity.	The exact error in the likelihood estimate or its direction.
All sampled raw weights are zero	The normalized weights and ESS are undefined for this draw set.	That the integral itself is zero; draws may have missed the region where the integrand is positive.