Weight Diagnostics for Importance Sampling

Why Weight Diagnostics Matter

Importance sampling can estimate a rare-event probability using simulations drawn more often from regions where the event occurs. Its estimates can be useful even when direct simulation would rarely observe a failure. But a proposal distribution that generates relevant events does not automatically produce a stable estimate: a few observations may carry very large weights and dominate the result. Diagnostics help reveal when apparent precision rests on too little effective information.

Let \(P\) denote the target distribution for a random quantity \(X\), and let \(Q\) be the proposal distribution used to generate simulations. Suppose the target distribution has density \(p(x)\), the proposal has density \(q(x)\), and \(p(x)=0\) wherever \(q(x)=0\). For \(X_i\) independently sampled from \(Q\), define the importance weight \(w_i=p(X_i)/q(X_i)\). For a target quantity \(\theta=E_P[h(X)]\), the ordinary importance-sampling estimator is the average of the weighted outcomes \(w_i h(X_i)\).

For a failure probability, \(h(X)=I(X\text{ is a failure})\), where the indicator is 1 for a failure and 0 otherwise. The estimator then averages the weights of simulated failures, including zero contributions from nonfailures. With exact density-ratio weights, the ordinary estimator is unbiased when the target expectation exists. Its usefulness still depends on finite simulation uncertainty and adequate coverage of the target distribution.

What the Weight ESS Measures

The ESS summarizes how concentrated the weights are. If all weights are equal and positive, \(ESS_w=N\). If one weight is positive and all other weights are zero, \(ESS_w=1\). It is a diagnostic summary, not a literal count of independent target-distribution observations and not a guarantee that an estimate is accurate.

One way to see the connection with weight variation is to write \(\bar{w}=N^{-1}\sum_i w_i\) and \(s_w^2=(N-1)^{-1}\sum_i(w_i-\bar{w})^2\). Since \(\sum_i w_i^2=(N-1)s_w^2+N\bar{w}^2\), the ESS can be written as

When \(N\) is large, this is approximately \(N/(1+CV_w^2)\), where \(CV_w=s_w/\bar{w}\). Thus greater relative weight variation generally means lower ESS. The approximation helps explain the diagnostic; the defining ESS formula does not require this large-sample approximation.

Check the Outcomes as Well as the Weights

Weight ESS alone can miss an important feature of a rare-event estimate: only simulations with \(h(X_i)=1\) contribute to its failure-probability estimate. A high weight ESS can coexist with very few simulated failures, or with failures whose weighted contributions are dominated by one draw. Inspect both the weight distribution and the weighted outcomes \(w_i h(X_i)\).

For a failure estimate, useful summaries include the number of failures observed under \(Q\), the largest weight among those failures, the total weighted failure contribution, and the share of that total supplied by the largest one or few contributions. The latter shares are meaningful only when the total contribution is positive. Report the simulation count and these summaries together; no single cutoff makes an estimate reliable in every application.

Assumptions and Practical Diagnostics

Before interpreting diagnostics, check the conditions that make the estimator meaningful. The proposal must assign positive probability wherever the target distribution can contribute to the quantity of interest. In density notation, \(q(x)>0\) wherever \(p(x)|h(x)|>0\). Otherwise, relevant target regions cannot be sampled and the density ratio cannot recover their contribution. The proposal and target densities must also be correctly specified for the weights to be valid.

A finite sample can look manageable even when the theoretical weight distribution has a very heavy tail. In particular, if \(E_Q[(w(X)h(X))^2]\) is infinite, the usual finite-variance reasoning for a standard error and normal approximation does not apply. A finite ESS cannot prove this second moment is finite. Diagnostics should therefore look for warning signs, not certify assumptions that the observed sample cannot verify.

• Inspect the weight distribution. Summarize its range, upper quantiles, largest weights, and cumulative weight shares. A long upper tail or a few dominant weights warrants investigation.
• Inspect weighted event contributions. For a failure probability, check the number of failures and whether one or a few failures account for most of the weighted total.
• Check stability across batches or independent runs. Large shifts among estimates, or estimates that change sharply when a dominant draw is absent, indicate sensitivity. Batches are diagnostic comparisons, not independent evidence if they reuse the same simulations.
• Verify support and weight calculations. Investigate zero proposal density in relevant regions, numerical overflow or underflow, and unexpectedly extreme ratios. These may indicate a conceptual or computational problem.

Truncating or capping large weights may make a summary appear less variable, but it changes the estimator and can introduce bias. A capped estimate should be treated as a separate sensitivity analysis, with the cap and its consequences stated, not as proof that the original estimate is stable.

Comparison with Direct Simulation

If direct simulation from the target distribution is feasible, it provides a valuable check because it estimates the same probability without reweighting. For \(M\) independent direct draws and \(K\) observed failures, the direct estimate is \(\hat{p}_{direct}=K/M\), with estimated Monte Carlo standard error \(\sqrt{\hat{p}_{direct}(1-\hat{p}_{direct})/M}\). When estimates from independent direct and importance-sampling runs are compared, their difference should be considered relative to the uncertainty of both estimates.

Direct simulation may be uninformative for extremely rare events: observing no failures does not show the target probability is zero. Conversely, agreement between direct and importance-sampling estimates in a feasible check is reassuring but does not establish that all weights or scenarios are well behaved. The approaches have different failure modes: direct simulation can have too few events, while importance sampling can suffer from unstable weights or inadequate proposal coverage.

Common Mistakes and Limits

• “The ESS is 420, so the estimate is as precise as 420 direct draws.” The weight ESS summarizes concentration only. Precision also depends on the outcomes and their weighted variability; it is not a substitute for an uncertainty calculation.
• “A high weight ESS means the rare-event estimate is stable.” Many nonfailures can have balanced weights while very few failures determine the estimate. Examine event counts and weighted event contributions.
• “No extreme weights appeared, so the proposal covers the target.” Finite simulations cannot demonstrate coverage of regions that were never visited. Check the proposal’s support and construction, as well as the observed weights.
• “Capping the largest weights fixes instability.” Capping changes the estimator and may bias it. Use it only as a transparently reported sensitivity analysis, not as a silent correction.

Weight diagnostics are most useful as a set: weight concentration, event-specific contributions, repeat-run stability, support checks, and direct simulation where feasible. A low ESS is a warning to investigate; a high ESS is not a certificate. When the evidence remains sensitive to a few draws, report that limitation rather than presenting the point estimate as reassuringly precise.