Importance Sampling for Rare Failure Probabilities

Why Change the Simulation Distribution?

In design assurance, a failure may be an operating-characteristic event such as exceeding a maximum type I error rate, falling below a power target, or failing a safety rule. If that event is very rare under the design assumptions of interest, ordinary simulation may produce few or no failures. A zero count does not establish that the failure probability is zero; it may simply mean that the simulation did not run long enough to encounter the event.

Importance sampling addresses this problem by drawing simulated outcomes from a distribution that makes the failure event more common, then correcting the results with likelihood-ratio weights. The objective remains the failure probability under the target distribution. The changed distribution is a computational device, not a replacement for the design scenario being assessed.

Definition: Let $X$ denote a simulated outcome under the target distribution with density or probability mass function $f$, and let $F$ be the failure event. The target failure probability is $p_F=P_f(X\in F)$. Importance sampling draws independent outcomes $Y_1,\ldots,Y_N$ from a proposal distribution with density or mass function $g$, and weights each outcome by $w(y)=f(y)/g(y)$. Its estimator is $$ \hat{p}_{IS}=\frac{1}{N}\sum_{i=1}^{N} I(Y_i\in F)w(Y_i), $$ where $I(Y_i\in F)$ equals 1 for a failure and 0 otherwise.

The ratio $f(y)/g(y)$ is the likelihood ratio: it compares how much probability the target and proposal distributions assign to the same outcome. Densities and mass functions must be defined with respect to the same underlying measure. For simulations with several stages, $f$ and $g$ refer to the probability of the complete simulated path, including the randomized decisions that generated it.

Conditions and a Justification

Two conditions are central. First, the proposal must cover the target failure outcomes: whenever an outcome in $F$ has positive probability under the target distribution, it must also have positive probability under the proposal. More generally, the target distribution must be absolutely continuous with respect to the proposal on the part of the outcome space relevant to estimating $p_F$. Second, the likelihood ratio must be calculated for the full outcome under the two specified distributions. Changing the simulation but omitting a changed factor from the ratio does not correct the result.

The estimator is unbiased under these conditions. For a single proposal draw $Y$, its expected weighted contribution is

\begin{aligned} E_g[I(Y\in F)w(Y)] &=\int I(y\in F)\frac{f(y)}{g(y)}g(y)\,dy\\ &=\int_F f(y)\,dy\\ &=P_f(X\in F)=p_F. \end{aligned}

Taking the mean of $N$ independent contributions preserves this expectation. If the contribution $I(Y\in F)w(Y)$ has finite variance, the usual sample variance of the $N$ contributions, divided by $N$, estimates the variance of $\hat{p}_{IS}$; its square root estimates the Monte Carlo standard error. This quantifies simulation uncertainty, not uncertainty about whether the chosen target model is scientifically adequate.

Worked Example: Tilting a Normal Simulation

Suppose a standardized simulated quantity $X$ follows $N(0,1)$ under the target scenario, and failure is $X\geq 3$. To see failures more often, simulate instead from $g=N(3,1)$. Both normal distributions have positive densities everywhere, so every target outcome remains represented.

For a proposal draw $y$, the likelihood ratio is

w(y)=\frac{\exp(-y^2/2)}{\exp(-(y-3)^2/2)} =\exp(4.5-3y).

Imagine ten hypothetical proposal draws: $2.2, 2.5, 2.7, 2.9, 3.0, 3.1, 3.3, 3.5, 3.8,$ and $4.0$. The six draws at least 3 are failures. Their weighted contributions, rounded to four significant figures, are $0.01111, 0.008230, 0.004517, 0.002479, 0.001008,$ and $0.0005531$. The other four contributions are zero. Therefore,

\hat{p}_{IS} =\frac{0.01111+0.008230+0.004517+0.002479+0.001008+0.0005531}{10} \approx 0.002790.

The proposal made failures common enough to observe, and the weights bring the estimate back toward the target distribution. For comparison, the exact target probability is $P(X\geq 3)\approx 0.001350$. The estimate from these ten draws is not especially close: ten simulations provide little precision, and importance sampling does not guarantee that every small run gives an accurate answer. This example illustrates the calculation, not a sufficient simulation size for assurance.

Discrete Outcomes and Scenario Mixtures

For a discrete simulation, the likelihood ratio is a ratio of probabilities rather than densities. When a failure can arise through distinct scenario pathways, the proposal should also be checked pathway by pathway. A high overall failure rate under the proposal is not enough if it achieves that rate by oversampling only one pathway and leaving another target pathway poorly represented or absent.

Worked Example: Two Rare Failure Mechanisms

Suppose two independent events, $A$ and $B$, must both occur for failure. Under the target distribution, $P(A)=0.01$ and $P(B)=0.02$, so the target failure probability is $0.01\times0.02=0.0002$. A proposal simulation raises both probabilities to 0.20. All four combinations still have positive probability under both distributions, so target outcomes are represented.

A proposal draw with both events occurring has likelihood ratio

w(A=1,B=1)=\frac{0.01}{0.20}\frac{0.02}{0.20} =0.05\times0.10=0.005.

Suppose 100 hypothetical proposal draws contain four failures. Each contributes $0.005$; all other draws contribute zero. The estimate is

\hat{p}_{IS}=\frac{4(0.005)}{100}=0.000200.

The unweighted proposal failure fraction is $4/100=0.04$, which estimates the failure probability under the proposal, not under the target distribution. Using that fraction as the design-assurance result would therefore be incorrect.

Worked Example: Weighting Two Scenario Pathways

Consider a simulation with two scenario types, $A$ and $B$. Under the target distribution, each type has probability 0.5; the conditional failure probabilities are 0.001 for $A$ and 0.009 for $B$. Thus the target failure probability is

$$ p_F=0.5(0.001)+0.5(0.009)=0.005. $$

For efficiency, suppose the proposal selects $A$ with probability 0.9 and $B$ with probability 0.1, and uses conditional failure probabilities 0.01 for $A$ and 0.05 for $B$. The likelihood ratio for an $A$-failure is the target joint probability divided by the proposal joint probability:

w(A,\text{failure})=\frac{0.5(0.001)}{0.9(0.01)} =\frac{0.0005}{0.009}\approx 0.05556.

For a $B$-failure it is

w(B,\text{failure})=\frac{0.5(0.009)}{0.1(0.05)} =\frac{0.0045}{0.005}=0.9.

Suppose 1,000 proposal runs contain eight $A$-failures and five $B$-failures. The estimate is

\hat{p}_{IS} =\frac{8(0.05556)+5(0.9)}{1000} \approx 0.004944.

The unweighted failure fraction, $13/1000=0.013$, is not an estimate of the target probability. Notice also that the two failure types receive different weights. The calculation is valid here because both scenario types and their failure outcomes have positive proposal probability.

Choosing and Checking a Proposal

A useful proposal makes important failures easier to encounter without making the likelihood ratios so variable that a few simulated paths dominate the estimate. In adaptive-trial simulations, this may mean shifting input assumptions toward a performance threshold, increasing the frequency of operational deviations, or sampling more often from a difficult subgroup-prevalence scenario. The correct change depends on the failure definition and on which inputs can produce it.

Before trusting an estimate, check the following:

Support: Can every target failure pathway occur under the proposal? A pathway with target probability greater than zero but proposal probability zero cannot be recovered by weighting.
Likelihood-ratio completeness: Does the ratio include every part of the simulated path whose distribution changed, including scenario selection and sequential random choices?
Failure coverage: Do proposal runs include the distinct failure mechanisms that matter under the target, rather than only a convenient subset?
Stability: Are estimates from independent simulation batches broadly compatible, or are results changing sharply with a small number of weighted failures? Such instability calls for investigation, not a claim of assurance.
Validation: Where feasible, compare with crude Monte Carlo for a less-rare version of the event, or with an analytically available probability. This can reveal coding or weighting errors.

These checks do not prove that a proposal is optimal. Weight variability and the effective contribution of observations require their own diagnostics; a proposal that satisfies support conditions can still produce an imprecise estimate. Importance sampling also cannot repair a misspecified target model, an incorrectly defined failure event, or uncertainty in inputs that were not included in the simulation.

When Importance Sampling Helps—and When It Does Not

Crude Monte Carlo draws directly from the target distribution and estimates $p_F$ as the fraction of runs that fail. It is simple and needs no weights, but for a very rare event it may need an enormous number of runs to observe enough failures. Importance sampling can reduce that burden when a proposal increases failure frequency and the corresponding weights remain usable.

Other options include exact calculation when the model is simple enough, or structured approaches such as stratifying simulation by scenario. These alternatives can be preferable when probabilities are available analytically, when failure pathways are difficult to cover with a single proposal, or when a defensible proposal is hard to construct. Importance sampling is not automatically more efficient: a poor proposal may yield unstable weighted estimates even when it produces many failures.

Common mistake: “The proposal produced 50 failures in 1,000 runs, so the target failure probability is 5%.” That fraction estimates the proposal probability. The target estimate is the average of the failure indicators multiplied by the likelihood ratios $f(Y_i)/g(Y_i)$. A second serious error is assigning weight zero in practice to a target failure pathway that the proposal never generates; no finite weight can recover an unrepresented pathway.

Check Your Understanding

Try each question, then compare your reasoning with the solutions.

If the target failure probability is positive for an outcome but the proposal assigns that outcome probability zero, can likelihood-ratio weighting estimate the target probability? Why?
A proposal run has a failure indicator of 1 and likelihood ratio 0.04. What is its weighted contribution? What is the contribution of a nonfailure with likelihood ratio 0.04?
In the two-event example, what is the target probability of both events if $P(A)=0.02$, $P(B)=0.03$, and they are independent?
Why is the unweighted failure fraction not generally the importance-sampling estimate?

Solutions. (1) No. The target failure outcome is not represented under the proposal, so the required likelihood ratio cannot supply its missing contribution. (2) The failure contributes $1(0.04)=0.04$; the nonfailure contributes $0(0.04)=0$. (3) Independence gives $0.02(0.03)=0.000600$. (4) The unweighted fraction estimates the failure probability under the proposal distribution. The target estimate averages failure indicators multiplied by the target-to-proposal likelihood ratios.