Rare-Event Assurance Via Importance Sampling

Why Change the Simulation Distribution?

Assurance is the probability, before a trial begins, that its eventual data will satisfy a prespecified success rule, averaging over uncertainty about the design parameters. When success is rare, ordinary simulation can be inefficient: most simulated trials fail, so the estimate is built from very few nonzero contributions. Importance sampling addresses this by drawing more often from parameter and outcome combinations that can lead to success, then correcting for that change with likelihood weights.

The correction is essential. A proposal that makes success common does not change the assurance being estimated; it changes only how the simulations are generated. Without the weights, the result estimates success under the proposal rather than under the trial design.

Definition: Let $\theta$ denote design parameters with design prior density or mass function $\pi(\theta)$, let $Y$ denote the trial data with sampling density or mass function $f(y\mid\theta)$, and let $\delta(y)$ equal 1 when the prespecified Bayesian decision rule declares the trial successful and 0 otherwise. Assurance is $$ A=\int \pi(\theta)\int f(y\mid\theta)\delta(y)\,dy\,d\theta =E_{\pi f}\{\delta(Y)\}. $$ The inner integral is a success probability conditional on $\theta$; the outer integral averages those probabilities over the design prior. The analysis prior used to apply the decision rule is part of $\delta$, and need not equal the design prior.

This expression is the target distribution for simulation. The earlier material on assurance and simulation estimation supplies the setting: the trial model and decision rule are specified, and the target is the probability that the rule succeeds. Here the issue is how to estimate that probability efficiently when successful simulated trials are scarce.

Deriving the Importance-Sampling Estimator

Suppose simulations instead draw $\theta$ from a proposal density or mass function $g(\theta)$, then draw $Y$ from a proposal sampling distribution $q(y\mid\theta)$. The proposal joint distribution is $g(\theta)q(y\mid\theta)$. Wherever the target assigns positive probability, the proposal must also assign positive probability. Under that support condition, define the likelihood weight $w(\theta,y)=\pi(\theta)f(y\mid\theta)/[g(\theta)q(y\mid\theta)]$.

\begin{aligned} A &=\int\int \pi(\theta)f(y\mid\theta)\delta(y)\,dy\,d\theta\\ &=\int\int g(\theta)q(y\mid\theta) \frac{\pi(\theta)f(y\mid\theta)}{g(\theta)q(y\mid\theta)} \delta(y)\,dy\,d\theta\\ &=E_{gq}\{w(\theta,Y)\delta(Y)\}. \end{aligned}

The derivation is a change of measure: multiply and divide the target integrand by the proposal joint density. It applies to discrete outcomes as well, with sums in place of integrals. For $M$ independent draws $(\theta_i,Y_i)$ from the proposal, the estimator is

\widehat A_{\mathrm{IS}}=\frac{1}{M}\sum_{i=1}^{M}w_i\delta(Y_i), \qquad w_i=\frac{\pi(\theta_i)}{g(\theta_i)} \frac{f(Y_i\mid\theta_i)}{q(Y_i\mid\theta_i)}.

The estimator is unbiased under the stated support condition and exact weights: $E_{gq}(\widehat A_{\mathrm{IS}})=A$. If only the outcome distribution is changed, draw $\theta\sim\pi$ and use $w=f/q$. If only the design-parameter distribution is changed, retain the trial sampling model and use $w=\pi/g$. Changing both requires both factors.

Assumptions: The target model and success rule are correctly specified; proposal probabilities or densities are evaluable; the proposal covers every target-relevant region; and the simulated proposal draws follow the stated proposal distribution. The usual sample-variance Monte Carlo standard error also requires a finite second moment $E_{gq}\{[w\delta(Y)]^2\}$. Support alone guarantees the expectation identity but does not guarantee a useful variance.

Worked Examples

Worked Example: Tilting a Rare Binomial Success Event

Consider a hypothetical trial-design calculation in which, at a fixed design parameter, the success rule is $Y\geq5$, where $Y$ is the number of responders among 10 participants. Under the target trial model, $Y\sim\mathrm{Binomial}(10,0.20)$. This conditional success probability is a component of assurance; assurance averages such probabilities over the design prior. To generate more successes, simulate instead from $q=\mathrm{Binomial}(10,0.50)$. Both distributions assign positive probability to every count from 0 through 10, so the support condition holds.

For a proposed count $y$, the likelihood weight is

w(y)=\frac{\binom{10}{y}(0.20)^y(0.80)^{10-y}} {\binom{10}{y}(0.50)^y(0.50)^{10-y}} =(0.40)^y(1.60)^{10-y}.

Suppose 10 proposal draws are $4,5,5,6,3,5,7,4,5,2$. The four draws equal to 5 each have weight $0.40^5(1.60)^5=0.107374$. The draw equal to 6 has weight $0.40^6(1.60)^4=0.0268435$, and the draw equal to 7 has weight $0.40^7(1.60)^3=0.00671089$. The other four draws fail, so their weighted indicators are zero. Thus

\widehat A_{\mathrm{IS}} =\frac{4(0.107374)+0.0268435+0.00671089}{10} =0.0463.

This is an estimate from a small hypothetical run, not the exact probability. The exact binomial tail is $P(Y\geq5)=0.03279$, so this particular run overestimates it. With $Z_i=w(Y_i)I(Y_i\geq5)$, the sample variance of the 10 contributions is approximately $0.002827$, giving estimated Monte Carlo standard error $\sqrt{0.002827/10}=0.0168$. That large error relative to the estimate is a warning not to treat 0.0463 as precise.

Worked Example: Changing Both the Design Prior and Outcome Proposal

Suppose a hypothetical design prior assigns probability $0.80$ to a low-effect scenario $L$ and $0.20$ to a high-effect scenario $H$. Conditional on these scenarios, the target trial model gives success probabilities $0.01$ and $0.05$, respectively. The target assurance is therefore $A=0.80(0.01)+0.20(0.05)=0.018$. To obtain more successful runs, use a proposal with $g(L)=g(H)=0.50$, $q(S\mid L)=0.20$, and $q(S\mid H)=0.40$, where $S$ denotes success. The complementary failure probabilities are $0.80$ and $0.60$.

For a successful draw under $L$, the weight is $(0.80/0.50)(0.01/0.20)=0.0800$. For a successful draw under $H$, it is $(0.20/0.50)(0.05/0.40)=0.0500$. Failure weights are not zero: they are $(0.80/0.50)(0.99/0.80)=1.98$ under $L$, and $(0.20/0.50)(0.95/0.60)=0.6333$ under $H$. Their contribution to the assurance estimate is zero because the success indicator is zero.

Suppose eight independent proposal draws happen to include two of each scenario-outcome combination: two $L$-successes, two $L$-failures, two $H$-successes, and two $H$-failures. The weighted estimate is

\widehat A_{\mathrm{IS}} =\frac{2(0.0800)+2(1.98)(0)+2(0.0500)+2(0.6333)(0)}{8} =0.0325.

The estimate differs from the exact value $0.018$, as a short random run can. The example also shows why the design-prior factor matters: oversampling the high-effect scenario without multiplying by $\pi/g$ would give too much influence to that scenario.

Diagnostics for Unstable Weights

Importance sampling can have high variance even when the estimator is unbiased. A proposal may produce many successes, yet assign them weights so variable that a few draws determine nearly the entire estimate. Check diagnostics on the contributions $Z_i=w_i\delta(Y_i)$, not just on the count of successful simulations.

For independent proposal draws, estimate Monte Carlo error using the sample variance $s_Z^2$ of the $Z_i$:

\widehat{\mathrm{SE}}(\widehat A_{\mathrm{IS}})=\frac{s_Z}{\sqrt{M}}, \qquad s_Z^2=\frac{1}{M-1}\sum_{i=1}^{M}(Z_i-\bar Z)^2.

This standard error is informative only if the run is large enough to reveal the variability. A very small number of nonzero contributions can make it deceptively unstable. Also inspect the largest weights, the range or upper tail of weights, the proportion of the estimate attributable to the largest few $Z_i$, and whether independent runs give broadly similar estimates. The weight effective sample size $ESS_w=(\sum_i w_i)^2/\sum_i w_i^2$ summarizes concentration among all weights, but it does not by itself measure precision for a rare-event probability. A success-contribution diagnostic is $ESS_S=(\sum_i Z_i)^2/\sum_i Z_i^2$, when at least one $Z_i$ is nonzero. A value near 1 means one successful contribution dominates.

Worked Example: Detecting a Dominant Successful Draw

In a hypothetical run of $M=100$, suppose the nonzero success contributions are $0.02, 0.03, 0.05,$ and $0.90$, with the other 96 contributions equal to zero. The estimate is $\widehat A_{\mathrm{IS}}=(0.02+0.03+0.05+0.90)/100=0.0100$. The largest contribution supplies $0.90/1.00=90\%$ of the total weighted successes.

Here $\sum Z_i^2=0.02^2+0.03^2+0.05^2+0.90^2=0.8138$, so $ESS_S=1.00^2/0.8138=1.23$. Although there are four successful draws, the information in their weighted contributions is concentrated in approximately one draw. The sample variance is $(0.8138-1.00^2/100)/99=0.00812$, and the estimated standard error of the assurance estimate is $\sqrt{0.00812/100}=0.00901$. This uncertainty is almost as large as the estimate itself; substantially more reliable computation is needed.

A practical response is to use a pilot run to identify where successful outcomes occur, construct a proposal that places mass in those regions, and then freeze that proposal for an independent production run. A mixture proposal can guard against a pilot missing an important region. For example, $q_{\epsilon}=(1-\epsilon)q_{\mathrm{tilt}}+\epsilon f$, with $0<\epsilon<1$, retains target support wherever $f$ has support. The mixture likelihood in the denominator must be the full mixture density, not just its tilted component. This safeguard does not guarantee low variance, but it can prevent a proposal from assigning too little probability to relevant target outcomes.

Choosing and Evaluating a Proposal

A useful proposal increases sampling in regions that contribute to assurance, without making likelihood ratios excessively variable. In clinical-trial applications, those regions may involve design parameters associated with larger treatment effects, outcome data that satisfy the posterior decision rule, or both. Tilting only the outcome distribution may help when success is conditionally rare across many plausible $\theta$ values. Tilting only the design prior may help when success is concentrated in a small part of its support. The joint weight formula permits either choice, but every modified factor must be corrected.

Specify the target.
Write the design prior, trial sampling model, and success indicator so the target assurance is unambiguous.

Choose a supported proposal.
Increase draws in plausible success regions while retaining positive proposal probability wherever the target can contribute.

Calculate full weights.
Use target joint density divided by proposal joint density, including every changed design or outcome distribution.

Estimate and diagnose.
Compute the weighted indicator mean, its Monte Carlo error, and weight and contribution concentration diagnostics.

Direct Monte Carlo draws from the target distribution are simplest and provide an uncomplicated estimate, but may yield almost no successful trials when assurance is very small. Importance sampling spends computation more strategically, at the cost of weight instability and extra implementation checks. Exact integration, when feasible for the model and decision rule, avoids simulation error and is a valuable benchmark; it is not always available for complex trial designs. Common random numbers, covered in the earlier material on variance reduction, can improve comparisons between designs by pairing simulations, but do not by themselves ensure enough rare successes in either design.

Importance sampling is not a remedy for a misspecified trial model, an incorrectly implemented success rule, or a proposal whose likelihood is evaluated incorrectly. Nor does an apparently large number of proposal successes guarantee precision. Always verify the proposal-to-target density ratio on test cases, check support explicitly, and compare independent runs or a direct-simulation benchmark when the event is not too rare. Report the proposal, the weighting formula, the number of draws, and diagnostics alongside the assurance estimate.

Common Mistakes and Limitations

Omitting the weights. Counting successes under a tilted proposal estimates proposal success probability, not target assurance. Multiply each success indicator by the target-to-proposal likelihood ratio.
Using only part of the joint weight. If both the design-parameter and outcome distributions change, correct both factors. If the proposal uses a mixture, evaluate the full mixture density.
Ignoring support. A target-relevant event region absent from the proposal cannot be recovered by weighting. Positive target density must imply positive proposal density.
Trusting a small estimated standard error blindly. A run with only a handful of influential contributions may not reveal the tail behavior that governs variance. Use concentration checks, larger independent runs, and sensitivity to proposal choice.
Confusing importance sampling with a changed design. Tilting is a computational device; it does not alter the trial’s design prior, sampling model, or decision rule when weights are correct.

The method works best when the proposal captures the important success region and its likelihood ratio remains manageable. A proposal that is too narrow can create extreme weights; a proposal too close to the target may retain the original scarcity of successes. There is no universal optimal proposal in routine use, so diagnostics and transparent reporting are part of the analysis rather than optional refinements.

Check Your Understanding

Answer each question, then compare with the solutions below.

If only $f(y\mid\theta)$ is replaced by $q(y\mid\theta)$, what is the importance weight?
Why must the proposal assign positive probability wherever the target joint distribution can produce a successful outcome?
Suppose four successful contributions are $0.10, 0.10, 0.10,$ and $0.70$. What fraction of their total is supplied by the largest contribution, and what is $ESS_S$?
For the two-scenario example, what is the target assurance if the high-scenario success probability changes from $0.05$ to $0.10$, with all other target probabilities unchanged?
Does drawing more successes under a proposal necessarily reduce the Monte Carlo error of the weighted estimate? Explain.

Solutions. (1) The weight is $f(y\mid\theta)/q(y\mid\theta)$, because $\theta$ is still drawn from its target design prior. (2) Otherwise some target contribution has zero chance of being sampled, so no finite likelihood weight can restore it; the change-of-measure identity fails. (3) The total is $1.00$, so the largest supplies $70\%$. Also $ESS_S=1.00^2/(0.10^2+0.10^2+0.10^2+0.70^2)=1/0.52=1.92$. (4) The assurance is $0.80(0.01)+0.20(0.10)=0.0280$. (5) No. The weights may become more variable, and a few weighted successes may dominate; judge precision from the weighted contributions, their Monte Carlo error, and concentration diagnostics.