Confidence Intervals for Simulated Probabilities

Why Put an Interval Around a Simulated Probability?

Suppose a simulation estimates the probability of an event by generating many independent trials and counting how often the event occurs. The observed proportion is an estimate, not the event's known probability. A confidence interval summarizes the sampling uncertainty in that estimate: it gives a range of probability values compatible with the simulation result under a stated procedure.

This tutorial considers three intervals for a probability estimated from independent Bernoulli trials: the Wald interval, the Wilson score interval, and the exact binomial interval. They use the same trial data but can give noticeably different endpoints, especially with few events or an estimated probability near 0 or 1. The methods are aligned with the SOA Exam ASTAM (Advanced Short-Term Actuarial Mathematics) syllabus.

The equal-probability assumption means that each trial estimates the same target probability. For example, independent simulations of identically specified hypothetical claim-generating scenarios may estimate the probability that a simulated annual loss exceeds a specified threshold. If trials are dependent, or if their event probabilities differ, the binomial interval formulas below need not be valid.

A \(100(1-\alpha)\%\) confidence procedure is designed so that, over repeated samples under its assumptions, its intervals contain the fixed probability \(p\) at the stated rate (or at least that rate for a conservative procedure). After observing one interval, it is not correct to say that the fixed \(p\) has probability \(1-\alpha\) of being inside it. The interval is random before the simulation; \(p\) is fixed.

The Wald Interval

The Wald interval uses the estimated standard error of the simulated proportion and a normal approximation. For a two-sided confidence level \(1-\alpha\), let \(z_{1-\alpha/2}\) be the \(1-\alpha/2\) quantile of the standard normal distribution. The interval is

The estimated standard error follows from the binomial variance. Since \(K\) is a sum of independent Bernoulli variables, \(\operatorname{Var}(K)=np(1-p)\), so \(\operatorname{Var}(\hat p)=p(1-p)/n\). Replacing the unknown \(p\) by \(\hat p\) gives the estimated standard error in the formula. The normal approximation is most defensible when the event and non-event counts are both sufficiently large; it can behave poorly near the boundaries.

The Wilson Score Interval

The Wilson interval avoids simply centering a normal interval at \(\hat p\). Instead, it inverts the score test for each candidate probability \(p\). For a two-sided confidence level \(1-\alpha\), write \(z=z_{1-\alpha/2}\). The Wilson endpoints are

The “plus or minus” in this expression gives the lower and upper endpoints, respectively. The resulting interval remains within \([0,1]\) and usually behaves better than the Wald interval when the event count is small or the estimate is close to a boundary.

Multiplying through by \(p(1-p)/n\) and collecting terms gives a quadratic inequality in \(p\):

The roots of this quadratic are the boundary values that satisfy equality. Applying the quadratic formula and dividing the numerator and denominator by \(2n\) gives the Wilson endpoints above. Thus the interval is obtained by inverting a test, rather than by attaching a symmetric margin directly to \(\hat p\).

The Exact Binomial Interval

The exact binomial interval, also called the Clopper–Pearson interval, uses the binomial distribution directly rather than a normal approximation. For observed count \(K=k\), a two-sided interval assigns tail probability \(\alpha/2\) to each side. For \(0<k<n\), its lower endpoint \(p_L\) and upper endpoint \(p_U\) solve

These equations are solved numerically in general. The tails are discrete, so the procedure is conservative: its actual coverage is at least the nominal confidence level, but can exceed it. At a boundary, the missing tail is handled by setting the corresponding endpoint to 0 or 1. In particular, if \(k=0\), the lower endpoint is 0 and the upper endpoint solves

Choosing and Interpreting an Interval

The interval describes simulation error in estimating the event probability, conditional on the simulation model and its assumptions. It does not account for uncertainty about whether the model itself is appropriate, uncertainty in parameters supplied to the model, or uncertainty in the underlying real-world process. Those are distinct sources of uncertainty.

A frequent exam trap is to treat “no events observed” as evidence that \(p=0\), or to apply a Wald formula mechanically when \(\hat p\) is 0 or 1. If \(\hat p=0\), the Wald standard error is zero, which incorrectly produces the degenerate interval \([0,0]\). The exact boundary calculation above instead gives a positive upper endpoint. The Wilson interval also has a positive upper endpoint at this boundary.

Common Calculation Trap

For the Wald example with \(25\) events in \(100\) trials, a tempting but incorrect result is \([0.1653,0.3347]\). That pair corresponds to a margin of \(0.0847\), which is not the margin obtained from the stated standard error. The correct calculation is

so the endpoints are \(0.25-0.0848705=0.1651295\) and \(0.25+0.0848705=0.3348705\), or \([0.1651,0.3349]\) to four decimal places. Keep enough digits in the standard error and margin, and round only the final endpoints.

• Use the actual number of independent trials \(n\), not the number of events, as the denominator in \(\hat p=K/n\).
• Use \(z_{1-\alpha/2}\) for a two-sided interval; for a 95% interval this is approximately \(1.96\).
• Check that endpoints are between 0 and 1. An out-of-range Wald endpoint is a warning about that approximation, not a valid probability.
• Interpret a confidence level as a repeated-sampling property of a procedure, not as a probability statement about a fixed \(p\) after observing the interval.