Why Sequential Monitoring Changes the Problem
Simulation estimates are often examined before the run is complete. A team might check an estimated event probability after each block of trials and stop once the result looks stable. The difficulty is that a confidence interval designed for one fixed sample size is generally guaranteed to cover only at that fixed sample size. Checking repeatedly and stopping when an interval is appealing changes the procedure: the final sample size depends on the observed results.
For example, an estimate may temporarily be far from the true probability and later move back toward it. If monitoring stops at the first look where the estimate appears sufficiently precise or where a target is crossed, the selected estimate is not simply an estimate at a sample size fixed in advance. The nominal coverage of an ordinary fixed-sample interval therefore does not, by itself, guarantee the same coverage at the selected stopping time.
The distinction is about the guarantee, not a claim that every sequence of ordinary intervals must have poor coverage. Repeated monitoring creates additional opportunities for an interval to miss. A sequential procedure accounts for those opportunities in advance.
A Simultaneous Procedure for Simulated Probabilities
Suppose each simulation produces an independent Bernoulli result \(I_k\), where \(I_k=1\) if a specified event occurs and \(I_k=0\) otherwise. Assume the results have the same event probability \(p\), so \(I_k\sim\operatorname{Bernoulli}(p)\). After \(n\) trials, the estimator is \(\hat p_n=n^{-1}\sum_{k=1}^n I_k\). The method below uses Hoeffding’s inequality and is valid without estimating a standard error.
Before running the simulation, specify the look sample sizes \(n_1<n_2<\cdots\) and allocate an error probability \(\alpha_j\) to look \(j\), with \(\sum_{j=1}^{\infty}\alpha_j\leq\alpha\). One convenient allocation is \(\alpha_j=\alpha/[j(j+1)]\), since the sum of \(1/[j(j+1)]\) over all positive integers is 1.
An interval may be clipped to the known range of a probability, giving \([\max(0,\hat p_{n_j}-r_j),\min(1,\hat p_{n_j}+r_j)]\). Clipping cannot remove the true value when it is in \([0,1]\) and the untruncated interval contains it. Thus the clipped intervals retain the simultaneous coverage guarantee.
Why the Guarantee Holds
For each fixed look \(j\), Hoeffding’s inequality for independent variables in \([0,1]\) gives \(P(|\hat p_{n_j}-p|\geq r)\leq2\exp(-2n_jr^2)\). Substituting \(r=r_j\) makes the right-hand side equal to \(\alpha_j\):
Let \(M_j\) be the event that the interval at look \(j\) misses \(p\). The probability of at least one miss is bounded by the union bound, which does not require the miss events to be independent:
Therefore, with probability at least \(1-\alpha\), every interval at every prespecified look contains \(p\). If a stopping rule chooses a look using the observed simulation results, the interval at that selected look still contains \(p\) on this simultaneous-coverage event. This is the key justification for sequential stopping.
Stopping for a Target Width
For the untruncated interval, the total width is exactly \(2r_j\). If the target width is \(w\), a sufficient rule is to stop at the first scheduled look \(j\) for which \(2r_j\leq w\). Since the radius is determined by the prespecified look and error allocation, this width condition can be checked before observing the estimate.
A clipped interval has width at most \(2r_j\), and may have a smaller width. Consequently, the same condition \(2r_j\leq w\) guarantees that the clipped interval has width at most \(w\), but it need not be the first look at which the actual clipped width is at most \(w\). Alternatively, one may stop when the actual clipped width is at most \(w\); simultaneous coverage still protects the selected interval. These are distinct width rules.
Worked Example: Applying Alpha Spending at a Prespecified Look
A hypothetical simulation estimates the probability that an aggregate loss exceeds a specified threshold. The results are independent Bernoulli trials with a common, unknown probability \(p\). The analyst sets an overall error level \(\alpha=0.05\), uses \(\alpha_j=0.05/[j(j+1)]\), and examines the estimate at \(n_2=600\). At that look, 72 of the 600 trials exceed the threshold. Construct the simultaneous-procedure interval.
Method. Use the error allocation for look \(j=2\), then apply the Hoeffding radius. This avoids treating the second look as though it were the only look planned.
First calculate the allocated error probability: \(j(j+1)=2(3)=6\), so \(\alpha_2=0.05/6=0.0083333\). The estimate is \(\hat p_{600}=72/600=0.1200\).
The radius is \(r_2=\sqrt{\log(2/0.0083333)/(2(600))}=\sqrt{\log(240)/1200}\). Since \(\log(240)=5.48064\), \(r_2=\sqrt{0.00456720}=0.06758\).
The untruncated interval is \([0.1200-0.06758,\;0.1200+0.06758]=[0.05242,\;0.18758]\). It lies inside \([0,1]\), so clipping does not change it. To four decimal places, the interval is \([0.0524,\;0.1876]\).
Check. Its untruncated width is \(2(0.06758)=0.13516\), which agrees with \(0.18758-0.05242=0.13516\). This interval is one member of a sequence with simultaneous coverage of at least 95%, not merely a nominal interval chosen after the second look.
Worked Example: Stopping at the First Look That Meets a Width Target
A hypothetical model-validation simulation estimates an event probability using independent Bernoulli trials. Use \(\alpha=0.05\), \(\alpha_j=0.05/[j(j+1)]\), and prespecified look sizes \(n_1=500\), \(n_2=1000\), and \(n_3=2000\). The observed event counts are respectively 55, 112, and 225. The target is an untruncated interval width no greater than \(w=0.08\). Find the stopping look and report its interval.
Method. Calculate the Hoeffding radius at each scheduled look and stop at the first for which \(2r_j\leq0.08\). The width criterion is based on the untruncated intervals.
At look 1, \(\alpha_1=0.05/(1(2))=0.025\), and \(\hat p_{500}=55/500=0.1100\). Thus \(r_1=\sqrt{\log(80)/1000}=\sqrt{4.38203/1000}=0.06620\), giving width \(2r_1=0.1324\). This does not meet the target.
At look 2, \(\alpha_2=0.05/6=0.0083333\), and \(\hat p_{1000}=112/1000=0.1120\). Thus \(r_2=\sqrt{\log(240)/2000}=\sqrt{5.48064/2000}=0.05235\), giving width \(2r_2=0.1047\). This also does not meet the target.
At look 3, \(\alpha_3=0.05/[3(4)]=0.0041667\), and \(\hat p_{2000}=225/2000=0.1125\). Thus \(r_3=\sqrt{\log(480)/4000}=\sqrt{6.17379/4000}=0.03929\), giving width \(2r_3=0.07858\). This is the first look that meets the target, so the procedure stops at \(n_3=2000\).
The interval is \([0.1125-0.03929,\;0.1125+0.03929]=[0.07321,\;0.15179]\), or \([0.0732,\;0.1518]\) to four decimal places. Its width is \(0.15179-0.07321=0.07858\), as required. The simultaneous coverage guarantee applies at this selected look.
Worked Example: Clipping Can Reduce the Actual Width
At the first look of a hypothetical Bernoulli simulation, \(n_1=100\) trials produce no events. Use \(\alpha=0.05\) and \(\alpha_j=0.05/[j(j+1)]\). Determine the untruncated and clipped interval widths, and decide whether the clipped interval meets a target actual width of \(0.20\).
Method. Calculate the radius at \(j=1\), form both intervals, and compare their widths separately. This avoids assuming that clipping leaves an interval’s width unchanged.
Here \(\alpha_1=0.05/2=0.025\), \(\hat p_{100}=0/100=0\), and \(r_1=\sqrt{\log(2/0.025)/(2(100))}=\sqrt{\log(80)/200}=\sqrt{4.38203/200}=0.1480\).
The untruncated interval is \([-0.1480,\;0.1480]\), with width \(0.1480-(-0.1480)=0.2960\). It does not meet a width target of \(0.20\). Clipping to \([0,1]\) gives \([0,\;0.1480]\), with actual width \(0.1480-0=0.1480\). Therefore the clipped interval does meet the target actual width.
The difference is important: the rule \(2r_1\leq0.20\) is false, since \(0.2960>0.20\). It would be incorrect to claim that the untruncated width rule triggered. Rather, the clipped interval itself has width \(0.1480\). Its coverage remains protected by the simultaneous guarantee.
Common Mistakes and Practical Conditions
- Reusing a pointwise interval at a random sample size. A fixed-sample confidence level does not automatically apply after repeatedly checking and choosing when to stop. Use a procedure whose guarantee covers the monitored looks.
- Spending the full error probability at every look. Assigning \(\alpha\) to each look makes the union-bound total exceed \(\alpha\). The allocations must satisfy \(\sum_j\alpha_j\leq\alpha\).
- Mixing clipped and untruncated widths. The untruncated width is \(2r_j\); the clipped width can be smaller. State which width is being tested and calculate it accordingly.
- Changing the look schedule without accounting for it. The displayed guarantee covers the prespecified sequence. If additional looks are introduced, their error probabilities must also be allocated so the total remains at most \(\alpha\).
- Ignoring the data assumptions. The Hoeffding calculation here assumes independent, identically distributed Bernoulli outcomes with a fixed probability \(p\). Correlated simulation output or changing event probabilities require a method whose assumptions cover that setting.
Alpha spending can be conservative, especially when many looks are planned, because it protects against a miss at any look in the sequence. Its value is transparent: the interval can be reported after a data-dependent stopping decision without pretending that the selected sample size was fixed in advance. This method is aligned with the SOA Exam ASTAM (Advanced Short-Term Actuarial Mathematics) syllabus.
Check Your Understanding
For each question, use the stated alpha-spending procedure where applicable. Give probabilities and interval endpoints to four decimal places.
- At look \(j=3\), use \(\alpha=0.05\), \(n_3=900\), and \(\hat p_{900}=0.0800\). Find \(\alpha_3\), the Hoeffding radius, and the untruncated interval. (Answer: \(\alpha_3=0.0042\), \(r_3=0.0586\), interval \([0.0214,\;0.1386]\).)
- At the first look, \(n_1=400\), \(\hat p_{400}=0.1500\), and \(\alpha=0.10\). Using \(\alpha_j=\alpha/[j(j+1)]\), calculate the untruncated interval width. (Answer: \(0.1358\).)
- At look \(j=4\), \(\alpha=0.05\) and \(n_4=1600\). Find the radius and state whether an untruncated target width of \(0.10\) is met. (Answer: \(r_4=0.0457\); yes, the width is \(0.0914\).)
- A first-look simulation has \(n_1=100\), zero observed events, and \(\alpha=0.05\). Find the clipped interval and its actual width. Does this meet an actual-width target of \(0.15\)? (Answer: clipped interval \([0,\;0.1480]\), width \(0.1480\); yes.)
- Explain why intervals with error allocations \(\alpha_j=\alpha/[j(j+1)]\) have simultaneous coverage of at least \(1-\alpha\), even though the estimates at successive looks use overlapping simulation results. (Answer: each look’s miss probability is at most \(\alpha_j\); the union bound gives probability of any miss at most \(\sum_j\alpha_j=\alpha\), without requiring the miss events to be independent.)