Why Evaluate Confidence Procedures by Simulation?
A confidence procedure is intended to produce intervals that capture an unknown parameter with a stated long-run frequency. For example, a procedure described as a 90% confidence procedure aims to cover the true parameter in about 90% of repeated samples under its assumed model. Its behavior in one dataset cannot establish whether it meets that aim. Simulation provides a direct evaluation: generate many datasets when the true parameter is known, apply the procedure to each, and record whether each resulting interval contains that value.
Coverage alone does not describe the whole result. A procedure could attain high coverage by producing intervals so wide that they are not useful. Average interval width gives a simple summary of the intervals’ typical span, in the same units as the parameter. Reading coverage and width together helps distinguish a procedure that misses too often from one that covers adequately but may be imprecise.
The true value \(\theta\) is known in a simulation because the data-generating model is specified. In an applied analysis it is generally unknown, which is exactly why a confidence interval is needed. A simulation study therefore evaluates a procedure under its chosen model and parameter settings; it does not reveal the true coverage for every possible real-world situation.
How to Calculate and Interpret the Diagnostics
For each simulated dataset, first obtain the interval using the procedure being evaluated. Then compare both endpoints with the known value. An interval covers if the true value is between its endpoints, including either endpoint. Its width is the upper endpoint minus the lower endpoint. After repeating those calculations for all \(R\) datasets, divide the number of covering intervals by \(R\), and divide the sum of all widths by \(R\).
The nominal confidence level is a target for the procedure’s repeated-sampling coverage under the assumed model. It is not a promise that exactly that proportion will cover in every finite simulation. If the nominal level is 90%, for instance, an empirical rate of 80% in a small run could reflect finite simulation variability, a procedure with actual undercoverage, or both. Conversely, a rate of 100% in a small run does not establish that the procedure is generally conservative.
Compare widths only when their units and target are meaningful together. If the parameter is a dollar amount, width is measured in dollars; if it is a probability, width is measured on the probability scale. A narrower interval is not automatically better: if it achieves narrowness by missing the true value too often, its coverage diagnostic reveals a problem. Likewise, among procedures with similar coverage, smaller average width may indicate greater precision under the simulated conditions.
Worked Example: Coverage and Width for Five Simulated Intervals
A hypothetical procedure is evaluated for estimating the mean \(\theta\) of a normal loss measure. The data-generating model is \(X_1,\ldots,X_{16}\overset{\mathrm{iid}}{\sim}N(\theta,16)\), where the second parameter is the variance, and the true mean is \(\theta=100\). The procedure produces these intervals from five simulated datasets: \([96,105]\), \([98,104]\), \([101,108]\), \([94,100]\), and \([97,102]\). Calculate empirical coverage and average width. The procedure is intended to have 90% coverage.
Method. Compare each interval with 100, then compute its width as upper endpoint minus lower endpoint.
The first interval covers 100 because \(96\leq100\leq105\), and its width is \(105-96=9\). The second covers 100, and its width is \(104-98=6\). The third does not cover 100 because its lower endpoint, 101, exceeds 100; its width is \(108-101=7\). The fourth covers 100 at its upper endpoint, and its width is \(100-94=6\). The fifth covers 100, and its width is \(102-97=5\).
The empirical coverage is 0.8000, or 80%, and the average width is 6.6 units of the mean. The empirical rate is below the nominal 90%, but five repetitions are not enough to diagnose a stable 10-percentage-point shortfall. The procedure’s performance should be assessed over a more substantial simulation, with the model and parameter value clearly stated.
Common mistake avoided: The interval \([94,100]\) covers because equality at an endpoint counts. Excluding it would incorrectly reduce the coverage numerator from four to three.
Comparing Two Procedures on the Same Simulations
When two procedures are evaluated using the same simulated datasets, coverage and width comparisons are easier to interpret because both procedures face the same generated cases. Their intervals may still differ in whether they cover and in how wide they are. Report both diagnostics rather than choosing a procedure from width alone.
Worked Example: Comparing Coverage and Width for Two Methods
For a hypothetical estimate of a normal mean, the generating model is \(X_1,\ldots,X_{12}\overset{\mathrm{iid}}{\sim}N(\theta,25)\), with true value \(\theta=40\). Two candidate interval procedures are applied to each of the same eight simulated datasets. Their output intervals are listed by replication:
| Replication | Procedure A | Procedure B |
|---|---|---|
| 1 | [36, 44] | [34, 46] |
| 2 | [38, 43] | [36, 44] |
| 3 | [39, 42] | [37, 43] |
| 4 | [35, 41] | [33, 42] |
| 5 | [41, 47] | [38, 44] |
| 6 | [37, 40] | [35, 41] |
| 7 | [38, 45] | [36, 45] |
| 8 | [42, 48] | [41, 45] |
Procedure A. The intervals cover 40 in replications 1, 2, 3, 4, 6, and 7. Replications 5 and 8 miss because their lower endpoints exceed 40. The widths in order are \(8,5,3,6,6,3,7,6\), whose sum is 44. Thus \(\widehat{C}_A=6/8=0.7500\) and \(\overline{W}_A=44/8=5.5\).
Procedure B. Its first seven intervals cover 40. The eighth misses because \(41>40\). Its widths are \(12,8,6,9,6,6,9,4\), whose sum is 60. Thus \(\widehat{C}_B=7/8=0.8750\) and \(\overline{W}_B=60/8=7.5\).
Procedure B has higher empirical coverage in these eight runs, while its average interval is 2.0 units wider. Neither result alone identifies a universal winner: the simulation is small, and both results describe only the specified model and parameter setting. The useful comparison is the observed coverage–width trade-off, not a claim that one method must always dominate.
Common mistake avoided: Do not divide each procedure’s number of covering intervals by a different denominator or compare widths from different numbers of replications. Both procedures here use the same eight runs.
Why the Averages Estimate Long-Run Performance
The logic of the coverage calculation follows directly from the indicator for a single simulated interval. Under the specified data-generating model, let \(C=P(L\leq\theta\leq U)\) be the procedure’s true coverage probability. The indicator \(I\) is one on the event that the interval covers and zero otherwise. Its expectation is therefore the probability of that event:
For \(R\) independently generated datasets using the same procedure, each indicator has expectation \(C\). By linearity of expectation, the average of those indicators also has expectation \(C\):
The same averaging principle applies to width. If \(W=U-L\) has a finite expectation under the model, then the average of \(R\) simulated widths has expectation \(E[W]\). These facts justify the diagnostics as simulation estimates of the corresponding long-run quantities. They do not mean a finite simulation must equal those quantities exactly.
Worked Example: Interpreting a Large Coverage Shortfall
A hypothetical procedure estimates the mean \(\theta\) of an exponential loss distribution, parameterized by its mean. The model is \(X_1,\ldots,X_{20}\overset{\mathrm{iid}}{\sim}\operatorname{Exponential}(\text{mean }200)\), with true mean \(\theta=200\). In six simulated datasets, the procedure returns \([190,208]\), \([195,205]\), \([201,214]\), \([188,200]\), \([196,203]\), and \([187,198]\). Its nominal coverage is 95%. Calculate both diagnostics and interpret them.
The first interval covers 200 and has width \(208-190=18\). The second covers and has width \(205-195=10\). The third misses because its lower endpoint is 201; its width is \(214-201=13\). The fourth covers at its upper endpoint and has width \(200-188=12\). The fifth covers and has width \(203-196=7\). The sixth misses because its upper endpoint is below 200; its width is \(198-187=11\).
The estimated coverage is 0.6667, or 66.67%, well below the nominal 95% in this run. The average width is approximately 11.83 units of the mean. This is a warning signal worth investigating, not by itself proof of persistent undercoverage: only six simulated datasets were used. Increasing the number of runs would make the empirical diagnostic less sensitive to the particular small set of outcomes.
Common mistake avoided: The width of an interval is not its midpoint or its distance from the true value. For \([187,198]\), the width is \(198-187=11\), even though the interval misses the true mean.
Common Traps and Practical Reporting
- Using the wrong coverage denominator. Divide by the total number of simulated datasets, including those whose intervals miss. Do not divide by the number of intervals that look plausible or by the number of successful procedure runs unless the study explicitly defines a different estimand.
- Reversing the endpoints. Width is \(U_r-L_r\), not \(L_r-U_r\). A valid interval with ordered endpoints has nonnegative width.
- Dropping endpoint equality. For the conventional closed interval \([L_r,U_r]\), the true value is covered when \(L_r\leq\theta\leq U_r\). Equality counts.
- Treating a nominal level as an exact finite-run target. Nominal 90% does not require exactly 90 of every 100 simulated intervals to cover. It describes the intended long-run probability under the model.
- Reporting coverage without width. High coverage can be achieved with broad intervals. Width adds information about the precision of the procedure’s outputs.
- Generalizing beyond the simulation setup. Coverage is evaluated relative to the generating model, parameter value, sample size, and procedure used. A result at one setting need not hold at another.
A clear report states the data-generating distribution and parameter values, sample size, number of simulated datasets, nominal confidence level, empirical coverage, and average interval width with units. If two methods are compared, keep the simulation settings and runs aligned. Interpret a shortfall or a width advantage as evidence about performance under those stated conditions, not as a guarantee about every future dataset.
Check Your Understanding
For each problem, calculate the requested diagnostics before reading the answer.
- A hypothetical mean parameter is \(\theta=12\). Four simulated intervals are \([10,13]\), \([12,15]\), \([13,16]\), and \([8,11]\). Find empirical coverage and average width. (Answer: \(2/4=0.5000\) coverage; average width \(3\).)
- For a hypothetical severity mean \(\theta=25\), five intervals are \([21,27]\), \([24,26]\), \([26,30]\), \([20,25]\), and \([22,24]\). Find coverage and average width. (Answer: \(3/5=0.6000\) coverage; average width \(3.8\).)
- A procedure targets \(\theta=70\) and returns \([67,73]\), \([69,72]\), \([71,76]\), \([65,70]\), \([68,74]\), and \([72,78]\) in six simulated runs. Calculate the coverage rate and average width to two decimals. (Answer: \(4/6=0.6667\) coverage; average width \(31/6\approx5.17\).)
- For \(\theta=9\), four intervals are \([8,10]\), \([9,12]\), \([10,13]\), and \([6,9]\). Count the endpoint cases correctly and calculate both diagnostics. (Answer: \(3/4=0.7500\) coverage; average width \(11/4=2.75\).)
- Two procedures are evaluated on the same 20 simulated datasets under a fixed model. Procedure A covers in 18 runs and has average width 4.2; Procedure B covers in 19 runs and has average width 7.1. Both target 90% coverage. State the empirical coverage rates and a cautious comparison. (Answer: A has \(18/20=0.9000\) coverage and B has \(19/20=0.9500\); B covers more often in this run but is wider, and the finite simulation does not establish that it is generally superior.)