Why Assess Coverage and Width?
A confidence interval is useful only if its repeated-sampling behavior is appropriate for the question. A narrow interval may look precise, but if the procedure misses the true parameter too often, its precision is misleading. Conversely, a method may attain its intended coverage while producing intervals too wide to support a useful study conclusion. Simulation makes it possible to assess these properties for a specified study design, data-generating scenario, and analysis procedure.
For each simulated dataset, apply the planned analysis and construct the confidence interval using the specified method. Compare that interval with the known parameter value used to generate the data. Repeating this process estimates how often the procedure covers the truth and how wide its intervals tend to be. These are operating characteristics of the complete design-and-analysis procedure under the chosen scenario, not universal properties of the interval formula.
Definitions and Assumptions
Coverage is defined relative to a particular parameter, scenario, study design, and analysis procedure. A nominal 95% confidence procedure is intended to cover its target parameter in about 95% of repeated datasets when its assumptions and approximation are adequate. The true parameter is fixed in this repeated-sampling interpretation; it is the interval that varies from replicate to replicate.
For the empirical coverage to estimate the intended operating characteristic, the simulated datasets must be generated under the specified scenario, and the analysis must be applied consistently as planned. Replicates should represent independent draws from that data-generating process for the simple Monte Carlo uncertainty calculations below. The true parameter value must be known from the simulation setup, and the interval endpoints must be defined on a scale that includes that parameter. If the procedure produces no interval for some simulated datasets, those outcomes cannot simply disappear from the summary: the handling rule and the number of such cases must be reported alongside coverage.
Average width is meaningful only on a stated parameter scale and in its units. For example, an average width of 0.4 L/h describes a different degree of precision from a width of 0.4 mg/L. When comparing intervals for a positive parameter, width on the original scale can also behave differently from width on the log scale. Choose and report the scale that answers the study question.
Monte Carlo Uncertainty in the Summaries
The empirical coverage is itself an estimate because it is based on a finite number of simulated replicates. Under the independent-replicate assumption, each \(I_r\) is a Bernoulli random variable with success probability \(C\), the true coverage of the procedure in this scenario. Thus \(E(I_r)=C\) and \(\operatorname{Var}(I_r)=C(1-C)\). The variance of the average of \(R\) independent indicators is \(C(1-C)/R\). Replacing the unknown \(C\) by its estimate gives the estimated Monte Carlo standard error (MCSE):
This standard error describes simulation noise in the estimated coverage, not uncertainty about the true parameter and not the sampling error of an interval for that parameter. A rough 95% Monte Carlo interval for coverage is \(\hat{C}\pm1.96\widehat{SE}(\hat{C})\), when the normal approximation is reasonable. It is a diagnostic summary; near zero or one, or with few expected covering or noncovering replicates, a binomial interval such as a Wilson interval is preferable.
The average width is also estimated with simulation. Let \(s_W\) be the sample standard deviation of the \(R\) replicate widths. Its estimated MCSE is \(s_W/\sqrt{R}\). This follows because, for independent widths with finite variance, the variance of their average is the variance of one width divided by \(R\); \(s_W^2\) estimates that variance. Unlike coverage, width is not binary, so its Monte Carlo uncertainty depends on the observed spread of widths.
Worked Example: Monte Carlo Uncertainty in Estimated Coverage
In a hypothetical simulation of a 95% confidence interval for clearance, \(R=400\) datasets are generated under one fixed scenario. The interval contains the true clearance in 372 replicates. Estimate coverage and quantify its Monte Carlo uncertainty.
The empirical coverage is
Thus the procedure covered the true clearance in 93.0% of these simulated datasets, 2.0 percentage points below the nominal 95%. The estimated MCSE is
The approximate 95% Monte Carlo interval is
This interval expresses the finite-replicate uncertainty in estimating coverage. It includes 0.95, so these 400 replicates do not sharply distinguish the observed 93.0% from nominal coverage. That does not establish that the procedure truly has exactly 95% coverage; more replicates could reduce Monte Carlo uncertainty. The interval is not a confidence interval for clearance.
Compare Coverage and Width Together
A design comparison should report coverage and width side by side for each scenario and parameter. Coverage assesses calibration: whether the interval procedure includes the truth at the intended frequency. Average width assesses typical precision on the chosen scale. Neither metric alone gives a complete comparison. In particular, an interval can be made wider to increase coverage, so a higher coverage estimate is not automatically evidence of a better design.
The following example assumes independent simulated replicates within each candidate design. It compares point estimates and their Monte Carlo uncertainty; it does not establish that a difference between designs is statistically decisive. That distinction matters when replicate counts are limited.
Worked Example: Comparing Two Designs for Clearance
Two hypothetical designs are evaluated under the same scenario using 1,000 replicates each. The target is clearance in L/h, and both use the same nominal 95% interval procedure. Design A covers the true clearance in 944 replicates and has mean interval width 0.42 L/h with width standard deviation 0.18 L/h. Design B covers it in 918 replicates and has mean width 0.31 L/h with width standard deviation 0.14 L/h. Calculate the summaries and interpret the trade-off.
For Design A, empirical coverage and its MCSE are
Its average width is 0.42 L/h, and the MCSE of that average is
For Design B,
Its mean width MCSE is
Design A is closer to the nominal 95% coverage in this scenario: 94.4% versus 91.8%. Design B produces intervals that are narrower on average by \(0.42-0.31=0.11\) L/h, but its lower observed coverage indicates more frequent misses in these simulations. These summaries suggest a precision-calibration trade-off; they do not alone determine which design is preferable. The decision depends on whether the coverage shortfall is acceptable for the intended use and whether the narrower intervals materially improve the study’s purpose.
Parameter-Wise and Simultaneous Coverage
When an analysis estimates several parameters, “coverage” must specify whether it refers to an individual parameter or to all targets jointly. Parameter-wise, or marginal, coverage for \(\theta_j\) is the proportion of replicates in which that parameter’s interval contains \(\theta_j\). Simultaneous coverage is the proportion of replicates in which every interval in a specified collection contains its corresponding true parameter. Simultaneous coverage is generally no greater than any of the component marginal coverages, because the event that all intervals cover is contained within each event that one particular interval covers.
Worked Example: Marginal Versus Simultaneous Coverage
In a hypothetical study simulation with \(R=500\), the analysis reports separate intervals for clearance, volume, and an absorption rate. Their respective true values are covered in 470, 475, and 460 replicates. In 430 replicates, all three intervals cover their true values at once. Calculate the marginal and simultaneous coverage estimates.
The marginal estimates are \(470/500=0.940\) for clearance, \(475/500=0.950\) for volume, and \(460/500=0.920\) for the absorption rate. The simultaneous estimate is
Its estimated MCSE is
The 86.0% simultaneous coverage is not inconsistent with the marginal estimates: all three intervals must succeed in the same replicate for a simultaneous success. Reporting only the three marginal results would not answer whether the full parameter set is covered together. Conversely, the simultaneous figure does not say which parameter is most often missed; the marginal figures help locate that distinction.
Diagnostics, Alternatives, and Limitations
Interpret results against a stated target and within each relevant scenario. A useful summary table includes the scenario, design, parameter and scale, nominal confidence level, replicate count, empirical coverage, its MCSE, mean width with units, and width MCSE. If the purpose requires a minimum acceptable coverage, state that criterion before examining results. A small departure from nominal coverage may be ordinary Monte Carlo variation, while a large or consistent departure across scenarios calls for closer examination of the design and analysis procedure.
Several checks help prevent misleading summaries:
- Check the coverage event. Confirm that endpoints and the true value are on the same scale, and that inclusion is evaluated consistently, including at endpoints.
- Keep the denominator visible. State \(R\) and how many replicates yielded defined intervals. Quietly excluding replicates changes the population being summarized and can make coverage appear better or worse.
- Keep scenarios distinct. Pooling scenarios with different true parameters or data-generating conditions can hide poor coverage in an important case. If an overall summary is needed, its scenario weights should match the intended target population of scenarios.
- Check units and scale for width. Widths are comparable only when they describe the same parameter on a comparable scale. A numerical width without units is incomplete.
- Separate simulation noise from design behavior. A coverage estimate has Monte Carlo uncertainty. Increasing \(R\) reduces that uncertainty, but it does not repair a procedure that is systematically miscalibrated.
Coverage and width complement, rather than replace, other performance summaries. Bias and root mean squared error describe point-estimation error, while coverage concerns the interval procedure’s repeated-sampling inclusion rate. An interval-width distribution, including its median or upper quantiles, can reveal variation that a mean alone conceals. The nominal confidence level is a benchmark, not a guarantee: the realized coverage depends on the data-generating scenario, sample size, estimator, interval construction, and whether the model assumptions are adequate.
A frequent pitfall is to rank designs by width alone: “Design B has the narrowest intervals, so it is best.” That reasoning ignores whether the intervals cover the truth at the intended rate. The correction is to examine width alongside empirical coverage and its Monte Carlo uncertainty, then judge the trade-off against prespecified scientific requirements. Another error is to say that a 95% confidence interval has a 95% probability of containing the fixed parameter after it is computed. The 95% statement concerns the long-run coverage of the procedure under its assumptions, not the probability assigned to the parameter by one realized interval.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- In 800 replicates, a procedure covers the truth 752 times. What is empirical coverage, and what is its estimated MCSE?
- Why does increasing the number of simulation replicates reduce Monte Carlo uncertainty but not necessarily improve the interval procedure’s actual coverage?
- For a parameter measured in mg/L, a design has average interval width 0.25. What additional information is needed to interpret that width and compare it with another design?
- Three parameters have marginal coverage estimates of 0.94, 0.93, and 0.96. Can simultaneous coverage exceed 0.93? Explain.
- A simulation reports coverage only for replicates that produced an interval, without stating how many analyses failed. What is the interpretive problem?
Solutions. (1) \(\hat{C}=752/800=0.940\). Its estimated MCSE is \(\sqrt{0.940(0.060)/800}=\sqrt{0.0000705}=0.00840\), or about 0.84 percentage points. (2) More independent replicates reduce the Monte Carlo standard error, which is approximately proportional to \(1/\sqrt{R}\). They make the estimate more stable but do not change the underlying coverage probability of the specified procedure. (3) State the interval procedure, scenario, nominal level, replicate count, and uncertainty of the estimated mean width; comparisons also require the same parameter and a comparable scale. (4) No. The event that all intervals cover is a subset of the event that the parameter with marginal coverage 0.93 is covered, so simultaneous coverage cannot exceed 0.93. (5) The reported value is conditional on obtaining an interval and may not represent performance across all simulated datasets. The number and treatment of failed analyses must be reported so the coverage estimand and denominator are clear.