Why Calibrate Interval Width?
A posterior interval gives a range of values for an estimand after observing a trial’s data. Its width is a direct measure of how concentrated that interval is on the estimand’s scale: a narrower interval is more precise in this particular sense. But a single trial’s interval does not describe how interval precision varies across trials. For calibration, we simulate repeated trials under a specified scenario and examine the distribution of their interval widths.
This distribution answers questions such as: How wide are intervals typically? How variable is their width from trial to trial? Does a proposed design tend to produce narrower intervals than an alternative? These are questions about precision. They are not questions about whether the intervals include the true estimand. A narrow interval can miss the truth, and a wide interval can include it. Width must therefore be considered separately from coverage, the topic of the earlier material on Coverage Calibration Of Bayesian Credible Intervals.
The interval rule and estimand need to be held fixed for a meaningful comparison. For example, compare 95% equal-tailed intervals with 95% equal-tailed intervals for the same treatment-effect estimand, rather than mixing interval levels, interval construction rules, or outcome scales. If interval endpoints are expressed in days, the width is in days; if they are expressed as a risk difference, the width is in proportion units.
Summarizing the Width Distribution
Useful summaries answer different questions. The mean width estimates average interval width under the trial-generating scenario. The median describes the middle replicate and is less sensitive to a few unusually wide intervals. Quantiles show the spread and tail behavior; the minimum and maximum show the observed extremes but can depend strongly on the number of replicates. Reporting more than one summary is often helpful, especially if the distribution is skewed.
Here \(\bar{W}\) is the average width across the simulated trials. It is a Monte Carlo estimate of the expected interval width under the specified scenario and analysis procedure. It is not the width of a single posterior interval, nor is it a posterior expectation conditional on one observed trial.
Worked Example: Summarizing Widths in One Scenario
In a hypothetical simulation of eight trials, the estimand is the difference in time to symptom resolution, measured in days. Each trial produces a 95% posterior interval using the same interval rule. The observed widths are 4, 6, 5, 8, 7, 5, 9, and 6 days.
Mean width. The widths sum to \(4+6+5+8+7+5+9+6=50\) days, so
Median width. In sorted order, the widths are 4, 5, 5, 6, 6, 7, 8, and 9 days. With eight values, the median is the average of the fourth and fifth values: \((6+6)/2=6\) days.
Observed range. The smallest width is 4 days and the largest is 9 days. Thus, the simulated widths varied from 4 to 9 days. The mean and median give complementary descriptions: the average is 6.25 days, while the middle of the observed distribution is 6 days. Neither summary says whether any interval covered the true treatment effect.
Comparing Scenarios
A scenario may specify features such as sample size, allocation, or the data-generating conditions. To compare precision, simulate trials under each scenario, apply the same posterior analysis and interval rule, then compare their width distributions. State what differs between scenarios and what is held constant. If several features change at once, the width comparison describes the combined change; it does not isolate the effect of any one feature.
Worked Example: Comparing Two Trial Designs
Suppose two hypothetical designs are simulated under the same treatment effect and outcome model. Each of six simulated trials per design produces a 95% posterior interval for the treatment effect in days. Design A has widths 4, 5, 5, 6, 6, and 7 days. Design B has widths 3, 4, 6, 7, 8, and 14 days.
For Design A, the total is 33 days and the mean width is \(33/6=5.50\) days. The sorted widths are 4, 5, 5, 6, 6, 7, so the median is \((5+6)/2=5.50\) days.
For Design B, the total is 42 days and the mean width is \(42/6=7.00\) days. The sorted widths are 3, 4, 6, 7, 8, 14, so the median is \((6+7)/2=6.50\) days.
In this small hypothetical simulation, Design A has a lower mean and median width. Design B also has a notably wide observed interval of 14 days, showing why a single average can conceal variability. The result supports a comparison of simulated precision under these specified conditions; six replicates are not enough to characterize the tails reliably, and the result does not establish which design has better coverage.
When the same simulated replicate conditions are deliberately paired across scenarios, compare paired width differences rather than treating the two sets as unrelated. Define \(D_r=W_{r,A}-W_{r,B}\) for each matched replicate and summarize the \(D_r\) values. Negative values indicate that scenario A’s interval was narrower for that pair. Pairing can reduce Monte Carlo noise when the simulation design supports it, but it does not change the meaning of width or remove the need to report the scenarios clearly.
Monte Carlo Uncertainty in Average Width
The estimated mean width varies because only a finite number of trials are simulated. If the replicate widths are independent draws from the same scenario’s trial-generating process and have finite variance, the Monte Carlo standard error (MCSE) of their mean is estimated by the sample standard deviation of the widths divided by \(\sqrt{R}\). This describes simulation uncertainty in \(\bar W\), not the uncertainty in the treatment effect within any one trial.
The divisor \(R-1\) makes \(s_W^2\) the usual sample variance estimate of the replicate-width variance. As \(R\) increases, the MCSE typically decreases at a rate proportional to \(1/\sqrt{R}\), provided the simulation design and width distribution remain the same.
Worked Example: Quantifying Simulation Error
In a hypothetical run of \(R=100\) independent simulated trials, the average 95% posterior interval width for a treatment effect measured in days is 12 days, and the sample standard deviation of the widths is 3 days.
The estimated MCSE of the average width is
Thus, the Monte Carlo estimate of the mean width is 12 days with an estimated simulation standard error of 0.300 days. This quantifies the repeat-to-repeat variability of the estimated average if the simulation were rerun with new independent replicates. It is not an interval-width estimate for an individual future trial. If a confidence interval for the Monte Carlo mean is reported, its justification and assumptions should be stated; a normal approximation may be unreliable when \(R\) is small or the width distribution is strongly skewed.
Justification and Practical Checks
The mean width has a simple repeated-simulation interpretation. Let \(W_r\) be independent widths generated under one fixed scenario, with finite mean \(E(W_r)=\mu_W\) and variance \(\operatorname{Var}(W_r)=\sigma_W^2\). Linearity of expectation gives \(E(\bar W)=\mu_W\). Independence gives the variance of the average:
The middle equality uses independence, so the covariances between distinct replicate widths are zero. Taking the square root gives \(\operatorname{SD}(\bar W)=\sigma_W/\sqrt{R}\), which motivates estimating this standard deviation with \(s_W/\sqrt{R}\). If replicates are dependent, that MCSE formula need not apply as written; account for the dependence or redesign the simulation.
Before interpreting a comparison, check that every interval has ordered endpoints, that widths are nonnegative, and that the recorded units are consistent. Confirm that all scenarios used the intended interval level, estimand, posterior model, and interval rule. Review unexpectedly extreme widths and, where applicable, check that computation of posterior summaries converged reliably. A width of zero or an implausibly large width may reflect a data or coding error, a genuine feature of the model, or both; investigate rather than automatically deleting it.
| Summary or approach | What it describes | What it does not establish |
|---|---|---|
| Mean width | Average interval width across simulated trials | Coverage or accuracy in any particular trial |
| Median and width quantiles | Typical width and variation across trials, including distribution tails | Whether intervals include the true estimand |
| Posterior standard deviation | Posterior spread for the data from an individual trial | The repeated-trial distribution of a chosen interval width |
| Coverage rate | How often simulated intervals contain the generating value | How wide the intervals are by itself |
Width is attractive because it remains in the estimand’s units and is easy to explain clinically. However, interval width depends on the interval construction rule: equal-tailed and highest-posterior-density intervals can differ, particularly for skewed or multimodal posteriors. It also depends on the estimand’s scale. A change from a risk difference to a log odds ratio changes the units and interpretation, so raw widths on those scales should not be compared as though they were directly commensurate.
A common mistake is to say, “This design has narrower intervals, so it has better coverage.” That conclusion does not follow. Width measures the span of the intervals; coverage assesses whether they contain the generating value over repeated trials. Narrowing an interval could leave coverage unchanged, increase it under some circumstances, or reduce it. Report width summaries alongside a coverage assessment when both precision and interval performance matter. Keep the claims separate.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A simulated trial produces an interval from 1.2 to 4.7 days for a treatment effect. What is its width and unit?
- Two scenarios have the same mean width, but one has a much larger upper quantile. What does that say about their width distributions?
- In 64 independent replicates, the sample standard deviation of widths is 4 days. Calculate the estimated MCSE of the mean width.
- Why is it invalid to conclude that a scenario with narrower posterior intervals has better coverage?
- Two scenarios use different estimand scales and different interval rules. Name two reasons their raw width averages may not be directly comparable.
Solutions. (1) The width is \(4.7-1.2=3.5\) days. (2) Their average widths agree, but the scenario with the larger upper quantile has a more extended upper tail or more unusually wide intervals; additional quantiles or a distribution plot can clarify the difference. (3) The estimated MCSE is \(4/\sqrt{64}=4/8=0.500\) days. (4) Width describes the span of intervals, whereas coverage describes how often they contain the generating value. Narrow intervals can miss that value; width alone does not count coverage. (5) The estimands have different units or scales, and the interval rules can produce different widths even for comparable posterior information. A fair comparison should align the estimand scale and interval construction or justify a transformation.