Bayesian Credible Intervals and Frequentist Coverage

Two Different Meanings of “95%”

When a trial selects the treatment arm with the most favorable observed result, an interval for that arm must be understood in relation to the selection rule. A Bayesian credible interval and a frequentist confidence interval answer different probability questions. The distinction matters even when both intervals are calculated from the same normal model and both are labelled “95%.”

A Bayesian credible interval assigns posterior probability to a parameter, conditional on the observed data and the prior model. Frequentist coverage instead describes how often a procedure contains a fixed parameter over repeated trials. Selection can change that repeated-trial probability: among several noisy estimates, the largest tends to be above its true value. A prior can shrink the selected estimate toward its prior center, changing that effect but not automatically guaranteeing nominal coverage.

These statements do not conflict. The credible interval probability is conditional on the data and prior model; the coverage probability is evaluated before observing the data, with the parameter held fixed. Neither statement, by itself, guarantees the other.

A Normal Model for Selection

Consider \(K\) treatment arms with estimated effects \(X_1,\ldots,X_K\). To isolate selection from other complications, suppose the estimates are independent and \(X_j\mid\theta_j\sim N(\theta_j,v)\), where the sampling variance \(v\) is known and common across arms. Larger effects are favorable, and the selected arm is \(J=\arg\max_j X_j\); write \(M=X_J=\max_j X_j\). Assume a proper, independent normal prior \(\theta_j\sim N(0,\tau^2)\) for each arm. The prior center is zero, representing no effect on the chosen effect scale. These assumptions are simplifying: real trials may have correlated estimates, unequal variances, unknown variance, or a selection rule involving interim data and sample size.

For each arm, conjugacy gives a normal posterior. Define the shrinkage factor \(\kappa=\tau^2/(\tau^2+v)\) and posterior variance \(w=\tau^2v/(\tau^2+v)=\kappa v\). Conditional on all observed estimates, the posterior for the selected arm’s effect is \(N(\kappa M,w)\). Therefore its equal-tailed \(100(1-\alpha)\%\) credible interval is \(\kappa M\pm z_{1-\alpha/2}\sqrt{w}\), where \(z_{1-\alpha/2}\) is the standard normal quantile.

This is a posterior interval conditional on the complete vector of estimates and the specified model. Since the selected index \(J\) is determined by that vector, selection does not add information once the full vector is already observed. It would be different to report only the winning estimate and discard the other results: then the fact that it won is part of the information, and a likelihood for the reported maximum must account for selection.

Why Coverage Changes After Choosing the Maximum

For a frequentist calculation, first hold the true effects fixed. In the special case \(\theta_1=\cdots=\theta_K=\theta\), the estimates are independent \(N(\theta,v)\), and the distribution function of their maximum is \(P_\theta(M\le m)=\Phi((m-\theta)/\sqrt v)^K\). Here \(\Phi\) is the standard normal distribution function. The selected arm has true effect \(\theta\), so its credible interval covers that effect exactly when \(\kappa M-z\sqrt{w}\le\theta\le\kappa M+z\sqrt{w}\). For \(\kappa>0\), this is equivalent to \((\theta-z\sqrt{w})/\kappa\le M\le(\theta+z\sqrt{w})/\kappa\). The coverage is the probability of this interval for \(M\), namely the CDF at the upper endpoint minus the CDF below the lower endpoint.

This expression follows by substituting the two bounds for \(M\) into its maximum distribution. In particular, it is a difference of two CDF values, not just the probability that \(M\) is below the upper endpoint. It depends on the number of candidate arms, the prior through \(\kappa\) and \(w\), and the fixed true effect relative to the prior center.

When the true effects differ, the selected arm’s true effect is itself determined by the random winner. Under the independent normal model, arm \(j\) wins with estimate \(x\) in a small interval with density \(f_j(x)\prod_{\ell\ne j}F_\ell(x)\), where \(f_j\) and \(F_j\) are the density and CDF of \(N(\theta_j,v)\). Thus coverage of the selected-arm interval is

The indicator is one when the interval based on the winning estimate \(x\) contains the true effect of the winning arm, and zero otherwise. This formula makes clear why a coverage claim cannot generally be inferred from the posterior probability alone.

Assumptions, Diagnostics, and Alternatives

The calculations above require a correctly specified normal sampling model, independent arm estimates with known common variance, the stated prior, and selection by the largest observed estimate. The examples also assume the true effects are equal when using the simple maximum-CDF calculation. If estimates are correlated, variances differ, the winner is chosen using an interim statistic, or the trial changes its allocation or sample size, the distribution of the selected estimate must reflect that design. Using the formula for independent, equal-variance estimates in those settings can give an incorrect coverage calculation.

A practical diagnostic is to write down the actual selection rule and trace which observations determine both the winner and the reported interval. Check that the likelihood includes the data collected under the adaptation and that the prior is stated on the same effect scale as the estimates. For frequentist assessment, repeated simulation under fixed parameter configurations can estimate the procedure’s coverage, provided each simulated trial repeats the full adaptation and selection rule. Such simulation diagnoses performance for the scenarios examined; it does not prove a universal guarantee.

The main alternatives target different inferential goals. A Bayesian analysis can retain the posterior interval and report its model-conditional interpretation, possibly with sensitivity analyses over reasonable priors. A frequentist analysis can instead use an interval procedure calibrated to the selection design and evaluate its coverage under prespecified fixed effects. A simple interval that ignores the winner’s selection may be easier to calculate but should not be presented as having nominal post-selection coverage without justification. These approaches are not interchangeable: the appropriate choice depends on whether posterior probability under a model or repeated-sampling error control is the primary goal.

A further limitation is that nominal posterior credibility is only as relevant as the model and prior are to the scientific question. A prior centered at zero may be reasonable in one setting and poorly calibrated in another; independence may be implausible when arms share a control group. If only the selected result is available, conditioning on the fact that it was selected cannot be replaced by treating it as an unselected single-arm observation. The information lost by withholding the other estimates can materially change inference.

Common Mistake

A related arithmetic error is to calculate only \(P(M\le U)\) for an upper endpoint \(U\). For coverage between lower endpoint \(L\) and upper endpoint \(U\), the correct probability is \(P(M\le U)-P(M<L)\). Omitting the lower-tail probability does not give interval coverage.