Frequentist Calibration Of Bayesian Econometric Procedures

Why Calibrate Bayesian Procedures?

A Bayesian credible interval answers a conditional question: given the observed data and the specified model and prior, what range contains a stated proportion of the posterior distribution? A separate question is whether the procedure behaves well over repeated samples when a parameter is held fixed. In econometrics, this matters when a Bayesian interval or estimator will be used repeatedly across studies, policy settings, or data sets, and its long-run performance is of interest.

Frequentist calibration does not replace the Bayesian interpretation of a posterior. It evaluates the operating characteristics of a procedure under a specified data-generating process. The result depends on that process, the parameter value being held fixed, and the prior used to construct the Bayesian procedure. In particular, a credible interval with 90% posterior probability need not cover a fixed true parameter in 90% of repeated samples.

A Normal-Mean Model for Comparing Procedures

Use a simple model to make the distinction concrete. Suppose \(Y_1,\ldots,Y_n\) are independent observations with \(Y_i\sim N(\theta,\sigma^2)\), where the sampling variance \(\sigma^2\) is known. The parameter of interest is the population mean \(\theta\). The Bayesian analysis assigns the prior \(\theta\sim N(0,\tau^2)\), with \(\tau^2\) also specified. Under this model, the sample mean has distribution \(\bar Y\sim N(\theta,\sigma^2/n)\).

The posterior is normal. Write \(V\) for its variance and \(a\) for the weight on the sample mean in its posterior mean. Completing the square in the product of the normal likelihood and prior gives:

A central Bayesian \(100(1-\alpha)\%\) credible interval is \(a\bar Y\pm z_{1-\alpha/2}\sqrt V\), where \(z_{1-\alpha/2}\) is the corresponding standard-normal quantile. Its posterior probability is \(1-\alpha\) under the specified model and prior. For comparison, the usual normal-theory frequentist interval is \(\bar Y\pm z_{1-\alpha/2}\sigma/\sqrt n\); under the stated sampling model its coverage is exactly \(1-\alpha\) for every \(\theta\).

Deriving Fixed-Parameter Bias and Coverage

The posterior mean is a data-dependent estimator, \(\hat\theta_B=a\bar Y\). Under repeated sampling at a fixed \(\theta\), its expectation and variance follow from \(\bar Y\sim N(\theta,\sigma^2/n)\):

When \(0<a<1\), the estimator shrinks the sample mean toward the prior mean, zero. Its bias is therefore toward zero: negative for positive \(\theta\), positive for negative \(\theta\), and zero at \(\theta=0\). This bias is not automatically a defect; shrinkage can trade bias for reduced sampling variability. But it must be measured at the parameter values relevant to the application.

The Bayesian interval has a fixed half-width \(c=z_{1-\alpha/2}\sqrt V\). To find its repeated-sampling coverage, standardize the random posterior mean \(a\bar Y\), whose repeated-sampling distribution is normal:

where \(\Phi\) is the standard normal cumulative distribution function. This expression shows why credible level alone does not determine fixed-parameter coverage. The interval width comes from the posterior variance, while the repeated-sampling distribution of its center has its own variance and may be shifted away from \(\theta\) by shrinkage.

Estimators and Intervals Have Different Operating Characteristics

An interval procedure and a point estimator should be assessed separately. For the model above, the frequentist estimator \(\bar Y\) is unbiased because \(E_\theta[\bar Y]=\theta\), and its variance is \(\sigma^2/n\). The posterior mean \(a\bar Y\) is biased except at \(\theta=0\), but its variance is \(a^2\sigma^2/n\), smaller than that of \(\bar Y\) when \(0<a<1\). Bias alone does not rank estimators; a fuller comparison may consider mean squared error, which equals variance plus squared bias.

Likewise, the fixed-parameter coverage of a Bayesian interval does not say whether the interval is useful in a particular analysis. Its posterior probability statement remains valid relative to the specified posterior. Calibration instead asks how often the procedure contains the fixed truth under a designated repeated-sampling experiment. Both descriptions can be relevant, but neither should be substituted for the other.

Calibration Averaged Over a Prior

Fixed-parameter calibration is not the only way to average performance. If the parameter is itself generated from the same prior used in the analysis, Bayesian credible sets have a precise prior-averaged coverage property. This is sometimes called prior-predictive calibration. It does not guarantee nominal coverage at each fixed parameter value.

The justification is the law of total probability applied to the joint prior-predictive experiment:

This identity concerns the joint prior-predictive distribution. It can coexist with undercoverage at some fixed values and overcoverage at others. The prior-averaged property is informative when that prior represents the distribution of parameter values across the situations being evaluated; it is not a substitute for fixed-parameter assessment when a particular parameter value or range is the target.

A Practical Calibration Workflow

A useful evaluation begins by stating exactly which distribution is used to generate repeated data. For an econometric application, that statement should identify the parameter values or parameter-generating distribution, the likelihood, sample size, covariates or design, and any assumed error structure. Calibration conclusions apply only to the conditions represented in that experiment.

In Bayesian computations based on simulation, numerical approximation can introduce additional error into posterior summaries. That computational error should not be confused with Monte Carlo uncertainty from generating repeated data sets. Where practical, posterior computation should be sufficiently accurate that it does not materially distort the estimated bias or coverage.

Common Mistake

Wrong reasoning: “The interval is a 90% Bayesian credible interval, so it must contain the true coefficient in 90% of repeated samples.” The 90% statement is conditional on the observed data and the chosen posterior. Fixed-parameter coverage requires a separate calculation or simulation under a sampling distribution with the true coefficient held fixed.

Correction: State the averaging distribution before reporting a calibration percentage. For a fixed-parameter claim, use \(P_\theta\{\theta\in C(Y)\}\) at specified \(\theta\). For a prior-averaged claim, average over parameter values drawn from the prior as well as the data. Also report estimator bias separately from interval coverage: a procedure can have a biased point estimator and an interval with useful coverage, or the reverse.