Monte Carlo Error in Posterior Variance Estimates

Why Posterior Variance Estimates Need Their Own Error Measure

An MCMC run produces a finite sequence of draws, so a posterior variance calculated from those draws has simulation error. That error is not the posterior uncertainty being summarized: it is the uncertainty in the numerical estimate caused by using a finite simulation. Because MCMC draws are generally autocorrelated, the usual independent-draw standard error can substantially misstate this error.

For a variance component, the relevant serial dependence is especially important. Estimating a posterior variance involves squared deviations from a posterior mean, and those squared deviations can be more strongly autocorrelated than the original draws. The Monte Carlo standard error (MCSE) must therefore be based on the sequence of squared deviations, or an equivalent variance-estimating summary, rather than automatically reusing the dependence estimate for the raw parameter.

The \(N\) draws used here are post-warmup draws intended to represent a stationary chain. Warmup draws used to initialize or adapt a sampler are not treated as stationary posterior draws. The variance estimator uses divisor \(N\), because the target is the posterior expectation defining \(V\). Using \(N-1\) gives the familiar finite-sample correction for an independent sample; for a consistent MCMC estimate, the difference between these divisors is asymptotically negligible.

Deriving the Monte Carlo Standard Error

Set \(Y_t=(\psi_t-\mu)^2\). Then \(E(Y_t\mid y)=V\), and estimating the posterior variance is, to first order, estimating the posterior mean of the sequence \(Y_t\). The key distinction from an independent sample is that the variance of an average of dependent draws depends on autocovariances at all lags.

Write \(\gamma_k=\operatorname{Cov}(Y_t,Y_{t+k}\mid y)\). If these autocovariances are summable, the long-run variance of the mean of \(Y_t\) is \(\Omega_Y=\gamma_0+2\sum_{k=1}^{\infty}\gamma_k\). The asymptotic variance of its sample mean is \(\Omega_Y/N\). The estimated center \(\bar\psi\) does not change this first-order result:

Under a central limit theorem for \(\bar\psi\), \(\bar\psi-\mu=O_p(N^{-1/2})\), so \((\bar\psi-\mu)^2=O_p(N^{-1})\). Multiplication by \(\sqrt{N}\) makes this centering correction \(o_p(1)\). Thus the asymptotic fluctuation of \(\hat V_N\) is the same as that of the average of \(Y_t\).

In practice, \(\mu\) is unknown, so calculate \(h_t=(\psi_t-\bar\psi)^2\) and estimate the long-run variance of the \(h_t\) sequence. Centering \(h_t\) by its sample mean gives the same autocovariance calculation. This is a plug-in estimate: it replaces unknown posterior quantities with their MCMC estimates.

Estimating Long-Run Variance

Two common approaches are batch means and a lag-window estimate of the autocovariance sum. Both target the long-run variance of the variance-summary sequence, not that of the raw parameter sequence.

The factor \(b\) is essential: the sample variance of the batch means estimates the variance of a batch average, which is approximately \(\Omega_Y/b\). Multiplying by \(b\) puts the estimate on the long-run-variance scale. The batch length should be long enough that dependence across batch means is limited, while leaving enough batches to estimate their variability. Too few batches make the estimate noisy; very short batches may fail to capture dependence.

Alternatively, estimate autocovariances \(\hat\gamma_k\) of the \(h_t\) sequence and calculate a windowed sum such as \(\hat\Omega=\hat\gamma_0+2\sum_{k=1}^{L}w_k\hat\gamma_k\), where \(w_k\) decreases toward zero as lag \(k\) approaches the chosen truncation \(L\). Truncation and weighting matter because the empirical autocovariances at large lags are noisy. Batch means and spectral or lag-window estimates are alternative numerical estimators of the same long-run variance; neither removes the need for adequate chain length and stationarity.

Use the Dependence of the Summary Being Estimated

The effective sample size for a posterior mean of \(\psi\) is based on the serial dependence of \(\psi_t\). The MCSE for a posterior variance instead depends on the serial dependence of \((\psi_t-\mu)^2\). These can differ substantially. In particular, a chain can move smoothly through the center of a distribution while taking much longer to explore its tails; squared deviations then persist, making the variance summary mix slowly.

Diagnostics, Alternatives, and Limitations

A reported MCSE should be treated as an estimate, not as a guarantee that the simulation has explored the posterior adequately. Compare estimates from reasonable batch lengths or lag windows, and check whether the variance estimate and its MCSE are stable as more stationary draws are added. Strong changes across these choices indicate that the run may be too short to estimate long-run dependence reliably. Unstable tail exploration is particularly consequential because squared deviations give large weight to distant draws.

An MCSE calculation also cannot repair a chain that has not reached its stationary target. Nor does a small estimated MCSE by itself establish that the model, prior, or sampler is appropriate. Check chain behavior and mixing using the diagnostics introduced in earlier material on MCMC mixing and effective sample size before relying on variance-summary precision. Heavy-tailed posteriors or poorly explored tails can make the fourth-moment condition doubtful and the variance-summary MCSE unstable.

When the long-run variance is estimated adequately, the MCSE describes numerical precision, not posterior spread. A large posterior variance can be estimated precisely, and a modest posterior variance can have a large MCSE if the chain explores its squared deviations slowly. Under the central limit theorem conditions, increasing the number of stationary draws by a factor of four approximately halves the MCSE, since it scales as \(N^{-1/2}\). This improvement does not apply if the additional draws do not represent the target posterior or if the relevant long-run variance is not finite.