Effective Sample Size And Monte Carlo Standard Errors

Why Effective Sample Size Matters

MCMC output may contain thousands of retained draws, but consecutive draws are often dependent. If a chain moves slowly through the posterior distribution, many draws provide less information about a posterior summary than the same number of independent draws. The effective sample size (ESS) expresses that difference in a familiar way: it is the approximate number of independent draws that would yield the same Monte Carlo precision for a particular summary.

ESS is summary-specific. A chain may estimate one parameter’s posterior mean precisely while estimating a different function of the parameters less precisely. Here, the target is a scalar quantity \(h(\theta)\), where \(\theta\) denotes the trial model parameters and \(h\) may be a parameter itself, a treatment-effect contrast, or an indicator for a posterior event. The aim is to measure simulation error in the average of \(h\), not uncertainty about the unknown clinical effect itself.

When positive autocorrelation persists, \(\tau\) exceeds 1 and \(M_{\mathrm{eff}}\) is smaller than the number \(M\) of retained draws. Negative autocorrelation can reduce \(\tau\), so an ESS estimate can sometimes exceed \(M\). ESS is not a literal count of distinct posterior states; it is a variance-equivalent measure for the particular \(h\) being averaged.

Why Autocorrelation Determines Precision

The posterior expectation \(E[h(\theta)\mid y]\), where \(y\) denotes the observed trial data, is commonly estimated by the MCMC average \(\bar{h}=M^{-1}\sum_{t=1}^{M}h_t\). If the retained chain is stationary, its variance can be written in terms of autocovariances:

The first term is the contribution from each draw’s variance. The remaining terms account for dependence between draws separated by each lag. Under standard conditions for a Markov chain central limit theorem, including a finite variance and sufficiently controlled dependence, the large-\(M\) approximation is:

This gives the ESS interpretation directly. An independent sample of size \(M_{\mathrm{eff}}\), with variance \(\gamma_0\), has mean variance \(\gamma_0/M_{\mathrm{eff}}\). The chain’s Monte Carlo standard error (MCSE) is therefore estimated by substituting estimates of the marginal variance and integrated autocorrelation time:

For a posterior mean, \(\hat{\gamma}_0\) is often estimated by the sample variance \(s_h^2\) of the retained values of \(h\). The result measures the approximate standard deviation of the simulation error in \(\bar{h}\) over repeated runs of the computation, under the stated chain assumptions. It does not measure the spread of the posterior distribution.

Posterior Probabilities as Averages

A posterior probability for an event can also be estimated from MCMC output. For example, define \(I_t=1\) if the treatment effect in draw \(t\) satisfies a specified condition, and \(I_t=0\) otherwise. The posterior probability of that event is estimated by the fraction of draws satisfying it, which is the average of the indicator sequence. Its marginal variance is \(p(1-p)\), where \(p\) is the event’s posterior probability. The same ESS and MCSE logic applies to \(I_t\).

In practice, \(p\) is unknown, so estimate the indicator variance with \(\hat{p}(1-\hat{p})\), where \(\hat{p}\) is the observed fraction. The autocorrelation used for the ESS must be that of the indicator sequence, not necessarily the autocorrelation of an underlying continuous parameter. A slowly changing indicator can make a probability estimate less precise than one might infer from the parameter’s autocorrelation alone.

Estimating ESS and MCSE from a Finite Chain

An actual chain provides only finitely many draws, so its autocorrelation at every lag cannot be estimated reliably. A direct sum of all estimated autocorrelations is especially unstable: at large lags, the estimates are noisy, and summing that noise can produce a misleading ESS. Practical MCMC software therefore uses a rule or estimator that controls the contribution of noisy high-lag autocorrelations. Estimates may also be based on batch means or spectral variance methods, which estimate the long-run variance of the chain average directly.

For a transparent hand calculation, one can use a short, credible set of estimated autocorrelations as an approximation, provided the cutoff is justified rather than chosen to produce a preferred answer. The calculation below illustrates the arithmetic; it is not a universal prescription for truncation. In applied work, inspect the chain and use a reliable estimator implemented for MCMC output. For multiple chains sampling the same posterior, use a chain-aware ESS estimator rather than treating separate chains as one uninterrupted sequence across chain boundaries.

Assumptions, Diagnostics, and Limitations

The formulas are most useful when the retained chain is sampling its intended posterior distribution and behaves sufficiently like a stationary, ergodic chain for averages to stabilize. The Markov chain central limit approximation additionally requires suitable dependence and moment conditions; a finite second moment is essential for the variance-based formulas. Warmup draws should not be included as though they were draws from the target posterior. A good-looking ESS cannot establish that the chain has reached the right distribution or explored all relevant regions.

Interpret ESS and MCSE alongside trace plots and the convergence diagnostics used for multiple chains. Warning signs include trends, long stretches with little movement, chains occupying visibly different regions, or large changes in ESS and MCSE when the run is extended. A very large number of stored draws alone does not guarantee precision. If the estimated ESS is small relative to \(M\), dependence has substantially reduced the information in the retained draws. If estimated autocorrelations are noisy or the chain is not stable, the MCSE estimate itself may also be unreliable.

For a posterior mean, a useful practical question is whether its MCSE is small relative to the precision needed for the intended report. For a probability estimate, consider the MCSE in the probability’s units: an MCSE of \(0.01\) means about one percentage point. This is a computational precision check, not a decision threshold and not a substitute for assessing posterior uncertainty.

For independent draws, autocorrelations are zero, \(\tau=1\), and ESS equals \(M\); the usual independent-draw MCSE follows. Batch means and spectral variance estimators are alternatives for estimating the long-run variance, and can be useful when a direct autocorrelation sum is unstable. These approaches address the same computational uncertainty, but their estimates still depend on a stable chain and adequate simulation. ESS and MCSE also do not diagnose every possible sampling failure, such as a chain that has missed a separate posterior mode.

Common Mistake