Autocorrelation, Thinning, And Information Loss

Why Autocorrelation Matters

Posterior simulation produces a sequence of draws, not a collection of independent observations. In an MCMC chain, nearby draws are often similar because each new state is generated using the current state. Consequently, 2,000 saved draws may contain less information about a posterior quantity than 2,000 independent draws would. The relevant question is not just how many values were saved, but how strongly they depend on one another.

This tutorial connects that dependence to Monte Carlo precision and examines thinning, the practice of retaining only selected iterations. Thinning can make a file smaller, but it does not make the underlying chain explore the posterior better. Whether it helps a particular analysis depends on what is being optimized: storage, computation after sampling, or precision per MCMC transition.

Lag Autocorrelation and Effective Sample Size

Let \(\theta_1,\ldots,\theta_n\) be post-warmup draws from a stationary chain, and let \(g(\theta)\) be a scalar posterior quantity of interest, such as a treatment-effect parameter or a function of several parameters. Write \(Y_t=g(\theta_t)\). The lag-\(k\) autocorrelation describes the linear dependence between values \(k\) iterations apart.

The effective sample size (ESS) is an interpretation of precision, not a literal count of independent draws hidden inside the chain. If \(\tau=4\), for example, \(n\) correlated draws provide roughly the same precision for estimating the mean of \(g(\theta)\) as \(n/4\) independent draws with the same marginal variance. ESS is specific to the quantity being estimated: different parameters or functions \(g\) can have very different autocorrelation patterns.

The variance approximation relies on a chain that has reached its stationary target, has finite variance for \(g(\theta)\), and has dependence that weakens sufficiently for a central limit approximation to apply. It is not a correction for a chain that has failed to explore the target distribution. In practice, \(\rho_k\) and \(\tau\) must be estimated from a finite run, so ESS and Monte Carlo standard errors are themselves uncertain.

How the Variance Formula Arises

A short derivation explains why positive autocorrelation reduces precision. Suppose \(Y_t\) is stationary with mean \(\mu_g\) and autocovariance \(\gamma_k=\operatorname{Cov}(Y_t,Y_{t+k})=\sigma_g^2\rho_k\). The sample mean is \(\bar{Y}=n^{-1}\sum_{t=1}^{n}Y_t\). Expanding its variance and grouping pairs of observations by their lag gives

If the autocorrelations are summable and \(n\) is large, the factor \(1-k/n\) is close to 1 for lags that materially contribute to the sum, yielding \(\operatorname{Var}(\bar{Y})\approx \sigma_g^2\tau/n\). For independent draws, every \(\rho_k=0\) for \(k>0\), so \(\tau=1\) and the usual independent-draw variance is recovered. With positive dependence, \(\tau\) is generally greater than 1, reducing the effective sample size.

This is an asymptotic justification, not a license to sum noisy estimated autocorrelations across every available lag. At large lags, estimates are unstable. Practical estimators use truncation or smoothing rules; reported ESS and Monte Carlo standard errors should therefore be treated as diagnostics with finite-run uncertainty, rather than exact properties of the chain.

What Thinning Changes

Thinning by an integer \(m\) means retaining every \(m\)th post-warmup draw, for example iterations \(m,2m,3m,\ldots\). If the original chain has lag autocorrelation \(\rho_k\), the thinned sequence has lag-\(j\) autocorrelation \(\rho_{jm}\). Thinning therefore reduces both the number of retained draws and the dependence between adjacent retained values. Its effect on precision must account for both changes, not just the reduced autocorrelation.

When the Autocorrelation Is Negative

Autocorrelation need not be positive. Some samplers or parameterizations produce alternating values, and then negative lag correlations can make the sample mean more precise than an independent-draw calculation would suggest. In that case \(\tau\) can be below 1, and the corresponding asymptotic ESS can exceed the number of draws. This does not mean there are literally more draws; it means the dependence reduces the variance of the mean relative to independent draws with the same marginal variance.

Diagnostics, Alternatives, and Limitations

For a real analysis, inspect autocorrelation and ESS for the posterior quantities that matter to the scientific conclusion, not just for one convenient parameter. A slow-moving treatment effect, a variance parameter, and a clinically relevant function of parameters can have different dependence patterns. Compare chains and use the trace plots and rank-normalized \(\hat{R}\) diagnostics discussed in earlier material as complementary checks. A reassuring autocorrelation estimate or large ESS does not prove that the chain has reached or adequately explored the intended posterior.

Common warning signs include autocorrelations that remain large across many lags, substantial changes in estimated ESS as the run length increases, and Monte Carlo standard errors too large for the precision needed in the analysis. Estimated autocorrelations can also be noisy, especially near the end of a finite sequence. If estimates look erratic at high lags, that is not evidence that the chain has become independent at a particular cutoff; it may simply reflect estimation noise.

The usual alternative to thinning is to retain the full post-warmup sequence and improve precision by running longer or by improving the sampler or parameterization. For a fixed amount of MCMC computation, keeping all draws usually preserves more information for estimating posterior expectations. Thinning can nevertheless be reasonable when storage is genuinely constrained or when saving and processing every draw is costly. In such cases, choose the interval based on the practical constraint, and check the resulting ESS and Monte Carlo standard error rather than assuming a particular interval is statistically optimal.

Thinning is not a remedy for poor exploration. If distinct parts of the posterior are not visited, keeping fewer draws cannot recover what was missed. Nor does thinning make a biased or nonstationary sequence valid. Finally, ESS for a posterior mean does not automatically describe precision for every other summary; the dependence structure relevant to a different function may differ.