Effective Sample Size for Posterior Estimates

Why the Number of Draws Is Not the Whole Story

A Bayesian NLME analysis may produce thousands of retained MCMC draws, but consecutive draws can be similar. If so, those draws contain less information about the posterior than the same number of independent draws. Effective sample size (ESS) expresses this loss of information by comparing the precision of a correlated chain with that of an idealized independent sample. It is a summary of how efficiently a particular posterior feature is estimated, not a count of distinct patients or a replacement for checking convergence.

For an NLME model, the quantities of interest may include population parameters such as clearance, between-subject variability parameters such as an element of \(\Omega\), or derived quantities such as an exposure contrast. A chain can explore one quantity well and another poorly. ESS therefore needs to be assessed for the summaries that matter, rather than reported as one universal property of a fit.

Definition and Assumptions

Definition: For a stationary scalar Markov chain with \(N\) retained draws of a quantity \(g(\theta)\), let \(\rho_k\) be the lag-\(k\) autocorrelation of those draws. Its integrated autocorrelation time is \(\tau=1+2\sum_{k=1}^{\infty}\rho_k\), when the sum is well-defined. The theoretical effective sample size is \(N_{\mathrm{eff}}=N/\tau\). It is the size of an independent sample that would have the same asymptotic variance for estimating the chain's mean of \(g(\theta)\).

This definition assumes the chain has reached a stationary distribution, targets the posterior of interest, and has dependence that is sufficiently well behaved for the long-run variance and autocorrelation sum to be meaningful. In practice, autocorrelations are estimated from finite chains, so the infinite sum cannot be calculated directly. Implementations use chain-aware estimators and truncate or regularize the sum to reduce the instability that would result from summing noisy autocorrelations indefinitely. The reported ESS is therefore an estimate, not an exact property known without uncertainty.

With \(M\) chains of \(N\) retained draws each, there are \(MN\) stored draws. If they were independent, the ESS would be \(MN\). Positive serial dependence typically makes ESS smaller; negative dependence can make it larger than the stored draw count for some estimands. A large ESS does not establish that chains sampled the correct posterior: the stationary-chain interpretation depends on adequate convergence and exploration.

Bulk and Tail ESS

A single ESS does not describe every feature of a posterior distribution. Modern MCMC diagnostics commonly distinguish bulk ESS from tail ESS. Both are calculated across chains and account for serial dependence, but they target different aspects of the draws.

Bulk ESS: An ESS based on rank-normalized draws, intended to assess the information available for central posterior summaries. It is useful for diagnosing efficiency around the body of a distribution and for location-like summaries.

Tail ESS: An ESS based on indicators of whether draws lie below or above specified posterior quantiles. A commonly reported version takes the smaller ESS for the lower and upper 5% tails. It assesses how well the chains explore the posterior extremes relevant to tail quantiles.

Rank normalization replaces values by their pooled ranks and transforms those ranks to a roughly standard-normal scale before estimating bulk efficiency. This makes the bulk diagnostic less sensitive to the original scale and more robust to heavy tails. Tail ESS instead asks whether the chains visit the relevant tail regions often enough and with adequate mixing. A small tail ESS can occur even when the central portion of the posterior is well explored.

These diagnostics are not interchangeable. Bulk ESS does not guarantee accurate tail quantiles, and tail ESS is not a general measure of central estimation. Nor does either one certify that a nonlinear function of the parameters is estimated well. For an important derived quantity, evaluate its own draws and diagnostics where available.

How Autocorrelation Reduces ESS

The connection between autocorrelation and ESS follows from the variance of a chain average. Let \(X_t\) be a stationary chain with variance \(\gamma_0\), and let \(\gamma_k\) be its lag-\(k\) autocovariance. The variance of the average of \(N\) draws is

\operatorname{Var}(\bar{X})= \frac{1}{N^2}\left[N\gamma_0+2\sum_{k=1}^{N-1}(N-k)\gamma_k\right].

For large \(N\), if the autocovariances are summable, this is approximately \(\frac{\gamma_0}{N}(1+2\sum_{k=1}^{\infty}\rho_k)\), where \(\rho_k=\gamma_k/\gamma_0\). An independent sample of size \(N_{\mathrm{eff}}\) has mean variance \(\gamma_0/N_{\mathrm{eff}}\). Equating these variances gives \(N_{\mathrm{eff}}=N/(1+2\sum_k\rho_k)\), which is the stated definition. This is an asymptotic justification: actual finite-chain ESS estimates depend on estimated autocorrelations and the implementation's estimator.

Worked Example: ESS for a Positively Correlated Chain

Suppose a hypothetical single chain has \(N=1{,}000\) retained draws for a scalar posterior quantity. For illustration, assume its autocorrelation follows \(\rho_k=0.6^k\). The infinite geometric sum is

\tau=1+2\sum_{k=1}^{\infty}0.6^k =1+2\left(\frac{0.6}{1-0.6}\right) =1+3=4.

Therefore,

N_{\mathrm{eff}}=\frac{1{,}000}{4}=250.

Under this assumed autocorrelation pattern, the chain average has approximately the same asymptotic variance as an independent sample of 250 draws, not 1,000. This result is about the specified scalar quantity and its mean; it does not say that the chain contains only 250 distinct values or that every posterior summary has ESS 250.

Interpreting ESS for Bayesian NLME Summaries

In an NLME posterior, some quantities may be easier to sample than others. Population parameters, residual error parameters, and elements of the random-effects covariance structure can have different marginal shapes and different dependence patterns. Individual random effects may also mix differently across subjects. Review ESS for scientifically important parameters and derived quantities, rather than relying on the best-looking diagnostic.

For a posterior median or central location summary, bulk ESS is a useful indicator of how much effectively independent information supports the estimate. For a lower or upper credible bound, tail ESS is especially relevant because that bound depends on the corresponding part of the posterior. These are diagnostic indicators, not universal pass/fail thresholds. The required ESS depends on the desired precision, the importance of the summary, and the consequences of an inaccurate estimate.

For monotone transformations, such as converting log-clearance draws to clearance by exponentiation, rank-based bulk ESS and quantile-based tail ESS are generally unchanged when there are no ties: the ordering of draws is preserved. But the ESS for estimating a posterior mean of clearance need not match the ESS for the mean of log-clearance, because exponentiation changes the values and their autocorrelation. This is one reason to inspect the quantity actually being reported.

Worked Example: Central Estimates Look Better Than Tail Estimates

Consider hypothetical output from four chains, each with 2,000 retained draws for a population clearance parameter. There are \(4\times2{,}000=8{,}000\) stored draws. The diagnostic reports bulk ESS \(=1{,}600\) and tail ESS \(=120\).

For scale, the bulk estimate corresponds to an effective fraction of \[ \frac{1{,}600}{8{,}000}=0.20, \] while the tail estimate corresponds to \[ \frac{120}{8{,}000}=0.015. \] Thus, the central behavior of clearance is represented much more efficiently than its extreme quantiles. The stored count of 8,000 alone would obscure this difference.

If the analysis reports a posterior median, the bulk ESS is more directly relevant to central exploration. If it reports a 95% credible interval, the low tail ESS warns that the interval endpoints may be less reliably characterized. The appropriate response is not to declare the analysis adequate from the bulk ESS; inspect trace plots and rank-normalized Rhat, and consider whether additional sampling or improved exploration is needed. The ESS values alone do not establish convergence or determine the accuracy of a particular endpoint.

Worked Example: Checking a Tail Diagnostic for a Variability Parameter

Suppose a hypothetical NLME analysis reports posterior draws for a between-subject standard deviation \(\omega\). The estimated ESS for indicators below the 5th percentile is 90, and the estimated ESS for indicators above the 95th percentile is 75. Under the common tail-ESS convention,

\mathrm{ESS}_{\mathrm{tail}}=\min(90,75)=75.

The upper tail is the limiting side. If the scientific report emphasizes whether \(\omega\) could be unusually large, that low upper-tail information is particularly important. A large bulk ESS would not remove this concern: central exploration and upper-tail exploration are different features.

The diagnostic should be interpreted alongside plots and convergence checks. If chains occupy distinct ranges of \(\omega\), the issue may be incomplete exploration rather than simply a need for more draws from the same behavior. If the chains overlap and mix but the tail ESS remains low, additional effective sampling may help. In neither case does the value 75 alone establish whether a specific interval is acceptably precise.

Using ESS With Other Diagnostics

ESS answers an efficiency question conditional on chains sampling the target distribution adequately. Rank-normalized Rhat addresses a different concern by comparing within-chain and between-chain behavior; trace plots show time ordering and possible sticking or drift. These diagnostics complement one another. A reassuring ESS with a concerning Rhat or visibly separated chains should not be taken as evidence of reliable posterior estimation.

ESS also differs from the raw number of draws. Thinning, which keeps only every \(k\)th draw, reduces storage and may reduce dependence among retained values, but it does not create information. Unless storage is a constraint, discarding draws solely to make autocorrelation look smaller is not a remedy for poor mixing. Improving the sampler's exploration or running it longer after adequate warmup is more informative than reporting a thinned count alone.

Common implementation guidance sometimes uses minimum ESS values as rough screening heuristics. Such values are not universal guarantees: a threshold suitable for a broad central summary may be inadequate for a sensitive tail probability, and demanding analyses may require much more information. The decision should be tied to the posterior summaries and precision needed for the scientific question.

Common Pitfalls and Limitations

Equating draws with independent information. The wrong reasoning is that 10,000 stored draws guarantee a large effective sample. The correction is to inspect estimated dependence and report bulk and tail ESS for relevant quantities.
Treating ESS as proof of convergence. ESS is estimated from observed chains; chains that have not explored the target can give misleading diagnostics. Check rank-normalized Rhat and trace plots as well.
Using bulk ESS to justify tail claims. A central summary can be well supported while extreme quantiles are not. Use tail ESS when reporting posterior limits or making decisions driven by extremes.
Assuming one parameter's ESS applies to all summaries. Autocorrelation depends on the quantity being monitored. Check derived quantities that are important to the report.
Thinning to inflate apparent efficiency. Fewer, less-correlated stored draws do not automatically provide more total information. Prefer diagnosing and improving exploration.

Finally, ESS is estimated with uncertainty and is sensitive to finite-chain behavior. It may be unstable when chains are short, highly autocorrelated, heavy-tailed, or poorly mixed. It cannot correct a misspecified NLME model, identify a scientifically inappropriate prior, or show that the posterior itself answers the intended question. It is one part of a diagnostic assessment, not a model-validation result.

Key takeaway: Bulk ESS summarizes effective information for central posterior behavior, while tail ESS focuses on exploration of the extremes. Assess both for important NLME quantities, and interpret them together with chain plots and rank-normalized Rhat.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

A stationary chain has 600 draws and integrated autocorrelation time \(\tau=3\). What is its ESS?
Why can a high bulk ESS coexist with a low tail ESS?
Four chains each contain 1,500 draws. A reported tail ESS is 180. What fraction of the stored draws is this, and what does it suggest?
Why is ESS not sufficient evidence that an NLME posterior has converged?
Would exponentiating every draw necessarily change rank-based bulk ESS? Would it necessarily preserve the ESS for estimating the posterior mean? Explain.

Solutions. (1) \(N_{\mathrm{eff}}=600/3=200\). (2) Central regions may be explored efficiently while visits to one or both extremes are infrequent or strongly dependent, so the two diagnostics measure different features. (3) There are \(4\times1{,}500=6{,}000\) stored draws, and \(180/6{,}000=0.03\), or 3%. This indicates limited effective information for tail behavior and warrants review in context; it does not by itself prove failure. (4) ESS is estimated from sampled chains and assumes adequate sampling of the target; chains can fail to explore or can disagree despite a seemingly adequate ESS. Check Rhat and trace plots. (5) Exponentiation is strictly increasing, so it preserves ranks and generally preserves rank-based bulk ESS when there are no ties. It need not preserve ESS for a posterior mean, because the transformed values can have a different autocorrelation pattern.