Effective Sample Size and Monte Carlo Error in QSP Models

Why Posterior Draw Count Is Not the Same as Information

A QSP calibration may produce thousands of posterior draws, but consecutive draws from an MCMC algorithm are often dependent. If a parameter moves slowly through its posterior distribution, many stored values may repeat nearly the same information. The practical question is therefore not only how many draws were saved, but how precisely those draws estimate the posterior quantities of interest.

Effective sample size (ESS) expresses the information in correlated draws as the size of an equivalent independent sample. Monte Carlo standard error (MCSE) then quantifies the numerical uncertainty caused by using a finite simulation to estimate a posterior mean, quantile, or other summary. These are measures of simulation precision, not measures of posterior uncertainty: a parameter can have a wide posterior distribution and still have its mean estimated precisely, or a narrow posterior and an imprecisely estimated tail quantile.

The calculations below assume the chains have reached the target posterior distribution and are sampling it adequately. Convergence assessment, including the earlier material on rank-normalized R-hat, is a prerequisite: a small MCSE computed from chains that have not converged does not establish that the posterior estimate is trustworthy.

Autocorrelation and Effective Sample Size

Definition: Let $h(\theta_t)$ be a scalar quantity of interest evaluated at MCMC iteration $t$, such as a model parameter or a QSP output. If its stationary lag-$k$ autocorrelation is $\rho_k$, its integrated autocorrelation time is $$ \tau_h = 1 + 2\sum_{k=1}^{\infty}\rho_k. $$ For $N$ retained draws, the effective sample size for $h$ is approximately $$ N_{\mathrm{eff},h} = \frac{N}{\tau_h}. $$

The sum includes autocorrelations at all positive lags. Positive autocorrelation increases $\tau_h$, reducing ESS below the number of retained draws. Negative autocorrelation can reduce $\tau_h$ and, in some cases, yield ESS greater than $N$: negatively dependent draws can contain more information about a mean than the same number of independent draws. In practice, autocorrelation estimates are noisy at large lags, so reliable software uses stabilized estimators rather than summing an arbitrary long sequence of raw estimates.

ESS is specific to the quantity being estimated. A chain may move smoothly for one parameter but slowly for another, and a nonlinear function $h(\theta)$ can have a different autocorrelation pattern from any single parameter. Thus, “the ESS of the chain” is generally incomplete; report or inspect ESS for the relevant parameters and derived quantities.

From ESS to Monte Carlo Error for a Mean

For a stationary chain with finite variance, the variance of the sample average is approximately the marginal variance divided by the effective sample size. If $s_h$ is the estimated posterior standard deviation of $h$, the corresponding estimated MCSE of its posterior mean is

\operatorname{MCSE}(\bar{h}) \approx \frac{s_h}{\sqrt{N_{\mathrm{eff},h}}}.

Here $\bar{h}$ is the average of the retained values of $h$. This approximation describes repeated-run variation in the Monte Carlo estimate, not a confidence or credible interval for the unknown parameter. A rough interval for simulation error alone is $\bar{h}\pm 1.96\,\operatorname{MCSE}(\bar{h})$, when the estimator is approximately normal. It should not be confused with a posterior interval, which describes uncertainty about the parameter under the posterior distribution.

Justification: For $N$ stationary draws with marginal variance $\sigma_h^2$, the variance of their average is $$ \operatorname{Var}(\bar{h}) = \frac{\sigma_h^2}{N} \left(1+2\sum_{k=1}^{N-1}\left(1-\frac{k}{N}\right)\rho_k\right). $$ When $N$ is large and the autocorrelation sum is adequately behaved, the parenthesized term is approximately $\tau_h$. Therefore $\operatorname{Var}(\bar{h})\approx \sigma_h^2\tau_h/N=\sigma_h^2/N_{\mathrm{eff},h}$. Replacing $\sigma_h$ and $\tau_h$ with estimates gives the MCSE formula.

Worked Example: Autocorrelation Reduces the Information in a QSP Parameter Chain

Suppose a single retained chain contains $N=4{,}000$ draws of a hypothetical QSP degradation-rate parameter. Its estimated autocorrelation follows $\rho_k=0.60^k$ for positive lags, and the posterior standard deviation of the parameter is $s_h=0.80$ per day. Treat this geometric pattern as a simplifying approximation and assume the chain is stationary.

For a geometric autocorrelation sequence, $$ \tau_h=1+2\sum_{k=1}^{\infty}0.60^k =1+2\left(\frac{0.60}{1-0.60}\right)=4. $$

Thus the effective sample size is $$ N_{\mathrm{eff},h}=\frac{4{,}000}{4}=1{,}000. $$ The estimated MCSE of the posterior mean is $$ \operatorname{MCSE}(\bar{h})\approx\frac{0.80\ \text{per day}}{\sqrt{1{,}000}} =\frac{0.80}{31.62}\ \text{per day} =0.0253\ \text{per day}. $$

Although 4,000 values were retained, they provide roughly the mean-estimation information of 1,000 independent draws under this approximation. The posterior mean’s simulation error is estimated as about 0.0253 per day; this does not describe the posterior spread, which is 0.80 per day.

Monte Carlo Error for Posterior Quantiles

Quantiles need a different precision calculation from means. Let $q_p$ be the posterior $p$-quantile of a scalar quantity $h$, and let $f(q_p)$ be its posterior density at that quantile. If the density is positive and reasonably smooth there, the empirical quantile has an approximate standard error related to the variability of the empirical cumulative distribution at $q_p$.

The relevant series is the indicator $I_t=1\{h(\theta_t)\leq q_p\}$. Its posterior mean is $p$, and its marginal variance is $p(1-p)$. Define $N_{\mathrm{eff},I}$ as the ESS for this indicator series. A standard large-sample approximation is

\operatorname{MCSE}(\hat{q}_p) \approx \frac{\sqrt{p(1-p)/N_{\mathrm{eff},I}}}{f(q_p)}.

This formula shows why tail quantiles can require many draws: $p(1-p)$ reflects the limited information in a tail indicator, and a low density near the quantile means that even a small change in cumulative probability can correspond to a substantial change in the quantile value. The density has units inverse to those of $h$, so the resulting MCSE has the same units as $h$.

Important distinction: Quantile MCSE depends on the autocorrelation of the indicator $1\{h\leq q_p\}$, not necessarily on the ESS for $h$ itself. Use a quantile-specific ESS or an appropriate quantile-MCSE estimator when available. Substituting a parameter’s bulk ESS into the quantile formula is only an approximation and may be misleading, especially in the tails.

Worked Example: Checking the Precision of a Posterior 95th Percentile

In a hypothetical QSP biomarker model, the estimated posterior 95th percentile of a steady-state concentration is $q_{0.95}=8.4$ mg/L. Suppose the estimated density there is $f(q_{0.95})=0.080$ per (mg/L), and the effective sample size for the corresponding indicator series is 750. Assume the posterior density is smooth and positive near this quantile.

The estimated MCSE is $$ \operatorname{MCSE}(\hat{q}_{0.95}) \approx \frac{\sqrt{0.95(1-0.95)/750}}{0.080\ \text{per (mg/L)}}. $$

First, $0.95(0.05)/750=0.00006333$, so the numerator is $\sqrt{0.00006333}=0.007958$. Dividing by the density gives $$ \operatorname{MCSE}(\hat{q}_{0.95}) \approx\frac{0.007958}{0.080\ \text{per (mg/L)}}=0.0995\ \text{mg/L}. $$

An approximate 95% range for Monte Carlo error alone is $8.4\pm1.96(0.0995)$ mg/L, or approximately 8.21 to 8.59 mg/L. This is not a 95% posterior interval for the concentration. It indicates the approximate repeated-simulation variability of the estimated posterior 95th percentile, under the stated assumptions.

Assessing Whether Precision Is Adequate

There is no universal ESS threshold that guarantees adequate precision. The relevant standard is whether the MCSE is small enough for the intended interpretation or decision. Before increasing run length, specify a tolerable numerical error in the units of the reported quantity. For a mean, compare its MCSE with that tolerance or with a scientifically meaningful scale. For a quantile, use its quantile-specific MCSE and judge whether the uncertainty could alter the conclusion.

Worked Example: Deciding Whether a Posterior Mean Needs More Draws

Suppose four chains produce a combined 18,000 retained draws for a hypothetical QSP elimination half-life. The posterior standard deviation is $s_h=6.0$ days and the ESS for the mean is 900. The estimated posterior mean is 18.2 days. The analyst requires the MCSE to be no greater than 0.10 days for the planned report.

The current mean MCSE is $$ \operatorname{MCSE}(\bar{h})\approx\frac{6.0\ \text{days}}{\sqrt{900}} =\frac{6.0}{30}\ \text{days}=0.20\ \text{days}. $$

This exceeds the stated 0.10-day tolerance, so the current simulation is not precise enough for that reporting target. Since MCSE is approximately proportional to $1/\sqrt{N_{\mathrm{eff}}}$, the required ESS is $$ N_{\mathrm{eff,required}}\approx\left(\frac{6.0\ \text{days}}{0.10\ \text{days}}\right)^2=3{,}600. $$

The current ESS is 900, one quarter of 3,600. If the sampling efficiency remains similar and the chains continue to explore the same posterior, approximately four times as many total iterations may be needed. This is a planning approximation, not a guarantee: the added draws must actually provide adequate exploration, and MCSE and convergence diagnostics should be reassessed afterward.

Estimation Choices, Diagnostics, and Limitations

Modern MCMC software often estimates ESS and MCSE using multiple chains and stabilized estimates of autocorrelation. For means, a spectral-variance estimator or a related method estimates the long-run variance directly; bulk ESS is also commonly reported as a general measure of sampling efficiency. Tail ESS focuses on tail behavior. For a particular quantile, quantile-specific ESS and MCSE are more directly relevant than a single bulk ESS. These approaches use the same underlying idea—dependence reduces information—but differ in the target quantity and variance estimator.

Batch means offer another approach: divide a sufficiently long chain into batches, calculate the average within each batch, and estimate the variance of the overall mean from variation among batch averages. This can provide a useful check on a spectral or ESS-based MCSE when the chain is long enough and batches are chosen sensibly. Agreement between methods is reassuring, but neither method repairs nonconvergence or poor exploration.

Thinning retains only every $m$th draw. It can reduce storage or post-processing burden, but it usually discards information and does not make a poorly mixing chain converge. If the aim is to improve precision, continuing to sample and retaining the draws is generally more useful than thinning them merely to obtain apparently less correlated values.

Inspect more than a single reported number. Warning signs include very low ESS relative to the draw count, large MCSE compared with the intended precision, unstable ESS or MCSE estimates when the run is extended, and chains with visibly different behavior. A smooth trace plot alone is not evidence of small MCSE. Estimates can also fail when chains are too short to estimate autocorrelation reliably, when the posterior has heavy tails or irregular geometry, or when the target quantity has a density close to zero near the quantile of interest.

ESS and MCSE are conditional on sampling the intended posterior. They do not measure uncertainty caused by the QSP model structure, observation model, prior specification, or missing biological mechanisms. Nor do they establish that a posterior summary is scientifically appropriate. They answer a narrower question: how much numerical uncertainty remains because the posterior was approximated by a finite, dependent simulation?

Common Pitfall

Wrong reasoning: “I saved 20,000 draws, so the posterior mean and 95th percentile must be precise.”

Correction: The stored count alone does not determine precision. Strong autocorrelation can make the effective sample size much smaller, and a tail quantile can have a different, often lower, effective sample size than the parameter’s bulk behavior suggests. Check the MCSE for each reported quantity against a stated precision target, after assessing convergence.

A related error is treating an MCSE interval as a posterior credible interval. The former describes simulation error in an estimated summary; the latter describes posterior uncertainty in the modeled quantity. They answer different questions and should be reported distinctly.

Check Your Understanding

Answer each question, then compare with the solutions below.

A chain has 2,400 retained draws and integrated autocorrelation time $\tau_h=3$. What is the approximate ESS?
A posterior quantity has standard deviation 2.5 mg/L and ESS 625. Calculate the MCSE of its posterior mean.
Why might the ESS for a 95th percentile differ from the ESS for the posterior mean of the same parameter?
If a mean’s MCSE is 0.30 units and the required tolerance is 0.10 units, approximately how many times the current ESS is needed, assuming unchanged sampling efficiency?
State one reason thinning is not generally a way to increase the information available for estimating a posterior mean.

Solutions. (1) $N_{\mathrm{eff}}=2{,}400/3=800$. (2) $\operatorname{MCSE}=2.5/\sqrt{625}=2.5/25=0.100$ mg/L. (3) The quantile is informed by the indicator $1\{h\leq q_{0.95}\}$, whose autocorrelation can differ from that of the parameter values used for the mean; tail behavior can therefore have its own ESS. (4) Since MCSE scales approximately as $1/\sqrt{N_{\mathrm{eff}}}$, reducing it by a factor of three requires about $3^2=9$ times the current ESS. (5) Thinning discards draws; unless storage or computation constraints make it necessary, retaining additional draws generally preserves more information than keeping only every $m$th draw.