Monte Carlo Standard Errors for NLME Estimates

Why Monte Carlo Standard Errors Matter

An MCMC analysis approximates posterior summaries using a finite set of simulated parameter values. Even if the chain is sampling from the intended posterior distribution, a different run would generally produce a slightly different estimated mean or quantile. Monte Carlo standard error (MCSE) measures this simulation uncertainty. It does not measure the posterior uncertainty about a model parameter, and it does not establish that the chain has converged.

In Bayesian NLME models, posterior summaries may concern population parameters, variance components, individual predictions, or derived quantities. MCSE should be assessed for the specific summary being reported. A small MCSE for a parameter mean does not guarantee a small MCSE for a tail quantile of that parameter or for a nonlinear function of several parameters.

MCSE calculations rely on more than the nominal number of saved draws. The draws must represent the intended posterior, and their dependence must be accounted for. In particular, the usual independent-sample formula can understate simulation error when successive draws are correlated. The earlier material on effective sample size introduced the idea that a correlated chain contains less information for a summary than the same number of independent posterior draws.

MCSE for a Posterior Mean

For a scalar quantity \(g(\theta)\), the posterior mean is estimated by averaging its value across the retained draws. If those values were independent, the standard error would be their sample standard deviation divided by \(\sqrt{M}\). For an MCMC chain, the appropriate effective sample size, \(N_{\text{eff},g}\), accounts for dependence in the values of \(g(\theta)\). The usual estimated MCSE is therefore the sample standard deviation of \(g(\theta)\) divided by the square root of that effective sample size.

Here \(s_g\) is the sample standard deviation of the retained \(g(\theta)\) values. The effective sample size should correspond to the quantity being averaged: for example, the ESS for clearance is relevant to its posterior mean, while the ESS for a derived exposure measure should be calculated from that derived measure. A chain-aware variance estimator, such as a spectral-variance or batch-means estimator, can also estimate the standard error directly.

Why the Effective Sample Size Appears

A short justification shows why the independent-draw formula needs adjustment. Write \(Y_m=g(\theta^{(m)})\), let the stationary chain have marginal variance \(\sigma_g^2\), and let \(\rho_k\) be the correlation between values \(k\) draws apart. For a stationary chain with a suitable central limit theorem and a summable correlation sequence, the variance of the sample mean is approximately the long-run variance divided by \(M\):

The term in parentheses is the integrated autocorrelation time, \(\tau_g\). Defining \(N_{\text{eff},g}=M/\tau_g\) makes this variance approximately \(\sigma_g^2/N_{\text{eff},g}\). Replacing the unknown \(\sigma_g\) with \(s_g\) gives the MCSE formula above. If the draws are independent, the correlations at positive lags are zero, \(\tau_g=1\), and \(N_{\text{eff},g}=M\). When dependence is positive, \(N_{\text{eff},g}\) is typically less than \(M\).

This justification requires that the chain be sampling from its target distribution and that a suitable central limit theorem hold for the quantity of interest. A numerical MCSE is not a substitute for checking chain behavior: a chain that has not adequately explored the posterior can give a misleadingly small error estimate.

MCSE for a Posterior Quantile

Quantiles need a different calculation because their precision depends both on how many draws inform the relevant part of the distribution and on how much posterior probability is concentrated near the quantile. Let \(Q_q\) denote the posterior \(q\)-quantile of a scalar quantity \(g(\theta)\), and let \(f(Q_q)\) be its posterior density at that quantile. The relevant indicator is \(I_m=1\{g(\theta^{(m)})\leq Q_q\}\), whose posterior mean is \(q\). Its effective sample size, \(N_{\text{eff},I}\), can differ substantially from the ESS for the parameter values themselves.

Under regularity conditions—including a positive, sufficiently smooth density near \(Q_q\), and a central limit theorem for the indicator sequence—the quantile estimator has the approximate MCSE below. The density is in the inverse units of \(g(\theta)\), so the resulting MCSE has the same units as \(g(\theta)\).

The formula follows by considering the empirical cumulative distribution function at \(Q_q\). Its error is the error in the sample mean of the threshold indicator, with approximate variance \(q(1-q)/N_{\text{eff},I}\). Near the target quantile, a change of \(\Delta\) in the argument changes the cumulative distribution function by approximately \(f(Q_q)\Delta\). Thus the quantile error is approximately the cumulative-distribution error divided by \(f(Q_q)\), giving the displayed variance after squaring.

In practice, \(\hat{f}(\hat{Q}_q)\) must be estimated, and density estimation can be unstable, especially in the tails. Quantile-specific ESS estimates or chain-aware quantile MCSE procedures are preferable to inserting the overall draw count into an independent-draw formula. A low density at a tail quantile can produce a sizable MCSE even when the indicator ESS is substantial.

Judging Whether Simulation Error Is Small Enough

There is no universal acceptable MCSE. Its adequacy depends on the scale of the quantity and on the precision needed for the scientific interpretation. Compare MCSE with a relevant inferential scale: for a posterior mean, this may be the posterior standard deviation or a meaningful decision threshold; for a quantile, it may be the width of the posterior interval or the resolution needed to report that endpoint. State the comparison in context and units.

For example, an MCSE of 0.06 L/h may be negligible if posterior uncertainty is several L/h, but material if the conclusion turns on whether clearance is above a threshold only 0.05 L/h away. Reporting several decimal places does not make a summary precise; simulation error should be considered before choosing the displayed precision.

Diagnostics, Alternatives, and Limitations

Before relying on an MCSE, check that the chains plausibly represent the target posterior. Use the convergence and chain diagnostics introduced earlier, and examine the specific quantity whose summary is being reported. A reasonable overall ESS does not guarantee a good tail quantile estimate. For quantiles, also check whether the estimate is stable across chains or simulation segments and whether the estimated density near the quantile is credible.

• Ignoring dependence: Using \(s_g/\sqrt{M}\) treats correlated draws as independent and can understate error. Use an ESS-based or chain-aware variance estimate.
• Using the wrong ESS: The ESS for the parameter mean need not equal the ESS for a threshold indicator or a transformed quantity. Match the MCSE calculation to the estimand.
• Confusing simulation and posterior uncertainty: A posterior standard deviation describes the spread of the parameter in the posterior; an MCSE describes the precision of a computed summary. They answer different questions.
• Interpreting a small MCSE as proof of convergence: A chain can remain in a limited region and still produce an apparently small estimated MCSE. Assess exploration and convergence separately.

For mean MCSE, an ESS-based formula and a direct batch-means or spectral-variance estimate are alternative ways to account for dependence; their estimates may differ in a finite run. For quantiles, quantile-specific indicator ESS with a density estimate is one approach, while chain-aware quantile procedures can estimate uncertainty without relying on a single plug-in density estimate. Neither approach removes the need to assess chain adequacy. Thinning is not, by itself, a solution to high MCSE: it discards draws and does not generally create more information than retaining the original draws.

Finally, MCSE is an asymptotic approximation. It can be unreliable with short chains, unstable tail behavior, poor exploration, or a density estimate that is highly sensitive to its smoothing choices. For derived quantities, calculate the quantity for each posterior draw and assess its MCSE directly; a transformation can change both the distribution and the simulation precision.