Effective Sample Size and Monte Carlo Error in MICE

Why Iteration Count Is Not Information Count

A MICE trace can contain hundreds or thousands of iterations without providing that many independent pieces of information about a chain-level summary. If a summary changes gradually from one iteration to the next, adjacent values are correlated. The average of such a chain is usually less precise than an average of the same number of independent draws. Effective sample size (ESS) expresses this loss of information on an independent-draw scale; Monte Carlo standard error (MCSE) expresses the resulting precision of the estimated chain average.

For example, a chain-level summary might be the mean of the currently imputed values for a continuous variable, calculated at each iteration. ESS and MCSE then describe the information and numerical precision in the average of that trace summary. They do not establish that the imputation model is appropriate, that the chain has stabilized, or that the analysis has negligible missing-data uncertainty.

Definition: Let $g_t$ be a scalar chain-level summary at iteration $t$, for $t=1,\ldots,N$, after any iterations excluded as burn-in. Write $\rho_k$ for the lag-$k$ autocorrelation of the summary process. Its integrated autocorrelation time is $$ \tau_{\mathrm{int}}=1+2\sum_{k=1}^{\infty}\rho_k. $$ When this sum is well-defined and the summary process is stable, the effective sample size for estimating its mean is $$ N_{\mathrm{eff}}=\frac{N}{\tau_{\mathrm{int}}}. $$ The Monte Carlo standard error of the chain average $\bar{g}=N^{-1}\sum_{t=1}^{N}g_t$ is approximately $$ \mathrm{MCSE}(\bar{g})=\sqrt{\frac{\operatorname{Var}(g_t)\tau_{\mathrm{int}}}{N}} \ \approx\ \frac{s_g}{\sqrt{N_{\mathrm{eff}}}}, $$ where $s_g$ is the sample standard deviation of the retained chain summaries.

Why Autocorrelation Changes the Precision

The relationship between ESS and autocorrelation follows from the variance of a correlated average. Suppose the summary process is weakly stationary: its mean and variance do not change with iteration, and the covariance between two summaries depends only on their lag. Let its variance be $\sigma_g^2$, so the covariance at lag $k$ is $\sigma_g^2\rho_k$. Expanding the variance of the average gives

\begin{aligned} \operatorname{Var}(\bar{g}) &=\frac{1}{N^2}\left[N\sigma_g^2+ 2\sum_{k=1}^{N-1}(N-k)\sigma_g^2\rho_k\right]\\ &=\frac{\sigma_g^2}{N}\left[1+ 2\sum_{k=1}^{N-1}\left(1-\frac{k}{N}\right)\rho_k\right]. \end{aligned}

If the autocorrelations decline sufficiently quickly, then for a long run the bracketed quantity is approximated by $1+2\sum_{k=1}^{\infty}\rho_k=\tau_{\mathrm{int}}$. Thus $\operatorname{Var}(\bar{g})\approx \sigma_g^2\tau_{\mathrm{int}}/N$. An independent sample of size $N_{\mathrm{eff}}$ has mean variance $\sigma_g^2/N_{\mathrm{eff}}$. Equating these expressions yields $N_{\mathrm{eff}}=N/\tau_{\mathrm{int}}$. This is an approximation for the precision of a mean, not a universal measure of information for every possible feature of a trace.

In practice, the infinite autocorrelation sum cannot be computed from a finite trace. An estimator uses the observed autocorrelations and a truncation rule or a spectral-variance method to avoid letting very noisy high-lag estimates dominate. The reported ESS and MCSE therefore depend on the estimator. A software value should be treated as an estimate, not as an exact count.

Worked Example: Positive Autocorrelation Reduces ESS

Suppose a retained MICE chain has $N=500$ values of an imputed-mean summary. For illustration, assume its autocorrelations follow $\rho_k=0.60^k$, as in a stationary AR(1)-type process with lag-one correlation $0.60$. Assume the observed standard deviation of the summary is $s_g=2.00$ mm. The geometric series converges, so

\tau_{\mathrm{int}} =1+2\sum_{k=1}^{\infty}0.60^k =1+2\left(\frac{0.60}{1-0.60}\right) =4.

Consequently,

N_{\mathrm{eff}}=\frac{500}{4}=125, \qquad \mathrm{MCSE}(\bar{g})\approx\frac{2.00\ \mathrm{mm}}{\sqrt{125}} =0.1789\ \mathrm{mm}.

Although the trace contains 500 iterations, its mean has approximately the same Monte Carlo precision as an average of 125 independent values with the same marginal standard deviation. The MCSE describes numerical variation in the estimated chain average, not uncertainty about a patient’s imputed value or about a clinical effect.

Negative Autocorrelation and ESS Above the Iteration Count

ESS is an equivalent independent-sample size, not a literal number of unique draws. When successive values tend to alternate, negative autocorrelation can make an average more precise than an independent average of the same length. In that case $\tau_{\mathrm{int}}<1$, and ESS can exceed $N$. This is mathematically possible, but it does not mean that the chain contains more than $N$ recorded iterations.

Worked Example: A Chain With Negative Autocorrelation

Suppose $N=500$ retained summaries have autocorrelations $\rho_k=(-0.40)^k$, and the summary’s sample standard deviation is $3.00$ mm. This is a stationary AR(1)-type example with a convergent autocorrelation sum. Its integrated autocorrelation time is

\tau_{\mathrm{int}} =1+2\sum_{k=1}^{\infty}(-0.40)^k =1+2\left(\frac{-0.40}{1-(-0.40)}\right) =1-\frac{0.80}{1.40} =\frac{0.60}{1.40} \approx 0.4286.

Therefore the ESS and MCSE are

N_{\mathrm{eff}}=\frac{500}{0.60/1.40} \approx 1{,}167, \qquad \mathrm{MCSE}(\bar{g})\approx\frac{3.00}{\sqrt{1{,}167}} \approx 0.08782\ \mathrm{mm}.

Using the unrounded ESS gives $3.00/\sqrt{1{,}166.6667}\approx0.08783$ mm; the small difference is due only to rounding the ESS first. The estimate exceeds 500 because the negative serial dependence reduces the variance of the average. In real finite traces, estimates of negative autocorrelation can be noisy, so an unexpectedly large ESS should prompt inspection of the trace and the estimation method rather than automatic confidence.

Estimating and Interpreting MCSE

When software reports an ESS for the chain average, a common approximation is $\mathrm{MCSE}\approx s_g/\sqrt{N_{\mathrm{eff}}}$. The units of MCSE are the same as the chain summary. A small MCSE relative to a scientific tolerance indicates that the chain average is numerically precise to that tolerance, conditional on the chain behaving like a stable process. Whether an MCSE is “small” is therefore a contextual decision, not a universal cutoff.

A complementary strategy estimates the long-run variance directly, using methods such as batch means or spectral variance. Batch means divides a trace into consecutive batches, calculates the mean within each batch, and estimates the variance of the overall mean from the variability among batch means. This approach can be useful as a check on an autocorrelation-based ESS estimate, provided the batches are long enough to capture dependence and there are enough batches to estimate their variability.

Worked Example: Checking Precision With Batch Means

Suppose a retained trace has $N=1{,}000$ summaries and is divided into 20 consecutive batches of 50 iterations. The sample standard deviation of the 20 batch means is $0.80$ units. If batches are long enough that dependence between their means is negligible, the estimated standard error of the overall chain average is approximately

\mathrm{MCSE}_{\mathrm{batch}} \approx\frac{0.80}{\sqrt{20}} =0.1789\ \text{units}.

If the standard deviation of individual chain summaries is $2.00$ units, the corresponding ESS implied by this MCSE is

N_{\mathrm{eff}} \approx\left(\frac{s_g}{\mathrm{MCSE}}\right)^2 =\left(\frac{2.00}{0.1789}\right)^2 \approx125.

This agrees with an integrated autocorrelation time near 8, since $1{,}000/8=125$. The agreement is illustrative: batch means do not automatically solve a poor-mixing problem, and a different batch length can give a different estimate when the trace is short or highly persistent.

Assumptions, Diagnostics, and Failure Modes

The interpretation of ESS and MCSE relies on the summary trace being stable over the iterations used. In particular, the mean and variance should not be changing systematically, and dependence should weaken sufficiently for a meaningful long-run variance to exist. ESS calculated from a drifting or abruptly changing trace may look numerically precise while describing no stable chain average. Assess stabilization and mixing before using ESS as a precision diagnostic; a precision estimate cannot repair a chain that has not settled.

For a MICE analysis, compute these diagnostics for a clearly specified scalar summary at each iteration, such as the imputed mean for a named variable. ESS is specific to that summary: the imputed mean may mix well while a tail quantile or a summary for a sparse category mixes poorly. A single ESS should not be applied indiscriminately to all variables or summaries.

Check the trace first. Look for drift, long sticky stretches, abrupt shifts, or a changing spread. Such patterns undermine a stationary-chain precision interpretation.
Inspect autocorrelation across lags. A slowly declining pattern suggests a large integrated autocorrelation time and potentially a small ESS. Noisy high-lag estimates can make the raw infinite-sum estimate unstable.
Compare reasonable estimators. Large changes in ESS or MCSE under different windows, truncations, or batch lengths indicate that the finite trace does not support a dependable precision estimate.
Check Monte Carlo precision against the goal. If the MCSE is too large for a reported chain average, more iterations may help when the process is stable. If it reflects poor mixing, merely extending the run may be inefficient.

A common mistake is to treat $N$, the iteration count, as the effective sample size and report $s_g/\sqrt{N}$ as the MCSE even when the trace is strongly autocorrelated. The correction is to account for the integrated autocorrelation time: use $N_{\mathrm{eff}}=N/\tau_{\mathrm{int}}$, or estimate the long-run variance directly. Another mistake is to read a small MCSE as evidence that the imputation model is valid. MCSE measures simulation error in a chain average; it does not measure bias, model misspecification, or uncertainty from missing data.

What ESS and MCSE Do Not Replace

ESS and MCSE answer a narrow but important question: how precisely has a particular chain average been computed, given the serial dependence in that chain? They are not the standard errors used to quantify sampling uncertainty in a substantive analysis, and they are not the between-imputation and within-imputation uncertainty combined for inference. A small MCSE for a trace summary does not imply a small standard error for a treatment effect estimated after imputation.

They also do not certify convergence. A chain can have a high estimated ESS while targeting an inappropriate distribution, or while failing to explore a relevant region. Conversely, a low ESS signals that the reported average has appreciable Monte Carlo error, but does not by itself identify the cause. Trace inspection, sound imputation models, and appropriate checks across chains remain important. The central comparison is not “iterations versus a universal ESS threshold,” but estimated Monte Carlo error versus the precision needed for the intended diagnostic or calculation.

Key takeaway: For a stable chain-level summary, autocorrelation inflates the variance of its average by approximately $\tau_{\mathrm{int}}$. Thus $N_{\mathrm{eff}}=N/\tau_{\mathrm{int}}$ and $\mathrm{MCSE}\approx s_g/\sqrt{N_{\mathrm{eff}}}$. Both estimates are summary-specific and meaningful only when the trace and estimation method support them.

Check Your Understanding

Answer each question, then compare with the solutions below.

A stationary summary trace has $N=800$ iterations and integrated autocorrelation time $\tau_{\mathrm{int}}=4$. Find its ESS.
For the trace in question 1, $s_g=1.50$ units. Estimate the MCSE of its chain average.
Can a valid ESS estimate exceed the number of iterations? State the condition that can make this happen.
What does a small MCSE establish, and name one important conclusion it does not establish.

Solutions. (1) $N_{\mathrm{eff}}=800/4=200$. (2) $\mathrm{MCSE}\approx1.50/\sqrt{200}=0.1061$ units. (3) Yes. If negative autocorrelation makes $\tau_{\mathrm{int}}<1$, then $N_{\mathrm{eff}}=N/\tau_{\mathrm{int}}>N$. (4) A small MCSE indicates that the chain average has low estimated Monte Carlo error under the assumed stable-process interpretation. It does not establish that the imputation model is appropriate, that the chain has converged, or that substantive estimates have low missing-data uncertainty.