Reporting Posterior Computation Diagnostics

Why Posterior Computation Diagnostics Belong in a Trial Report

A reported posterior mean, quantile, or probability may be calculated from a numerical approximation rather than directly from a closed-form distribution. In Bayesian clinical trials, Markov chain Monte Carlo (MCMC) is one common way to obtain that approximation. The resulting draws are used to summarize treatment effects, probabilities of benefit, and other quantities of clinical interest. Diagnostics help readers assess whether the computation appears to have explored the relevant posterior distribution adequately and whether simulation error could affect the reported numbers.

Computation diagnostics answer a narrower question than clinical interpretation: how reliably did the numerical procedure approximate the specified posterior distribution? They do not establish that the model is appropriate, the prior is suitable, the data are accurate, or the trial conclusion is clinically justified. A computation can be numerically stable while the model is still a poor representation of the trial.

What to Report for MCMC

An MCMC output consists of dependent draws generated sequentially. Dependence means that the number of retained iterations is not generally the number of independent pieces of information in the simulation. A useful report therefore describes both the sampling procedure and evidence about its performance.

At minimum, state the sampler used, the number of chains, the number of warm-up iterations and retained iterations per chain, and any relevant tuning or adaptation details. Report the convergence diagnostics and effective sample sizes for the quantities that support the trial’s main claims. For a model fitted with Hamiltonian Monte Carlo, also report relevant sampler-specific warnings, such as divergent transitions, if present. State the software and version when needed for reproducibility.

Convergence checks should look for evidence that chains have reached and explored the same target distribution. A trace plot displays sampled values against iteration. After warm-up, a well-behaved trace for a parameter often looks like a stable, overlapping band without persistent drift, long periods stuck in one region, or chains occupying clearly different regions. A trace plot is a visual diagnostic, not a proof.

A commonly reported numerical diagnostic is split-\(\hat{R}\). It compares variation within chains with variation between chains, after splitting chains into segments. If chains have mixed adequately, these sources of variation should be similar and \(\hat{R}\) should be close to 1. Values appreciably above 1 suggest that chains may not have converged or mixed well. In practice, analysts often seek values below about 1.01, while also checking trace plots and other diagnostics. A value near 1 is not sufficient evidence on its own: chains can agree while all fail to explore an important region.

Report both bulk and tail ESS when available. Bulk ESS reflects sampling efficiency for central features of the distribution and is relevant to summaries such as the posterior mean. Tail ESS assesses sampling in the distribution’s tails and is important for quantiles and tail probabilities. A high bulk ESS does not guarantee precise estimation of an extreme quantile or a small tail probability.

Monte Carlo Standard Error

The Monte Carlo standard error (MCSE) estimates the numerical simulation error in a computed posterior summary. For a posterior mean estimated from \(M\) draws with effective sample size \(ESS\), a common approximation is the estimated posterior standard deviation divided by the square root of the ESS. For a posterior probability, use the sampled indicator that the event occurred.

Here \(\bar{\theta}\) is the simulated posterior mean, \(s_{\theta}\) is the sample standard deviation of the retained draws for \(\theta\), and \(\hat{q}\) is the fraction of draws satisfying an event, such as a treatment effect exceeding a clinical threshold. These are approximate calculations; use the ESS appropriate to the quantity being estimated. MCSE describes simulation precision, not the spread of the posterior distribution and not a frequentist standard error for the clinical effect.

Assessing a Reported Posterior Probability

Posterior probabilities are often central to a trial conclusion, so their simulation precision deserves direct attention. For example, a report might give the posterior probability that treatment reduces symptom duration by at least a prespecified clinically meaningful amount. The relevant ESS is the ESS for that event indicator, not automatically the bulk ESS for a model parameter.

When Diagnostics Signal a Problem

Poor diagnostics should change how confidently numerical summaries are reported. Warning signs include chains with visibly different locations or patterns, persistent trends after warm-up, a split-\(\hat{R}\) appreciably above 1, low ESS, large MCSE relative to the precision of reported results, and sampler warnings. For Hamiltonian Monte Carlo, divergent transitions can indicate that the algorithm is having difficulty exploring parts of a posterior with challenging geometry. They should be investigated rather than hidden by reporting only a long run.

The response depends on the warning. Analysts may inspect the model and parameterization, verify that data and code are correct, consider a suitable reparameterization, or use a sampler better suited to the posterior. Extending a run can increase ESS when the existing chains are exploring the target distribution adequately. It is not a general remedy for chains stuck in different regions, persistent divergences, or a model that creates difficult or weakly identified parameters.

Choosing and Reporting Diagnostics in Context

Diagnostics should be matched to the quantities that drive the trial’s interpretation. If the primary conclusion depends on a posterior mean, report its ESS and MCSE. If it depends on a credible interval, examine tail ESS and the numerical stability of its endpoints. If it depends on a posterior probability crossing a decision threshold, report an event-specific ESS and MCSE. In each case, assess trace plots and chain agreement for the relevant parameters and derived quantities.

MCMC is not the only way to compute posterior summaries. Some models permit exact calculations or deterministic numerical integration; others use approximate methods such as variational inference. Exact or deterministic calculations do not have MCMC chain diagnostics, although their own numerical accuracy and approximation assumptions still matter. MCMC provides simulation-based diagnostics but can be computationally demanding. The appropriate method depends on the model and the precision required; a method’s label alone does not guarantee accuracy.

The numerical precision required is also a reporting decision. If an MCSE is large enough to change a rounded probability, quantile, or decision threshold comparison, additional computation or a more careful qualification may be necessary. Conversely, increasing simulation far beyond what can affect the reported precision may add little practical value. Do not imply that more iterations resolve uncertainty about the model, prior, or trial design.

Common Mistakes and Limitations

• Equating retained draws with independent information. MCMC draws are generally correlated, so ESS can be much smaller than the raw number of iterations.
• Treating \(\hat{R}\) near 1 as proof of convergence. It is a useful diagnostic, but it cannot rule out all failures to explore the posterior.
• Using one ESS for every result. ESS depends on the quantity being estimated; means, tail quantiles, and event probabilities can have different effective sample sizes.
• Calling MCSE posterior uncertainty. MCSE describes numerical error from finite simulation. Posterior spread describes uncertainty represented by the fitted posterior.
• Responding to every problem by running longer. More iterations may help when exploration is sound but may not fix poor parameterization, chain separation, or sampler pathologies.
• Reporting only reassuring diagnostics. A transparent report includes material warnings and explains their implications, rather than selecting only favorable metrics.

Diagnostics are evidence about computation, not a certificate of validity. They cannot confirm that the prior, likelihood, endpoint definition, missing-data assumptions, or clinical interpretation is appropriate. Those considerations remain part of a complete trial analysis and report.