MCMC for Robust Mixture Priors

Why Sample the Mixture Indicator?

A robust prior can represent uncertainty about how much historical information is relevant by mixing a concentrated component with a more diffuse protective component. The parameter of interest, \(\theta\), is shared, but a latent indicator records which component generated its prior draw. MCMC can sample that indicator alongside \(\theta\), making it possible to estimate posterior component probabilities and to summarize uncertainty about the mixture allocation.

This tutorial focuses on that computation. In particular, an indicator update conditional on the current value of \(\theta\) is not the same quantity as the posterior probability of a component after integrating over \(\theta\). The distinction is central to implementing and interpreting the sampler.

Model and Gibbs Updates

Assume the observed data have likelihood \(L(\theta;y)\), and that the mixture weights and component densities are fixed and known. The joint posterior density of \((\theta,Z)\), up to a factor that does not depend on either, is

A Gibbs sampler alternates draws from the full conditional distributions of the unknowns, each conditional on the current values of the others. This method requires that the specified likelihood and prior define a proper posterior and that the conditional distributions can be sampled. If a component density or mixture weight also has unknown parameters, those parameters need their own valid updates; the two updates below alone are then not a complete sampler.

The parameter update, conditional on a component \(k\), has density

Together these give a simple Gibbs scheme when both conditional distributions are tractable or can be sampled by an appropriate method. A common conjugate example uses a normal likelihood for a summary and normal mixture components for the prior.

Why the Conditional Update Has This Form

The indicator update follows directly by conditioning the joint posterior on \(\theta\). For each component, the joint density is proportional to \(L(\theta;y)w_k g_k(\theta)\). To obtain the conditional probability across components, divide that expression by its sum over all possible component labels. The common likelihood factor cancels.

The cancellation does not mean that the data are irrelevant to the posterior component probability. The data affect the distribution of sampled \(\theta\), and those sampled values in turn affect the indicator draws. But for one indicator update at a fixed \(\theta\), the likelihood contributes no additional factor. Sampling \(Z\) using likelihood-weighted terms at that stage would count the data twice.

For comparison, the marginal posterior component probability integrates over \(\theta\). Define the component marginal likelihood \(m_k(y)=\int L(\theta;y)g_k(\theta)\,d\theta\). Then

This is the quantity estimated by the long-run proportion of MCMC draws assigned to component \(k\). Numerical marginalization, covered in the earlier material on numerical marginalization of mixture indicators, computes the same probability by integrating rather than by sampling the indicator and parameter jointly.

A Conjugate Normal Illustration

Suppose the current-data summary is \(\bar{x}\), with known sampling variance \(v\), so \(\bar{x}\mid\theta\sim N(\theta,v)\). Let the prior have two normal components, \(\theta\mid Z=k\sim N(\mu_k,\tau_k^2)\), with weights \(w_k\). The normal likelihood and normal component prior give a normal conditional distribution for \(\theta\):

The first argument is the conditional mean and the second is the conditional variance. This update combines the data summary and the selected prior component by their precisions. The indicator is then sampled using the component densities evaluated at the new \(\theta\), as in the derivation above.

Estimating Component Probabilities From MCMC Output

After discarding iterations used for initialization, retain \(M\) draws \((\theta^{(m)},Z^{(m)})\), for \(m=1,\ldots,M\). For a two-component model with \(Z=1\) denoting the concentrated component, the posterior component probability is estimated by the fraction of retained draws assigned to that component:

Here \(I(\cdot)\) equals 1 when its statement is true and 0 otherwise. For multiple components, replace the indicator by \(I(Z^{(m)}=k)\) to estimate the probability for component \(k\). The estimates across all components sum to 1 when they are calculated from the same retained draws.

An MCMC estimate has Monte Carlo uncertainty because the draws are dependent. An approximate Monte Carlo standard error for the estimated probability is \(\sqrt{\hat{q}(1-\hat{q})/N_{\mathrm{eff}}}\), where \(\hat{q}\) is the estimated probability and \(N_{\mathrm{eff}}\) is the effective sample size for the indicator sequence. The raw number of iterations is not generally a substitute for effective sample size. This Monte Carlo error describes simulation precision, not uncertainty about the component probability itself.

Implementation, Diagnostics, and Alternatives

A practical iteration for a two-component mixture proceeds as follows: draw \(\theta\) from its conditional distribution given the current indicator and data; calculate the normalized component probabilities at that \(\theta\); then draw the new indicator from those probabilities. For numerical stability with many components or sharply concentrated densities, calculate the log weights \(\log(w_k)+\log g_k(\theta)\), subtract their maximum, exponentiate, and normalize.

Check the implementation by verifying that the normalized indicator probabilities are nonnegative and sum to 1 at every update. Also check that the component-specific parameter updates use the correct component parameters and that the indicator coding matches the intended component labels. Inspect the sampled indicator sequence as well as the parameter draws: a probability estimate based on a small number of effective indicator changes can be imprecise even when many iterations were saved.

The principal alternative is to integrate out the indicator and use the mixture prior density \(p(\theta)=\sum_k w_k g_k(\theta)\) directly in the parameter update. This removes the discrete variable from the simulated state, while sampling indicators makes component allocation summaries immediately available. Which approach is convenient depends on the available conditional distributions and the inferential quantities needed. The marginal component probability can also be computed through the component marginal likelihoods when those integrals are tractable; that avoids estimating this particular probability from indicator draws.

A further limitation is that a posterior component probability is conditional on the specified mixture weights, component distributions, likelihood, and data model. It is not a data-only measure of compatibility and does not prove that a component is the true source of the parameter. Sensitivity to these modeling choices remains important. If the indicator sequence barely changes, or if its effective sample size is low, report the resulting component-probability estimate cautiously and investigate the computation rather than treating the raw iteration count as reassurance.