Why Sum Over a Mixture Indicator?
Robust borrowing models often represent prior uncertainty with a mixture: one component may support substantial borrowing from historical data, while another protects against conflict by allowing less borrowing. The component is uncertain, not observed. A posterior analysis must account for that uncertainty rather than choose a component in advance. For a finite set of components, this can be done directly by summing over the possible indicator values.
The sum is more than a computational convenience. It gives the marginal likelihood under the mixture and the posterior probability assigned to each component. It also provides a direct route to posterior means, variances, and probabilities by combining the corresponding within-component results. The calculations require a clearly specified likelihood, normalized component weights, and valid component priors.
The component evidence \(m_k(y)\) averages the likelihood over the prior within component \(k\); it is not the likelihood evaluated at a representative parameter value. If the component prior is proper and the likelihood is a probability density or probability mass function for the observed data, this integral is nonnegative but need not be finite. Accordingly, the posterior-weight and posterior-summary formulas below assume all component evidences are finite and \(0<p(y)=\sum_{k=1}^K w_km_k(y)<\infty\). The mixture marginal likelihood is a weighted sum of these evidences. Under these conditions, the posterior weights \(q_k\) are nonnegative and sum to one.
For components with \(m_k(y)>0\), the conditional posterior density is \(p_k(\theta\mid y)=L(\theta;y)p_k(\theta)/m_k(y)\). Write \(K_y=\{k:m_k(y)>0\}\); components with \(m_k(y)=0\) have \(q_k=0\) and are omitted from the posterior mixture. Therefore the full posterior is itself a mixture:
This identity is the basis for all the summaries below. It says that the posterior distribution is formed by averaging the component-specific posterior distributions using their updated, not prior, weights.
Derivation and Posterior Summaries
The marginal likelihood follows by summing the joint distribution over the unobserved indicator. Given \(Z=k\), integrating out \(\theta\) gives \(p(y\mid Z=k)=m_k(y)\). The law of total probability then gives:
Applying Bayes’ rule to the same joint distribution gives the posterior weight formula. For any integrable function \(g\), conditional expectation then yields a useful general summary:
In particular, if component \(k\) has posterior mean \(\mu_k\) and variance \(V_k\), the full posterior mean is \(\mu=\sum_k q_k\mu_k\). Its variance follows by decomposing squared deviations around each component mean:
The first term is the weighted average of within-component variances; the second is the variance of the component means under the posterior weights. Because the second term is nonnegative, the full mixture variance is at least the smallest component variance among components with positive posterior weight. It can be smaller than some component variances, but not smaller than all of them.
Worked Example: Normal Components for a Treatment Effect
Suppose a hypothetical trial yields an approximately normal estimate \(y=1.00\) with known standard error \(0.50\), so \(y\mid\theta\sim N(\theta,0.25)\). A robust prior has two components: \(Z=1\) with weight \(0.70\) and \(\theta\mid Z=1\sim N(0,0.25)\), and \(Z=2\) with weight \(0.30\) and \(\theta\mid Z=2\sim N(0,4.00)\). Assume the normal likelihood and the stated priors are the complete model.
For a normal likelihood with variance \(s^2\) and a normal component prior \(N(m_k,v_k)\), the marginal distribution of \(y\) in component \(k\) is \(N(m_k,v_k+s^2)\). Here the component evidences are:
Thus the marginal likelihood is \(0.70(0.2076)+0.30(0.1720)=0.19692\approx0.1969\). Using the unrounded evidences, the posterior weight for the narrower component is \(q_1=0.70(0.2075537)/0.1968988\approx0.7379\), and \(q_2=1-q_1\approx0.2621\). The data shift weight toward the narrower component relative to its prior weight of \(0.70\), but both components retain positive posterior probability.
Normal conjugacy gives posterior variance \(V_k=(1/v_k+1/s^2)^{-1}\) and posterior mean \(\mu_k=V_k(m_k/v_k+y/s^2)\). Therefore component 1 has \(\mu_1=0.5000\) and \(V_1=0.1250\); component 2 has \(\mu_2=0.9412\) and \(V_2=0.2353\). The overall posterior mean is:
The variance is not just the weighted average of the two conditional variances. Including the between-component term gives:
The posterior standard deviation is about \(0.4377\) treatment-effect units. The posterior probability that \(\theta>0\) is also a weighted sum: \(0.7379P(\theta>0\mid y,Z=1)+0.2621P(\theta>0\mid y,Z=2)\), approximately \(0.935\). These are posterior summaries under the stated model, not frequentist error rates.
Marginalizing in a Discrete Model
The same calculation applies when the parameter itself is discrete or when component integrations are done numerically. The next hypothetical example uses a finite parameter space so every sum can be checked directly.
Worked Example: Enumerating Three Components
Consider a hypothetical model with \(\theta\in\{0,1,2\}\), three prior components, and observed data whose likelihood values are \(L(0;y)=0.10\), \(L(1;y)=0.40\), and \(L(2;y)=0.80\). The component prior probabilities over \((0,1,2)\) are \(A=(0.80,0.20,0)\), \(B=(0.10,0.60,0.30)\), and \(C=(0.05,0.15,0.80)\). Their mixture weights are \(w_A=0.50\), \(w_B=0.30\), and \(w_C=0.20\). Each component distribution sums to one, and the component weights do too.
The component evidences are the likelihood averaged over each component prior:
The marginal likelihood is \(0.50(0.160)+0.30(0.490)+0.20(0.705)=0.368\). Hence the posterior component weights are \(q_A=0.080/0.368\approx0.2174\), \(q_B=0.147/0.368\approx0.3995\), and \(q_C=0.141/0.368\approx0.3832\). They sum to one, up to rounding.
Within component A, the unnormalized posterior probabilities for \((0,1,2)\) are \((0.080,0.080,0)\), giving posterior probabilities \((0.500,0.500,0)\) and mean \(0.500\). For B the unnormalized values are \((0.010,0.240,0.240)\), giving mean \((0.240+2(0.240))/0.490\approx1.469\). For C they are \((0.005,0.060,0.640)\), giving mean \((0.060+2(0.640))/0.705\approx1.901\). Thus the overall posterior mean is:
As a check, the prior mixture probabilities for \(\theta=(0,1,2)\) are \((0.440,0.310,0.250)\). Multiplying by the likelihood gives \((0.044,0.124,0.200)\), which normalize by \(0.368\) to posterior probabilities approximately \((0.1196,0.3370,0.5435)\). Their mean, \(0(0.1196)+1(0.3370)+2(0.5435)\), is also \(1.424\). Agreement between these two calculations is a useful implementation check.
Numerical Practice and Failure Modes
For a finite mixture, a reliable calculation proceeds in a fixed order: compute each component evidence, multiply by its prior weight, sum those products, and normalize the products to obtain posterior weights. Then compute conditional posterior summaries and combine them using the posterior weights. Keeping these stages separate makes it easier to find errors.
In larger models, the evidence may be available from analytic integration, numerical quadrature, or a finite grid. The sum over components is still the same, but numerical precision matters. If component evidences are extremely small, direct evaluation may underflow to zero. Let \(a_k=\log(w_k)+\log(m_k)\). The log marginal likelihood can be calculated stably as:
Posterior weights are then \(\exp(a_k-\log p(y))\). This transformation changes the arithmetic scale, not the probability model. It is especially important when one component has evidence far smaller than the others.
For several independent indicators, direct enumeration can become expensive: \(J\) binary indicators have \(2^J\) joint configurations. If the model factorizes, some sums may be simplified; otherwise, summing all configurations may be impractical. Numerical marginalization is exact up to numerical integration and rounding when all configurations are included, while approximating the sum by retaining only selected components changes the calculation and requires justification.
Marginalization and Explicit Indicator Sampling
Marginalization removes \(Z\) from the calculation by summing over its possible values. Explicit indicator sampling instead retains \(Z\) as part of the unknown state and represents uncertainty about it through draws or other numerical sampling. Both approaches target the same joint model when correctly implemented; they differ in how computation represents the discrete uncertainty.
| Approach | What is computed | Main advantage | Main limitation |
|---|---|---|---|
| Numerical marginalization | Sum component evidences and combine conditional summaries | Direct component probabilities; avoids sampling noise in the finite sum | Can be costly when the number of configurations is large or conditional integrations are difficult |
| Explicit indicator sampling | Retain \(Z\) and generate draws that include its uncertainty | Can handle configurations that are too numerous to enumerate | Estimates of component probabilities and summaries have simulation error and depend on adequate exploration |
When there are only a few components and component calculations are tractable, marginalization is often a clear benchmark and may be more efficient for estimating posterior summaries. Explicit indicator sampling becomes attractive when enumeration is infeasible or when the model structure makes joint sampling more convenient. The later material on MCMC addresses how sampling algorithms are constructed and assessed; here the key distinction is exact summation versus representing the indicator explicitly in a numerical approximation.
Marginalization does not repair a poorly chosen mixture. Results still depend on the component distributions, their prior weights, the likelihood, and any assumptions used to obtain the component evidences. Diagnostics should include verifying that prior weights and component distributions are normalized, checking that posterior weights sum to one, and independently checking summaries when possible. Large sensitivity to component choices is a modeling issue, not a summation error.
List all components, their prior weights, their parameter distributions, and the likelihood for the observed data.
For each component, integrate the likelihood against that component’s prior, analytically or numerically.
Multiply each evidence by its prior weight, sum the products, and divide each product by that sum.
Average conditional posterior means and probabilities using the updated weights; include between-component variation when computing variance.
Check Your Understanding
Answer each question, then compare with the short solutions below.
- A two-component model has prior weights \(0.60\) and \(0.40\), and component evidences \(0.20\) and \(0.50\). Find the marginal likelihood and posterior component weights.
- Why is a component’s evidence not generally the likelihood evaluated at its prior mean?
- Two components have positive posterior weights, posterior means \(0\) and \(2\), and conditional variances \(1\) and \(0.5\). Explain why the mixture variance must exceed the weighted average of the conditional variances.
- State one reason to prefer marginalization and one reason explicit indicator sampling may be preferable.
- What numerical symptom can indicate underflow in a direct calculation of component evidences, and what calculation helps prevent it?
Solutions. (1) The marginal likelihood is \(0.60(0.20)+0.40(0.50)=0.32\). The posterior weights are \(0.12/0.32=0.375\) and \(0.20/0.32=0.625\). (2) Evidence averages the likelihood over all parameter values under the component prior; in general, an average of likelihood values is not the likelihood at the prior mean. (3) The variance is the weighted within-component variance plus the weighted variance of component means. Since both weights are positive and the means differ, the between-component term is positive. (4) Marginalization avoids simulation error in a manageable finite sum; explicit sampling can be more feasible when there are too many configurations to enumerate. (5) Very small evidences may be rounded to zero, producing invalid normalization or loss of components. Computing weighted evidences and their sum on the log scale with the log-sum-exp identity helps prevent underflow.