Diagnosing Heterogeneity With Posterior Distributions

What Posterior Heterogeneity Does—and Does Not—Tell You

A hierarchical basket model allows subgroup treatment effects to vary while sharing information across subgroups. Interpreting its fitted posterior requires looking at more than whether subgroup point estimates differ. The relevant questions are how uncertain each subgroup effect remains, how much between-subgroup variation is supported, and whether that variation is large enough to matter clinically. A visible difference between posterior means may be compatible with substantial uncertainty; conversely, modest differences can be informative when posterior distributions are sufficiently concentrated.

This tutorial focuses on interpreting posterior distributions, not on checking whether a fitted model reproduces the observed data. Model assumptions still matter: a posterior distribution describes uncertainty conditional on the specified likelihood, hierarchical structure, prior distributions, and data. It cannot by itself establish that subgroups are exchangeable or that the model is clinically appropriate.

Effects and Between-Subgroup Variation

Let \(\theta_j\) denote the treatment effect in subgroup \(j\), for \(j=1,\ldots,J\). For illustration, suppose effects are measured as treatment-minus-control differences in response probability, expressed in percentage points; a positive value then favors treatment. The same interpretation applies on another effect scale, such as a log odds ratio, provided the units and direction are stated.

A common hierarchical model represents the subgroup effects as draws around a shared mean effect. Its population-level parameters are \(\mu_\theta\), the mean of the subgroup-effect distribution, and \(\tau_\theta\), its standard deviation. The subgroup effects are conditionally independent given those parameters in this model. An observation model connects each subgroup’s patient outcomes to its effect; priors are assigned to the unknown parameters.

This distributional assumption is an exchangeability model: before seeing subgroup-specific outcomes, the subgroups are treated as draws from a common effect distribution, conditional on \(\mu_\theta\) and \(\tau_\theta\). It does not say that their effects are identical. The parameter \(\tau_\theta\) is measured in the same units as \(\theta_j\); larger values indicate greater dispersion of effects across the modeled population of subgroups. The model and its interpretation require clinically coherent subgroups, an adequate outcome likelihood, and a defensible exchangeability assumption.

The posterior distribution of \(\theta_j\) combines the subgroup’s data with information from the hierarchy. Its center and spread describe the effect and remaining uncertainty for that subgroup, conditional on the model. The posterior for \(\tau_\theta\) instead describes uncertainty about the extent of between-subgroup variation. These are related but distinct quantities: a broad posterior for a subgroup effect can result from sparse data even when the posterior for \(\tau_\theta\) is concentrated near small values.

Reading Subgroup-Effect Posteriors in Context

For a subgroup effect, examine its posterior location, spread, and probability of exceeding a clinically relevant threshold. A credible interval summarizes posterior uncertainty on the effect scale. For example, a 90% credible interval from 2 to 14 percentage points means that the posterior probability that the effect lies between 2 and 14 percentage points is 0.90, conditional on the model and observed data. It does not mean that a fixed unknown effect has a 90% probability of lying in this particular interval under a frequentist interpretation.

Clinical thresholds should be defined in advance or justified in the clinical context. If a 5-percentage-point increase is considered meaningful, a posterior probability such as \(P(\theta_j>5\mid\text{data})\) directly describes support for exceeding that threshold. The probability is not the probability that a treatment works in an absolute sense; its meaning depends on the effect definition, direction, scale, model, and threshold.

Relating Variation to Meaningful Differences

The posterior for \(\tau_\theta\) is often the most direct summary of overall heterogeneity. Report its scale and uncertainty, not just a point estimate. A posterior median of 6 percentage points means the posterior median of the modeled between-subgroup standard deviation is 6 percentage points; it does not mean every pair of subgroups differs by 6 points. Because \(\tau_\theta\) is nonnegative and may be weakly identified with few subgroups, its posterior can be skewed or concentrated near zero.

A useful model-based interpretation follows from the normal random-effects assumption. Conditional on fixed \(\mu_\theta\) and \(\tau_\theta\), two independently drawn subgroup effects have difference \(D=\theta_j-\theta_k\). Since the conditional means cancel and the two variances add, the difference has mean zero and variance \(2\tau_\theta^2\). Thus \(D\) has a normal distribution with standard deviation \(\sqrt{2}\tau_\theta\). This describes the model-implied difference for two subgroup effects conditional on the population parameters; it is not the posterior distribution of a particular pair’s contrast after incorporating their observed data.

The probability is two-sided because either subgroup could have the larger effect. It can help connect a standard deviation to a clinically interpretable difference \(c\). Its assumptions are the conditional normal distribution and independence of the two subgroup effects given \(\mu_\theta,\tau_\theta\). It should not be applied to a known pair’s data-conditioned contrast in place of examining that contrast’s posterior draws.

In an actual analysis, \(\tau_\theta\) is uncertain. A posterior probability that accounts for this uncertainty can be approximated by evaluating the conditional probability at each posterior draw of \(\tau_\theta\) and averaging. For example, suppose three illustrative posterior draws are 4, 6, and 8 percentage points, given equal weight. For \(c=10\), the corresponding conditional probabilities are approximately 0.0771, 0.2386, and 0.3768. Their average is:

This 0.2308 is a Monte Carlo approximation to the posterior expectation of the model-implied probability for a pair of independent subgroup effects. In a real analysis, use many draws and their actual posterior weights. For a particular pair of fitted subgroups, instead compute the fraction of joint posterior draws in which \(|\theta_j-\theta_k|>10\); that quantity incorporates their observed data and posterior dependence.

Interpreting the Posterior for \(\tau_\theta\)

Practical Diagnostics, Comparisons, and Limitations

Before interpreting posterior summaries, verify that they are reliable summaries of the fitted posterior. Inspect trace plots across chains and check convergence diagnostics and effective sample sizes for \(\tau_\theta\), subgroup effects, and contrasts of clinical interest. Poor mixing, strong autocorrelation, or chains that disagree can make posterior probabilities unstable. A long right tail for \(\tau_\theta\), concentration near its boundary, or very wide subgroup-effect distributions should be reported rather than hidden by a single mean or median.

A common mistake is to conclude that heterogeneity is absent because subgroup credible intervals overlap, or because a point estimate of \(\tau_\theta\) is small. The correction is to examine the posterior distribution of \(\tau_\theta\) and clinically relevant probabilities, and, when comparing particular subgroups, use their joint posterior contrast. Another mistake is to treat any nonzero \(\tau_\theta\) as clinically important. A positive standard deviation indicates variation under the model, but its practical importance depends on the effect units and on how likely differences are to exceed a meaningful threshold.

A point estimate such as the posterior median is easier to communicate than a full distribution, but it suppresses uncertainty and can be misleading when the posterior is skewed or boundary-concentrated. Separate analyses that report only subgroup estimates may obscure the dependence induced by partial pooling. Conversely, a hierarchical posterior can look precise while depending strongly on its exchangeability assumptions, especially with few subgroups and sparse data. Posterior summaries do not repair a poorly specified likelihood, incoherent subgroup definitions, or model misspecification. Posterior predictive checks address a different question—whether the model can reproduce relevant features of the observed data—and are not replaced by inspecting heterogeneity parameters.