Posterior Geometry in Hierarchical Models

Why Posterior Geometry Matters

A hierarchical NLME posterior can be difficult to explore even when its density is well-defined and the model is scientifically sensible. The difficulty may come from its geometry: the way posterior probability is distributed across the joint parameter space. Some regions may be broad while others are extremely narrow, or parameters may be constrained to move together along a long ridge. These features matter because gradient-based sampling methods use local changes in the log posterior to propose movement. If the local scale or direction of the posterior changes sharply, a method that works well in one region may work poorly in another.

A common source of such geometry is a weakly informed random-effect variance component. When data provide little information about individual effects, the posterior can retain a strong dependence between the size of a random-effect scale and the range of plausible random effects. Correlations can arise as well: for example, a population mean and an individual random effect may compensate for one another in the likelihood.

Here, \(\eta_i\) denotes an individual random effect and \(\Omega\) its population covariance matrix. For a single random-effect component, write \(\tau\) for its standard deviation. In a centered parameterization, the model specifies \(\eta_i\mid\tau\sim N(0,\tau^2)\). The likelihood for subject \(i\) is written \(f(y_i\mid\eta_i,\theta)\), where \(\theta\) denotes population and other model parameters.

How a Weakly Informed Scale Creates a Funnel

In centered coordinates, the random effects themselves are parameters, and their prior distribution depends on \(\tau\). For \(m\) scalar random effects, their conditional prior density is

When \(\tau\) is small, the prior concentrates the \(\eta_i\) close to zero; when \(\tau\) is larger, it permits them to spread farther out. If the data do not strongly determine the individual \(\eta_i\), this scale dependence remains prominent in the posterior. In a plot with \(\tau\) on one axis and an \(\eta_i\) on another, the narrow region at small \(\tau\) opens out as \(\tau\) grows, giving the characteristic funnel shape.

The factor \(\tau^{-m}\) is important: it ensures the conditional density integrates to one for each \(\tau\). It also means that the joint density can become high near the narrow end even though that region occupies little volume. Density height alone therefore does not describe how much posterior probability a region contains.

The following conditional calculation makes the changing spread explicit. Consider one random effect with prior \(\eta\mid\tau\sim N(0,\tau^2)\) and an observation \(y\mid\eta\sim N(\eta,s_y^2)\), with \(\tau\) treated as fixed. Completing the square in the product of the normal prior and likelihood gives

If the observation is imprecise relative to the random-effect scale, then \(s_y^2\) is large compared with \(\tau^2\), and the conditional variance is close to \(\tau^2\). The data have not removed the scale-dependent spread imposed by the hierarchical distribution.

How Dependence Creates Posterior Ridges

A ridge can occur when different parameter combinations make nearly the same predictions. In an NLME model, a single subject’s observation may depend on the sum of a population effect and an individual deviation. If the data constrain that sum more strongly than either component separately, increasing one component can be offset by decreasing the other.

This dependency can be derived in a simple Gaussian model. Suppose \(y\mid\beta,\eta\sim N(\beta+\eta,\sigma_y^2)\), with independent priors \(\beta\sim N(0,v_\beta)\) and \(\eta\sim N(0,\tau^2)\). The conditional posterior precision matrix for \((\beta,\eta)\) is

The positive off-diagonal precision expresses the shared contribution of \(\beta\) and \(\eta\) to the prediction. Inverting this matrix produces a negative posterior covariance: conditional on their sum being informed by \(y\), a larger \(\beta\) tends to accompany a smaller \(\eta\). In richer models, dependencies may involve several fixed effects, random-effect scales, correlations, and residual-error parameters.

Centered and Noncentered Coordinates

The funnel can sometimes be made easier to explore by expressing random effects in standardized coordinates. Define \(z_i=\eta_i/\tau\), or equivalently \(\eta_i=\tau z_i\), where \(z_i\sim N(0,1)\). In this noncentered parameterization, the prior for \(z_i\) does not change its scale as \(\tau\) varies. The likelihood still depends on the product \(\tau z_i\), so the transformation does not remove all posterior dependence; it relocates and may reduce some of it.

This is a change of coordinates, not a change in the statistical model. To see why the density must transform, use \(\eta=\tau z\) with \(\tau>0\). The absolute Jacobian for the change from \(z\) to \(\eta\) is \(\tau\) for one random effect. Thus

where \(\phi\) is the standard normal density. For multiple independent scalar random effects, the Jacobian is \(\tau^m\), cancelling the \(\tau^{-m}\) factor in their centered conditional density.

Choosing and Assessing a Parameterization

Neither centered nor noncentered coordinates are uniformly superior. Centered coordinates can work well when observations strongly inform individual random effects, because the likelihood then helps determine \(\eta_i\) directly. Noncentered coordinates can be helpful when individual effects are weakly informed: the standardized \(z_i\) retain a fixed prior scale while \(\tau\) controls how strongly they affect the model predictions. In practice, the appropriate choice depends on the information in the data and on the model’s nonlinear structure.

Before sampling, geometry can be assessed by examining the model and representative posterior regions. Useful clues include:

• Changing conditional spread: ask whether plausible random-effect values become much narrower at small variance-component values.
• Parameter trade-offs: determine whether multiple parameter combinations make nearly identical predictions, as with \(\beta+\eta\) in the worked example.
• Different local scales: note whether one direction in parameter space is much narrower than another, or whether the narrow direction changes across the posterior.
• Weak variance-component information: check whether the likelihood meaningfully distinguishes small from moderate random-effect scales, rather than relying mostly on the prior.

These clues do not by themselves establish that a sampler has failed. They identify features that can make exploration difficult and motivate careful assessment of sampling behavior. A posterior density may also have skewness, boundaries, multiple regions, or other features not captured by a simple funnel or elliptical ridge.

Why Gradient-Based Sampling Can Struggle

A gradient describes the local direction and rate of change in log posterior density. In a funnel, the width available to a random effect can vary substantially with \(\tau\). A step that is modest relative to the broad part of the posterior may be too large for the narrow part; a step small enough for the narrow part may move inefficiently through broad regions. In a ridge, gradients can point steeply across the ridge while changing only gradually along it. The sampler must contend with sharply different behavior in different directions.

This is a geometric explanation of a challenge, not a guarantee of failure. The same model can be easier or harder under different parameterizations, and a well-chosen parameterization does not substitute for checking actual sampling performance. The details of Hamiltonian Monte Carlo and its diagnostics are treated separately; here, the key point is that changing local scales and strong posterior dependencies make a single local description of the posterior inadequate.

Common Pitfalls and Limitations

• Assuming every small variance component causes a funnel. A scale near zero is not, by itself, proof of problematic geometry. The concern is the joint dependence between the scale and random effects, especially when data leave those effects weakly informed.
• Treating noncentering as a universal fix. Noncentering can help when individual effects are weakly informed, but it can create difficult dependence when the data strongly determine those effects. Compare parameterizations in the context of the likelihood.
• Confusing posterior correlation with population correlation. A posterior ridge describes uncertainty and dependence in parameter values given the data. It is not automatically a claim about an association among subjects.
• Ignoring coordinates and the Jacobian. A transformed density has a corresponding volume adjustment. Density heights can change under reparameterization even though the underlying probability model does not.

Simple Gaussian calculations reveal the source of a ridge or changing conditional scale, but real NLME posteriors can be nonlinear and non-Gaussian. Local correlations may vary over the posterior, and a two-dimensional plot may hide dependencies involving other parameters. Geometry checks are therefore aids to understanding and model parameterization, not complete diagnostics on their own.