Priors for Random-Effect Standard Deviations

Why the Prior on a Standard Deviation Matters

In a nonlinear mixed-effects model, between-subject variability is often represented by random effects. For one random-effect component, a common model is \(b_{ik}\mid \omega_k \sim N(0,\omega_k^2)\), where \(b_{ik}\) is subject \(i\)'s deviation for component \(k\), and \(\omega_k\) is its population standard deviation. The standard deviation must be nonnegative, but the data may provide limited information about its size, especially when the number of subjects is modest or the component is weakly identified. Its prior can therefore matter appreciably.

This tutorial compares three continuous priors for a single standard deviation \(\omega\): the half-normal, half-Student-t, and exponential. Their densities all have support on nonnegative values and are largest at zero, but they differ in how quickly their densities and tail probabilities decrease. Those differences affect how much prior weight they place on very small variability and on unusually large variability.

The densities, with respect to \(\omega\), are

In the half-Student-t expression, \(t_\nu(x)\) denotes the density of a standard Student-t random variable with \(\nu\) degrees of freedom, evaluated at \(x\). The factor of 2 makes the density integrate to 1 on the nonnegative half-line. The half-normal scale \(s\), half-Student-t scale \(A\), and exponential mean \(1/\lambda\) have the same units as \(\omega\). Choose those units to match the random effect: a standard deviation on a log-parameter scale is dimensionless, whereas one on a clearance scale has units of clearance.

Behavior at Zero and in the Tails

All three densities have a finite, positive value at zero. Each distribution is continuous, so \(P(\omega=0)=0\): a high density near zero does not assign positive probability to exactly zero variability. A prior that explicitly allows the random effect to be absent with positive probability would require a separate point mass at zero; that is not any of the three continuous priors defined here.

The half-normal density decreases at a rate proportional to \(\exp(-\omega^2/(2s^2))\), which is a Gaussian, or quadratic-exponent, tail. The exponential density decreases as \(\exp(-\lambda\omega)\), so its tail is lighter than the half-Student-t tail but heavier than the half-normal tail at sufficiently large \(\omega\). For a half-Student-t with finite \(\nu\), the density decreases polynomially, proportional to \(\omega^{-(\nu+1)}\), and its upper-tail probability decreases proportional to \(\omega^{-\nu}\). Thus, it assigns more probability to extreme standard deviations than either of the other two families, for fixed positive scales.

At zero, the values of the densities are

These values depend on the scales, so comparing them is meaningful only after explaining how the scales were chosen. Nor does the tail ordering mean that the half-Student-t is always more dispersed under every possible calibration; comparing priors requires a shared, interpretable calibration.

Calibrating Scales With an Upper Quantile

A scale parameter is not itself generally a useful statement of prior belief. A more interpretable calibration is to choose a scientifically plausible upper value \(U\) and specify, for example, that \(P(\omega\leq U)=0.95\). This says that 95% of the prior probability lies at or below \(U\), before observing the data. It does not say that there is a 95% probability the eventual parameter lies below \(U\) after data are observed.

For a half-normal prior, \(\omega=s|Z|\), so \(P(\omega\leq U)=0.95\) requires \(U/s=z_{0.975}\), the 97.5th percentile of the standard normal. For a half-Student-t, the corresponding condition uses the 97.5th percentile \(t_{\nu,0.975}\). For an exponential prior, its cumulative distribution is \(1-e^{-\lambda U}\). Solving these equations gives

The 0.975 quantile appears for the two half-distributions because \(P(|Z|\leq c)=0.95\) requires 2.5% in each tail of the untruncated, symmetric distribution. The exponential expression follows by setting \(1-e^{-\lambda U}=0.95\) and solving for \(\lambda\).

Interpreting the Standard Deviation in Context

Prior calibration should refer to the role of the random effect in the model, not merely to a number that seems small. If \(b_i\) is a random effect on log clearance, then a one-standard-deviation deviation in \(b_i\) corresponds to a multiplicative factor \(e^\omega\) in clearance relative to its typical value. This interpretation follows from exponentiating the log-scale difference; it does not mean the clearance itself is normally distributed.

Assumptions, Diagnostics, and Limitations

These comparisons concern a scalar standard deviation in a model where the random effect is conditionally normal with mean zero and variance \(\omega^2\). The prior is placed on the standard deviation itself, not on the variance \(\omega^2\). The conclusions about densities at zero also depend on using density with respect to \(\omega\). If instead one defines \(v=\omega^2\), the change-of-variables formula gives

Thus, a finite density for \(\omega\) at zero does not imply a finite density for the variance \(v\) at zero. A prior specified on variance and a prior specified on standard deviation are different prior choices.

Before fitting, examine the implied prior on \(\omega\) in the units and parameterization of the model. Check its density near zero, several upper quantiles, and the probability it assigns to values that would imply implausibly large subject-to-subject variation. After fitting, compare the posterior for \(\omega\) with its prior and check whether the posterior is concentrated near zero or has a long upper tail. Such patterns can indicate weak information in the data or a likelihood that does not distinguish small from moderate variability; they do not, by themselves, prove that the model or computation has failed.

A common failure is to choose a very large scale under the label “weakly informative” without checking the consequences. A broad half-Student-t can assign substantial prior probability to extreme standard deviations, which may imply implausible individual parameter values or make inference sensitive to a small number of subjects. Conversely, an overly tight prior can constrain variability toward zero and conceal genuine heterogeneity. Scale and tail behavior must both be checked.

The half-normal is often suitable when very large standard deviations should be strongly downweighted. The exponential provides a simple one-parameter prior with an exponentially decreasing tail. The half-Student-t is useful when one wants appreciable tail probability for larger values, but that robustness can also permit implausibly large variability if its scale is not calibrated. No family is universally best: the choice depends on the random-effect scale, scientific plausibility, and how much upper-tail probability is intended. These scalar priors specify marginal standard deviations; they do not specify correlations among multiple random effects.