Exponential and Penalized-Complexity Priors for Group-Level Scales

Why Use a Prior on a Group-Level Scale?

In a generalized linear mixed model, a group-level random effect represents variation among groups after accounting for the fixed effects and the specified response distribution. For example, a random-intercept model may assign group $j$ an intercept deviation $b_j$ with $b_j\mid\sigma\sim N(0,\sigma^2)$. The standard deviation $\sigma$ measures the scale of group heterogeneity on the model’s linear-predictor scale. For a binary model with a logit link, that scale is in log-odds units.

The value $\sigma=0$ is a meaningful boundary: all group deviations are zero, so the random intercept contributes no group-level variation. An exponential prior on $\sigma$ gives a direct way to put substantial prior density near that simpler model while retaining a right tail for larger scales. A penalized-complexity (PC) prior formalizes this preference for a simpler base model, with a tail-probability condition providing a transparent calibration.

Definition: An exponential prior on a group-level standard deviation $\sigma\geq 0$, with rate $\lambda>0$, has density $$ \pi(\sigma)=\lambda\exp(-\lambda\sigma),\qquad \sigma\geq 0. $$ Its cumulative distribution function is $P(\sigma\leq s)=1-\exp(-\lambda s)$ for $s\geq0$, and its upper-tail probability is $P(\sigma>s)=\exp(-\lambda s)$. The rate $\lambda$ has units reciprocal to those of $\sigma$.

Exponential Priors as PC Priors for Gaussian Scales

The PC-prior construction begins by identifying a simpler base model and penalizing movement away from it. For a random-effect scale, the base model is $\sigma=0$, which removes that source of group-level variation. In general, the construction uses a distance $d$ from the base model and assigns that distance an exponential distribution: $\pi(d)=\lambda\exp(-\lambda d)$, for $d\geq0$. If $d=d(\sigma)$ is an increasing function of the scale, the density on $\sigma$ is obtained by change of variables:

\pi(\sigma)=\lambda\exp\{-\lambda d(\sigma)\} \left|\frac{d\,d(\sigma)}{d\sigma}\right|.

For the canonical Gaussian random-effect scale, the PC distance is $d(\sigma)=\sigma$. Consequently, the PC prior on this scale is exactly the exponential prior in the definition. This is a statement about the chosen scale parameterization and PC distance. It is not obtained by inserting the singular point-mass model at $\sigma=0$ into the ordinary Kullback–Leibler divergence between two nonsingular normal densities.

The density is largest at the boundary, $\pi(0)=\lambda$, and decreases as $\sigma$ increases. Its negative log density, apart from a constant, is $\lambda\sigma$. Thus larger scales receive a linear penalty on the standard-deviation scale. This does not put a point mass at zero: under a continuous exponential prior, $P(\sigma=0)=0$, even though arbitrarily small positive scales can receive appreciable probability.

Key result: For the Gaussian group-level scale with base model $\sigma=0$ and PC distance $d(\sigma)=\sigma$, the PC prior is $\sigma\sim\operatorname{Exponential}(\lambda)$. Its rate is uniquely determined by one specified tail condition $P(\sigma>U)=\alpha$, where $U>0$ and $0<\alpha<1$: $$ \lambda=-\frac{\log(\alpha)}{U}. $$

Calibrating the Rate from a Tail Probability

The calibration condition says that the prior assigns probability $\alpha$ to scales larger than the specified threshold $U$. Substituting the exponential survival function gives $\exp(-\lambda U)=\alpha$. Taking logarithms and solving yields the rate formula above. Because $-\log(\alpha)>0$, the rate is positive. A smaller $\alpha$, holding $U$ fixed, gives a larger rate and more concentration toward zero; a larger $U$, holding $\alpha$ fixed, gives a smaller rate and a more dispersed prior.

The threshold $U$ must be expressed in the same units and parameterization as $\sigma$. This tutorial treats $U$ and $\alpha$ as specified calibration inputs; interpreting a scientifically plausible value for $U$ is a separate judgment. Once calibrated, the prior can be checked by calculating its median, other quantiles, and probabilities at scales of interest.

Worked Example: Calibrating a Log-Odds Scale

Suppose a hypothetical binary GLMM uses a random intercept with standard deviation $\sigma$, measured in log-odds units. Specify the prior so that $P(\sigma>0.8)=0.05$. Find the rate, median, and prior probability that $\sigma\leq0.2$.

Rate. The model assumption for this prior is a single nonnegative scale with the canonical exponential PC distance $d(\sigma)=\sigma$. The tail condition is $e^{-0.8\lambda}=0.05$, so

\lambda=\frac{-\log(0.05)}{0.8} =\frac{2.995732}{0.8} =3.744665\ \text{per log-odds unit}.

Median. The median $m$ satisfies $1-e^{-\lambda m}=0.5$. Using the unrounded calibrated rate,

m=\frac{\log(2)}{\lambda} =\frac{0.6931472}{3.744665} \approx0.1851\ \text{log-odds units}.

Probability below 0.2. Applying the exponential CDF,

P(\sigma\leq0.2) =1-\exp(-3.744665\times0.2) =1-\exp(-0.7489331) \approx1-0.4729 =0.5271.

Under this prior, the probability that the random-intercept standard deviation is at most 0.2 log-odds units is about 0.5271. The stated 5% tail probability is prior probability, not a posterior conclusion or a probability that a model hypothesis is true.

From Standard Deviation to Variance

The parameter in the exponential prior is the standard deviation, not the variance. If the variance is $v=\sigma^2$, then $\sigma=\sqrt v$ and $d\sigma/dv=1/(2\sqrt v)$ for $v>0$. Applying the change-of-variables formula gives the induced prior density on $v$:

\pi(v)=\frac{\lambda}{2\sqrt v}\exp(-\lambda\sqrt v), \qquad v>0.

This density is proper because it is the transformation of a proper exponential density. It becomes unbounded as $v$ approaches zero, but that behavior does not create a point mass at zero. The tail event transforms as $P(v>V)=P(\sigma>\sqrt V)=\exp(-\lambda\sqrt V)$. This makes it especially important to state clearly whether a quoted threshold refers to a standard deviation or a variance.

Worked Example: Induced Prior on a Variance

Suppose a hypothetical random-intercept standard deviation has prior $\sigma\sim\operatorname{Exponential}(2)$, with $\sigma$ measured in log-odds units. Let $v=\sigma^2$. Find the density on $v$, the probability that $v\leq0.25$, and the density value at $v=0.25$.

The transformation uses $\sigma=\sqrt v$ and $d\sigma/dv=1/(2\sqrt v)$. Therefore,

\pi(v)=\frac{2}{2\sqrt v}\exp(-2\sqrt v) =\frac{1}{\sqrt v}\exp(-2\sqrt v),\qquad v>0.

Since $v\leq0.25$ is equivalent to $\sigma\leq0.5$,

P(v\leq0.25)=1-\exp(-2\times0.5) =1-e^{-1} \approx0.6321.

At $v=0.25$, the induced density is

\pi(0.25)=\frac{1}{\sqrt{0.25}}\exp(-2\sqrt{0.25}) =2e^{-1} \approx0.7358

per squared-log-odds unit. The density’s units are reciprocal variance units; it is not itself a probability. The probability calculation, by contrast, is dimensionless.

What the Tail Calibration Means

A tail condition is easy to verify and communicate. If $P(\sigma>U)=\alpha$, then $U$ is the prior upper-tail threshold and $\alpha$ is the probability assigned above it. The same condition also determines the prior median and any other quantile. In particular, the $q$ quantile, for $0<q<1$, is

\sigma_q=\frac{-\log(1-q)}{\lambda}.

For example, the median follows by setting $q=0.5$. A useful check is to substitute the calibrated rate back into the tail expression: $\exp\{-[-\log(\alpha)/U]U\}=\alpha$. This identity confirms that the calibration is internally consistent.

Worked Example: A Second Tail Calibration

For a hypothetical group-level scale, suppose the prior should satisfy $P(\sigma>0.6)=0.10$, with $\sigma$ measured in the model’s linear-predictor units. Calculate the rate and the probability below 0.2 in those same units.

The prior assumes a nonnegative scale and the exponential PC form. The specified tail condition gives

\lambda=\frac{-\log(0.10)}{0.6} =\frac{2.302585}{0.6} =3.837642\ \text{per linear-predictor unit}.

The requested lower-tail probability is

P(\sigma\leq0.2) =1-\exp(-3.837642\times0.2) =1-\exp(-0.7675284) \approx1-0.4642 =0.5358.

Thus, this prior assigns about 0.5358 probability to scales at or below 0.2 and exactly 0.10 probability to scales above 0.6, up to rounding. The two probabilities refer to the specified prior distribution, not to data-based evidence.

Assumptions, Diagnostics, and Alternatives

The construction is appropriate when the scale is nonnegative, the group-level effect is represented by the intended Gaussian scale model, and the zero-scale model is the desired simpler base. The exponential prior is proper for any finite $\lambda>0$. Its rate must match the units of the scale: if the scale is rescaled by a positive constant, the numerical rate must change inversely to preserve the same prior distribution on the rescaled quantity. The group-level distribution and link function determine the interpretation of the scale; for example, a random-intercept scale in a logit model is not measured in probability units.

Prior checks should at minimum verify the calibration equation, the implied median or another quantile, and probabilities at representative scales. A mismatch between the computed and requested tail probability often indicates a unit error, use of the variance in place of the standard deviation, or a sign error in the logarithm. Very small $\alpha$ or a small $U$ produces a large rate and strong concentration near zero; this should be recognized as a substantive prior choice rather than a harmless numerical setting.

Prior choice	Scale behavior	Practical distinction
Exponential / canonical Gaussian-scale PC prior	Density decreases exponentially in $\sigma$	Direct tail calibration; linear penalty in the scale
Half-normal prior	Obtained by restricting a centered normal distribution to nonnegative values	Tail decreases quadratically in the scale on the log-density scale
Half-Student-$t$ prior	Nonnegative Student-$t$ scale with heavier tails	Can retain more prior probability at very large scales, depending on degrees of freedom and scale

These choices are alternatives, not interchangeable labels. In particular, an exponential prior on $\sigma$ is not an exponential prior on $v=\sigma^2$. The induced variance density contains the Jacobian factor $1/(2\sqrt v)$, so assigning the same named density directly to the variance defines a different prior. A continuous exponential prior also cannot test a point-null hypothesis $\sigma=0$ by assigning positive probability to it; that requires a prior with a point mass at zero.

Common pitfall: A student sets $v=\sigma^2$, uses the calibrated rate $\lambda$, and writes $\pi(v)=\lambda e^{-\lambda v}$. This silently changes the prior: it is exponential in the variance, not the variance induced by an exponential prior on the standard deviation. The correct transformation is $\pi(v)=\lambda e^{-\lambda\sqrt v}/(2\sqrt v)$. Always identify the parameter being assigned the prior before transforming or interpreting its rate.

An additional limitation is that a single marginal prior on one scale does not, by itself, specify dependence among several random-effect scales or correlations. Nor does a mathematically transparent tail calibration ensure that its chosen threshold is suitable for a particular application. Those questions require substantive decisions and assessment of the implications of the prior, not just algebraic calibration.

Name the scale and units.
State whether the parameter is a standard deviation and identify the scale on which it is measured.

State the base model and distance.
For the canonical Gaussian random-effect scale, use the simpler model $\sigma=0$ and $d(\sigma)=\sigma$.

Apply the tail condition.
For $P(\sigma>U)=\alpha$, calculate $\lambda=-\log(\alpha)/U$.

Check the implications.
Substitute the rate back into the tail equation and inspect a quantile or additional probability in the stated units.

Check Your Understanding

Answer each question, then compare with the solutions below.

For an exponential prior with rate $\lambda=1.5$ per unit, calculate $P(\sigma>2)$.
A prior must satisfy $P(\sigma>1.2)=0.20$. Find its exponential rate.
If $\sigma$ has an exponential prior, is there positive prior probability that $\sigma=0$? Explain.
For $v=\sigma^2$, write the density induced by $\sigma\sim\operatorname{Exponential}(\lambda)$.
Explain why an exponential density assigned directly to $v$ is not the same prior as an exponential density assigned to $\sigma$.

Solutions. (1) $P(\sigma>2)=e^{-1.5(2)}=e^{-3}\approx0.04979$. (2) $\lambda=-\log(0.20)/1.2=1.609438/1.2\approx1.341$ per unit; substitution gives $e^{-1.341\times1.2}\approx0.20$. (3) No. A continuous exponential distribution assigns probability zero to the single point $\sigma=0$, though it has density $\lambda$ there and can assign positive probability to intervals immediately above zero. (4) The change of variables gives $\pi(v)=\lambda e^{-\lambda\sqrt v}/(2\sqrt v)$ for $v>0$. (5) They apply exponential decay to different parameters, and transforming from $\sigma$ to $v$ introduces both $\sqrt v$ in the exponential and the Jacobian $1/(2\sqrt v)$.

Prior choice	Scale behavior	Practical distinction
Exponential / canonical Gaussian-scale PC prior	Density decreases exponentially in \(\sigma\)	Direct tail calibration; linear penalty in the scale
Half-normal prior	Obtained by restricting a centered normal distribution to nonnegative values	Tail decreases quadratically in the scale on the log-density scale
Half-Student-\(t\) prior	Nonnegative Student-\(t\) scale with heavier tails	Can retain more prior probability at very large scales, depending on degrees of freedom and scale