Inverse-Gamma Priors and Their Induced Random-Effect Scales

Why the Variance Prior Implies a Scale Prior

A GLMM random effect is often described by a standard deviation because that scale is easier to interpret in the model's units. Yet a prior may instead be specified on the variance. These are not two unrelated choices: a prior on the variance induces a particular distribution on the standard deviation. That induced distribution can be markedly different in shape from the variance density, especially near zero and in the upper tail.

For example, a random intercept might satisfy $u_j\mid\sigma^2\sim N(0,\sigma^2)$, where $\sigma$ is the between-group standard deviation on the model's linear-predictor scale. If a prior is assigned to $\sigma^2$, the distribution of $\sigma=\sqrt{\sigma^2}$ follows by a change of variables. Its shape affects which degrees of group heterogeneity receive substantial prior probability.

Definition: In this tutorial, an inverse-gamma distribution with shape $a>0$ and scale $b>0$ has density $$ \pi(v)=\frac{b^a}{\Gamma(a)}v^{-a-1}\exp\left(-\frac{b}{v}\right),\qquad v>0. $$ Here $v=\sigma^2$, and $\Gamma(a)$ is the gamma function. The induced standard-deviation prior is the distribution of $\sigma=\sqrt{v}$ when $v$ has this density. The parameter $b$ has the same units as a variance; $\sigma$ has the corresponding standard-deviation units.

The parameterization matters: inverse-gamma conventions differ across references, particularly in whether the second parameter is called a scale or a rate. All calculations below use the density displayed above. We assume $a,b>0$, so this is a proper probability distribution supported on positive variances. A prior density is not itself a statement about the posterior; the latter also depends on the likelihood.

Deriving the Induced Density

The transformation is $v=\sigma^2$, with derivative $dv/d\sigma=2\sigma$ for $\sigma>0$. The change-of-variables rule multiplies the variance density evaluated at $\sigma^2$ by this Jacobian.

\begin{aligned} \pi_\sigma(\sigma) &=\pi(\sigma^2)\left|\frac{dv}{d\sigma}\right|\\ &=\frac{b^a}{\Gamma(a)}(\sigma^2)^{-a-1} \exp\left(-\frac{b}{\sigma^2}\right)(2\sigma)\\ &=\frac{2b^a}{\Gamma(a)}\sigma^{-2a-1} \exp\left(-\frac{b}{\sigma^2}\right),\qquad \sigma>0. \end{aligned}

The factor $2\sigma$ is essential. Simply replacing $v$ by $\sigma^2$ in the inverse-gamma density would not yield a density with respect to $d\sigma$. As a check, the transformation preserves total probability: integrating $\pi_\sigma(\sigma)$ over $\sigma>0$ is equivalent, by $v=\sigma^2$, to integrating $\pi(v)$ over $v>0$, which equals one.

A useful alternate derivation starts with $T=b/V$, where $V=\sigma^2$. Under the stated inverse-gamma density, $T$ has a gamma distribution with shape $a$ and rate 1. Therefore, for any $\epsilon>0$,

P(\sigma<\epsilon) =P(V<\epsilon^2) =P\left(T>\frac{b}{\epsilon^2}\right).

This identity turns questions about scale thresholds into gamma-tail calculations. It also makes clear that the probability of an extremely small scale can be very small, even if the variance density appears high at some positive values.

Worked Example: Finding the Induced Density and Its Mode

Suppose, hypothetically, a random-intercept variance measured in squared logit units has an inverse-gamma prior with $a=3$ and $b=2$. Find the induced density for its standard deviation and the mode of that density.

Substitute $a=3$, $b=2$, and $\Gamma(3)=2$ into the transformation:

\pi_\sigma(\sigma) =\frac{2(2^3)}{2}\sigma^{-7}\exp\left(-\frac{2}{\sigma^2}\right) =8\sigma^{-7}\exp\left(-\frac{2}{\sigma^2}\right),\qquad \sigma>0.

To find the mode, differentiate the log density and set the derivative to zero:

\frac{d}{d\sigma}\log\pi_\sigma(\sigma) =-\frac{2a+1}{\sigma}+\frac{2b}{\sigma^3}=0 \quad\Longrightarrow\quad \sigma^2=\frac{2b}{2a+1}.

Thus the mode is $\sqrt{4/7}=0.7559$ logit units. The derivative is positive below this value and negative above it, so this is the density's maximum. This is the mode of the standard-deviation density; it is not the square root of the mode of the variance density. Modes need not be preserved by nonlinear transformations.

Tail Behavior, Moments, and Probability Near Zero

The induced scale density behaves differently at the two ends of its support. As $\sigma$ approaches zero, the exponential factor $\exp(-b/\sigma^2)$ dominates any power of $\sigma$. Consequently, $\pi_\sigma(\sigma)$ approaches zero faster than any positive power of $\sigma$. In particular, for every fixed $a,b>0$, this density does not diverge at zero.

The upper tail is polynomial rather than exponential. Let $T=b/V$, which has a gamma density with shape $a$ and rate 1. For large $x$, the event $\sigma>x$ is $T<b/x^2$. The gamma density near zero is proportional to $t^{a-1}$, so integrating it from zero to $b/x^2$ gives the tail approximation below.

P(\sigma>x) \sim \frac{b^a}{\Gamma(a+1)}x^{-2a} =\frac{b^a}{a\Gamma(a)}x^{-2a}, \qquad x\longrightarrow\infty.

Thus $a$ determines the rate of tail decay: smaller $a$ gives a heavier scale tail. The parameter $b$ also affects its magnitude. This tail behavior has a direct implication for moments. For any real $r$ for which the moment exists, transformation or gamma integration gives

E(\sigma^r) =b^{r/2}\frac{\Gamma(a-r/2)}{\Gamma(a)} \quad\text{if }a>\frac{r}{2}.

The condition is necessary because the gamma integral requires $a-r/2>0$; if it fails, the upper tail makes the moment infinite. For example, $E(\sigma)$ is finite only when $a>1/2$, while $E(\sigma^2)=E(V)=b/(a-1)$ is finite only when $a>1$. A proper prior can therefore have an infinite mean scale or mean variance.

Probability near zero can be calculated directly from the gamma identity. With $z=b/\epsilon^2$, the probability is the upper tail probability of a shape-$a$, rate-1 gamma variable:

P(\sigma<\epsilon)=P(T>z) =\frac{\Gamma(a,z)}{\Gamma(a)}, \qquad z=\frac{b}{\epsilon^2},

where $\Gamma(a,z)=\int_z^\infty t^{a-1}e^{-t}\,dt$ is the upper incomplete gamma function. For small $\epsilon$, $z$ is large; the probability is approximately $z^{a-1}e^{-z}/\Gamma(a)$. This can become extremely small. A prior can have a positive density at every $\sigma>0$ and still assign little probability to a narrow interval immediately above zero.

Worked Example: Probability of a Small Random-Effect Scale

Suppose a hypothetical random-intercept variance, in squared logit units, has $a=2$ and $b=1$. Calculate the prior probability that its standard deviation is below $0.5$ logit units.

The event is $\sigma<0.5$, equivalently $V<0.25$. Under $T=1/V$, it becomes $T>4$, with $T$ gamma distributed with shape 2 and rate 1. Its density is $t e^{-t}$, so

\begin{aligned} P(\sigma<0.5) &=\int_4^\infty t e^{-t}\,dt\\ &=\left[-(t+1)e^{-t}\right]_4^\infty\\ &=5e^{-4}=0.09158. \end{aligned}

Thus this prior assigns about 9.16% probability to a random-intercept standard deviation below 0.5 logit units. This is a prior probability, not a posterior conclusion about a particular data set. The calculation also illustrates why evaluating a scale threshold through the induced gamma tail can be simpler than integrating the scale density directly.

Worked Example: A Proper Prior With an Infinite Mean Scale

Consider a hypothetical inverse-gamma prior with $a=0.5$ and $b=0.5$, where the variance is measured in squared units of a model's linear predictor. Determine whether the induced standard deviation has a finite mean and describe its upper tail.

The density is proper because both parameters are positive. But the moment condition for $E(\sigma^r)$ is $a>r/2$. For the mean, $r=1$, so finiteness requires $a>0.5$. Here $a=0.5$, and therefore $E(\sigma)$ is infinite.

For large $x$, the tail approximation gives

P(\sigma>x) \sim \frac{(0.5)^{0.5}}{\Gamma(1.5)}x^{-1} =\frac{\sqrt{0.5}}{0.5\sqrt{\pi}}x^{-1} \approx 0.7979x^{-1}.

The slow $x^{-1}$ decay explains why the mean diverges: the integral of this tail over large $x$ does not converge. Properness only means the total probability is one; it does not guarantee finite moments or a practically useful tail. A user choosing such a prior should not summarize its implied scale with a finite prior mean.

Interpreting and Assessing the Induced Prior

The mode, moments, and threshold probabilities answer different questions. The mode locates the peak of the density, but it does not show how much probability lies in a small interval around that point. A finite mean summarizes an average only when the relevant moment exists, and with a heavy tail it can be a poor description of typical values. Threshold probabilities such as $P(\sigma<\epsilon)$ directly describe prior mass below a scientifically interpretable scale.

When applying an inverse-gamma prior in a GLMM, first verify the parameterization and units, then examine the induced density on the standard-deviation scale. Useful checks include calculating the mode, confirming whether moments exist, and evaluating probability below and above scales that would correspond to plausible or implausible group heterogeneity. These checks assess the prior itself; they do not establish that the likelihood or fitted model is adequate.

The inverse-gamma construction can be compared with assigning a prior directly to $\sigma$. The former may be convenient in conjugate calculations and has a simple variance parameterization, but its induced scale distribution has a constrained shape: a rapidly suppressed density near zero and a polynomial upper tail. A direct scale prior allows the density to be chosen on the interpretable standard-deviation scale instead. These approaches are not interchangeable merely because both refer to the same random-effect scale; they encode different distributions and can imply different tail probabilities.

Common pitfall: It is incorrect to read an inverse-gamma variance density as though it were the density of the standard deviation, or to replace $v$ by $\sigma^2$ without a Jacobian. The correction is to use $\pi_\sigma(\sigma)=2\sigma\,\pi(\sigma^2)$, then assess the resulting scale density and probabilities in the units of $\sigma$.

Several warning signs merit attention. A claimed finite mean should be checked against $a>1/2$, and a claimed finite mean variance against $a>1$. A plotted density alone can conceal a long tail, particularly when the horizontal axis is truncated. Conversely, a density near zero that looks small does not by itself quantify the probability of practically negligible heterogeneity; calculate a threshold probability. Finally, results should not be compared across models unless their scale units and inverse-gamma parameterizations agree.

The inverse-gamma family is only one way to express prior beliefs about a random-effect scale. Its induced density is mathematically precise, but whether its near-zero behavior and tail are suitable depends on the application and chosen hyperparameters. Understanding the transformation makes that implication visible before the prior is used in posterior inference.

Check Your Understanding

Answer each question, then compare with the solutions below.

If $V$ has an inverse-gamma distribution with shape $a$ and scale $b$, what is the induced density of $\sigma=\sqrt V$?
For $a=2$ and $b=3$, find the mode of the induced standard-deviation density.
For what values of $a$ is $E(\sigma)$ finite? For what values is $E(V)$ finite?
Explain why a proper inverse-gamma prior can imply an infinite mean standard deviation.

Solutions. (1) The change of variables $v=\sigma^2$ has Jacobian $2\sigma$, so $\pi_\sigma(\sigma)=2b^a\sigma^{-2a-1}\exp(-b/\sigma^2)/\Gamma(a)$ for $\sigma>0$. (2) The mode is $\sqrt{2b/(2a+1)}=\sqrt{6/5}=1.095$ standard-deviation units. (3) $E(\sigma)$ is finite when $a>1/2$; $E(V)$ is finite when $a>1$. (4) Properness requires only $a,b>0$, whereas finiteness of $E(\sigma)$ additionally requires $a>1/2$. For $0<a\leq 1/2$, the prior is proper but its polynomial upper tail is too heavy for a finite mean scale.

Inverse-Gamma Priors and Their Induced Random-Effect Scales

What You'll Learn