Priors for Continuous-Outcome Variability

Why the Prior for Variability Matters

In a trial with a continuous outcome, the standard deviation describes how widely individual outcomes vary around their modeled mean. It is not usually the treatment effect of direct interest, but it can strongly affect uncertainty about that effect: a model that assigns substantial prior probability to very large standard deviations also anticipates highly variable observations. A prior concentrated on unrealistically small values can have the opposite problem.

The prior should therefore be assessed on the clinical measurement scale, such as points on a symptom score or millimeters of mercury, rather than judged only by its mathematical form. This tutorial compares three choices for the outcome standard deviation \(\sigma\): a half-normal, a half-\(t\), and an inverse-gamma prior on the variance \(\sigma^2\). Each is proper under the parameter conditions stated below.

The scale \(A\) has the same units as \(\sigma\). It is not generally the prior mean or standard deviation of \(\sigma\). The inverse-gamma alternative is usually placed on the variance, so its parameters have a different interpretation.

Support, Behavior Near Zero, and Tail Weight

All three priors put their continuous probability on positive standard deviations, which respects the definition of a standard deviation. None assigns point mass to exactly zero. They differ in how much density they place near zero and how quickly their upper tails decline.

The half-normal density is finite at zero and declines exponentially in \(\sigma^2\). The half-\(t\) density is also finite at zero, but declines polynomially in the tail. The inverse-gamma density on \(V\), when expressed as a density on \(\sigma\), approaches zero as \(\sigma\) approaches zero and has a polynomial upper tail. For \(a,b>0\), the transformation \(v=\sigma^2\) gives

The factor \(\exp(-b/\sigma^2)\) drives this density toward zero near zero. For large \(\sigma\), that factor approaches one, leaving a density proportional to \(\sigma^{-2a-1}\); the corresponding survival probability declines approximately as \(\sigma^{-2a}\). For a half-\(t\), the density declines approximately as \(\sigma^{-(\nu+1)}\), and its survival probability as \(\sigma^{-\nu}\). A half-normal tail declines much faster, at an exponential rate in \(\sigma^2\).

Tail weight is not automatically a measure of quality. A heavier tail can reduce the influence of an upper scale choice, but it can also give appreciable prior probability to clinically implausible variability. Conversely, a light tail may express a well-supported upper bound, or may be unjustifiably restrictive.

Calibrating on the Clinical Scale

A useful calibration question is: what prior probability should be assigned to the outcome standard deviation exceeding a threshold with a clear clinical interpretation? The threshold might correspond to variability that would make a planned measurement instrument difficult to use, or to a range of dispersion considered implausible from prior studies. The answer is a prior judgment, not a fact established by the distribution family.

For a half-normal with scale \(A\), the cumulative probability at \(c>0\) is \(2\Phi(c/A)-1\), where \(\Phi\) is the standard normal cumulative distribution function. For a half-\(t\), the corresponding cumulative probability is \(2T_\nu(c/A)-1\), where \(T_\nu\) is the \(t_\nu\) cumulative distribution function. These formulas make it possible to evaluate a proposed scale and degrees of freedom using clinically chosen thresholds.

Inverse-Gamma Priors and a Variance-to-Standard-Deviation Calculation

An inverse-gamma prior is specified for variance, not directly for standard deviation. Its parameters should not be read as though \(b\) were a typical standard deviation. For the stated parameterization, \(E(V)=b/(a-1)\) when \(a>1\); this is an expectation of variance and has squared outcome units. Moreover, moments of \(\sigma\) need not exist for all \(a\): \(E(\sigma^r)\) is finite only when \(r<2a\).

A convenient way to calculate prior probabilities is to use the transformation \(Y=b/V\). If \(V\sim\operatorname{IG}(a,b)\), then \(Y\) has a gamma distribution with shape \(a\) and rate \(1\). Thus \(V>c^2\) is equivalent to \(Y<b/c^2\). For integer \(a=3\), the gamma cumulative distribution function is \(1-\exp(-y)(1+y+y^2/2)\).

Justification for the Tail Comparison

The contrasting upper tails follow directly from the densities. For the half-normal, the density is proportional to \(\exp[-\sigma^2/(2A^2)]\), so its exponential factor becomes very small as \(\sigma\) grows. For the half-\(t\), the standard \(t_\nu\) density is proportional in the tail to \((1+t^2/\nu)^{-(\nu+1)/2}\). Substituting \(t=\sigma/A\), the density is proportional to \(\sigma^{-(\nu+1)}\) for large \(\sigma\). Integrating a density with this power-law behavior gives survival probability proportional to \(\sigma^{-\nu}\).

For the inverse-gamma prior, the density transformation above gives a factor \(\sigma^{-2a-1}\exp(-b/\sigma^2)\). As \(\sigma\) grows, \(\exp(-b/\sigma^2)\) tends to one. Integrating the resulting power law yields survival probability proportional to \(\sigma^{-2a}\). This establishes the stated tail-rate comparison; it does not imply that one prior always has a heavier tail than another, because the parameters must also be considered.

Model Checks, Alternatives, and Limitations

A variability prior is interpreted within a likelihood model. For example, if a continuous outcome is modeled as \(X\mid\mu,\sigma\sim N(\mu,\sigma^2)\), the prior describes plausible values of the residual standard deviation in that model, conditional on its assumptions. It does not by itself account for skewness, outliers, measurement error, or unmodeled subgroups. A prior that looks reasonable in isolation may still produce implausible outcomes when combined with the likelihood.

Useful checks include calculating prior probabilities at several clinically interpretable thresholds and simulating prior-predictive outcomes under the proposed outcome model. Look for values that are implausibly concentrated, implausibly dispersed, or incompatible with the instrument’s possible range. Also check the units: converting an outcome from points to tens of points changes the numerical standard deviation and requires rescaling \(A\), or \(b\) by the square of the unit-conversion factor. In particular, failing to square a unit conversion for \(b\) changes the implied prior.

The three families offer different compromises. A half-normal is simple and has a rapidly declining tail; it can be suitable when very large variability is genuinely implausible, but may be too restrictive if that judgment is uncertain. A half-\(t\) allows more upper-tail probability, with \(\nu\) controlling how heavy the tail is; it still requires careful scale selection. An inverse-gamma prior works naturally when a model is parameterized in terms of variance, but its two parameters can be harder to interpret clinically, and its behavior near zero and in the tail may not match substantive beliefs. Other positive distributions may also be considered, but changing families does not remove the need for scale-based calibration and predictive checks.

Sensitivity analysis is especially useful when the clinical evidence does not determine the upper tail clearly. Compare a small set of defensible priors—for example, a half-normal and a half-\(t\) calibrated to have similar probabilities below a clinically important threshold—and examine whether trial conclusions change. Sensitivity analysis cannot make an implausible prior acceptable; each candidate must first be checked against clinical knowledge and the outcome scale.