Priors for Residual Error Parameters

Why Residual-Error Priors Need Their Own Calibration

A residual-error prior affects how much variation the model attributes to measurement or unexplained within-individual differences, rather than to the structural model or between-individual variation. It therefore changes the likelihood across observations and can influence estimates of other model parameters. A prior that looks numerically small may still imply excessive error at high predicted concentrations, while a prior that seems broad may be too restrictive near zero.

The key is to assess the prior on the observation scale. An additive residual standard deviation has the same units as the observation. A proportional residual scale is dimensionless, but its implied standard deviation changes with the model prediction. A combined model has both components. Prior calibration should therefore use plausible predictions over the range of the data, not just a stand-alone numerical scale.

This formulation assumes the residuals have conditional mean zero and the stated variance, and that observations are conditionally independent given the model quantities, unless a different dependence structure is explicitly specified. Normality is an additional modeling assumption, not a consequence of choosing a prior. The prior determines plausible values of the residual parameters; it does not repair a misspecified residual distribution or dependence structure.

Three Common Residual-Error Structures

For an additive-error model, the residual standard deviation is constant across predictions. For a proportional-error model, it increases in proportion to the magnitude of the prediction. A common combined-error convention adds the component variances. The formulas below define that convention explicitly; software may use different parameterizations, so check its definition before transferring a prior.

The absolute value in the proportional model ensures that the standard deviation is nonnegative even if a prediction can be negative. When predictions are positive concentrations, \(\sigma_p\) is the conditional residual coefficient of variation under the normal model: the residual SD divided by the prediction is \(\sigma_p\). For example, \(\sigma_p=0.10\) corresponds to a residual SD equal to 10% of a positive prediction. The combined formula does not add the two SDs. It adds variances, as would result from independent additive and prediction-scaled error components.

Choosing Positive-Support Priors

Residual SDs and CVs cannot be negative. A prior directly on one of these scales should therefore have support only on positive values. Common choices include a half-normal, half-\(t\), and lognormal distribution. Each can be reasonable, but its scale and tail behavior must be calibrated to the context.

• Half-normal: the absolute value of a zero-centered normal variable. Its scale parameter \(\tau\) has the same units as the residual SD. It concentrates near zero but has a relatively light tail.
• Half-\(t\): a positive-valued \(t\)-distributed scale, often used when more prior probability for large values is desired. Its scale parameter also uses the units of the residual SD; degrees of freedom control tail heaviness.
• Lognormal: if \(\log(\sigma)\sim N(m,s^2)\), then \(\sigma\) is positive, with median \(\exp(m)\). This parameterization is useful for quantities that vary multiplicatively, but its upper tail can be substantial.

A prior on a variance is not interchangeable with a prior on its standard deviation. For instance, a uniform prior on \(\sigma^2\) induces a density on \(\sigma\) proportional to \(\sigma\), rather than a uniform density on \(\sigma\). Likewise, putting a prior on \(\log(\sigma)\) and putting a prior on \(\sigma\) are different choices. If computation uses \(u=\log(\sigma)\) but the intended prior is specified on \(\sigma\), the density in the sampled coordinate must include the change-of-variable factor:

The factor \(\exp(u)\) is the Jacobian. Omitting it changes the prior on \(\sigma\). Alternatively, one may intentionally specify a normal prior directly on \(u=\log(\sigma)\); that induces a lognormal prior on \(\sigma\), and no additional Jacobian is needed if the model is formulated with that prior on \(u\).

Combined Error: Check the Whole Prediction Range

For a combined model, calibrate both components and inspect the resulting SD at several representative predictions. Neither component can be judged in isolation: the additive term can dominate near zero, while the proportional term can dominate at larger predictions. This behavior follows directly from the variance formula.

For example, the additive and proportional contributions to variance are equal when \(\sigma_a^2=\sigma_p^2 f_{ij}^2\), or, for a positive prediction, when \(f_{ij}=\sigma_a/\sigma_p\). Below that prediction the additive component contributes more variance; above it the proportional component contributes more.

Calibration, Diagnostics, and Common Alternatives

A practical prior check is to draw residual parameters from their prior, combine each draw with plausible model predictions, and inspect the resulting residual SDs or simulated observations on the measurement scale. Include predictions near the low end, a typical value, and the high end. If observations have different units across outcomes, calibrate each outcome's error parameters in its own measurement units. This is a prior predictive check: it asks what data variability the specified model and priors imply before using the observed data.

The alternatives have different consequences. A half-normal is straightforward to calibrate by its scale and suppresses very large values relatively quickly. A half-\(t\) allows more upper-tail mass, which may be useful when large residual SDs are plausible but can also permit implausible observation variability. A lognormal expresses uncertainty multiplicatively and is often easier to discuss through its median and quantiles. A bounded prior can rule out values outside known physical or operational limits, but the bound must be defensible; an arbitrary tight bound can force the likelihood to compensate elsewhere.

A common mistake is to put a normal prior directly on a residual SD because normal priors are familiar. That assigns positive probability to negative SDs, which are not valid values. Another is to put a prior on the residual variance while interpreting its parameter as an SD. The correction is to define the error parameter precisely, choose a prior with appropriate support, and verify the induced variability in the likelihood. Also avoid assuming that a proportional scale is a fixed SD: its implied SD is \(\sigma_p|f_{ij}|\). If residual plots or scientific knowledge indicate correlation, skewness, or a different variance trend, changing the prior family alone does not address that model inadequacy.