Deriving Residual Variance Functions From Observation Models

From an Observation Model to a Variance Function

In an NLME model, the structural model gives a prediction for an observation, while the observation model describes how measured values vary around that prediction. The observation model may be specified by a likelihood or by an equation involving a residual error. Either way, it implies a variance at the scale on which the data are recorded. Deriving that variance makes clear what assumptions about precision are built into the model.

Write \(Y_{ij}\) for the observation on subject \(i\) at observation occasion \(j\), and \(F_{ij}\) for its model prediction. The prediction may depend on time, dose, subject effects, and occasion effects. The residual variance considered here is conditional on the prediction and those effects. It describes observation-level variability after holding the model state fixed; it is not the total variability among observations from a population.

A variance function is a consequence of the observation model, not an independent description that can be chosen without regard to the likelihood. For example, if an observation is modeled as its prediction plus a mean-zero error with constant variance, the implied variance is constant. If the error is scaled by the prediction, the implied variance depends on the squared prediction. The same symbol for a residual term should not obscure these different assumptions.

Additive and Proportional Error

Under an additive observation model, write \(Y_{ij}=F_{ij}+\epsilon_{ij}\), with \(E(\epsilon_{ij}\mid F_{ij})=0\) and \(\operatorname{Var}(\epsilon_{ij}\mid F_{ij})=\sigma_a^2\). The recorded observation has conditional mean \(F_{ij}\). Adding a fixed prediction does not change variance, so its conditional variance is \(\sigma_a^2\), regardless of the size of \(F_{ij}\).

This is constant absolute precision: the typical residual scale is the same in the units of \(Y\) across predictions. It is appropriate as a description when measurement scatter is approximately constant in those units. The derivation itself does not establish that the assumption is suitable for a particular data set.

Under a proportional-error model, write \(Y_{ij}=F_{ij}(1+\epsilon_{ij})\), where the relative error has conditional mean zero and variance \(\sigma_p^2\). For a fixed prediction, multiplying a random variable by \(F_{ij}\) multiplies its variance by \(F_{ij}^2\). Therefore,

Here \(\sigma_p\) is dimensionless, and the conditional standard deviation is \(|F_{ij}|\sigma_p\). For positive predictions, the conditional coefficient of variation—the standard deviation divided by the conditional mean—is \(\sigma_p\). Thus proportional error represents approximately constant relative precision rather than constant absolute precision. The normal-error version of this model can assign probability to negative observations; that may be negligible for small relative error but is a limitation when nonnegative measurements are required.

Transformed-Scale Error and the Delta Method

A transformed observation model places the residual error on a transformed scale. Let \(g\) be a transformation and write \(g(Y_{ij})=g(F_{ij})+\epsilon_{ij}\), where \(E(\epsilon_{ij}\mid F_{ij})=0\) and \(\operatorname{Var}(\epsilon_{ij}\mid F_{ij})=\tau^2\). To translate this model to the observation scale, let \(h=g^{-1}\), so that \(Y_{ij}=h(g(F_{ij})+\epsilon_{ij})\).

For a local first-order approximation, the inverse transformation must be differentiable at \(g(F_{ij})\), with derivative \(h'(g(F_{ij}))=1/g'(F_{ij})\). A standard sufficient condition is that \(g\) be continuously differentiable near \(F_{ij}\) and \(g'(F_{ij})\ne0\). Taylor expansion around the prediction on the transformed scale gives the justification:

Taking the variance of the first-order expression, conditional on \(F_{ij}\), yields the delta-method approximation

The squared derivative is essential: variance scales by the square of a multiplicative change. This result is approximate because the Taylor expansion omits curvature and higher-order terms. It is most useful when residual variation on the transformed scale is small enough that the local linear approximation is adequate. It does not, by itself, imply that the observation-scale conditional mean is exactly \(F_{ij}\).

What the Variance Function Does—and Does Not—Describe

The residual variance function is conditional. If predictions differ because subjects have different random effects, the population-level variance also reflects variation in those effects. In general, for a random prediction \(F_{ij}\), the law of total variance states

The first term averages conditional residual variance over predictions; the second captures variation in conditional means. Consequently, a constant additive residual variance does not mean that all observations in the population have the same total variance, and a prediction-dependent residual variance does not by itself describe between-subject or between-occasion variability.

A useful check is to compare the assumed variance pattern with residuals at different prediction levels, while remembering that structural-model errors can also appear as residual patterns. Under a proposed variance function \(V(F_{ij})\), standardized residuals are \(r_{ij}=(Y_{ij}-E[Y_{ij}\mid F_{ij}])/\sqrt{V(F_{ij})}\), when the conditional mean and variance are defined by the model. If their spread changes systematically with prediction, the variance specification or another part of the model may be inadequate. Such a pattern is a warning, not proof that a particular alternative is correct.

Assumptions, Alternatives, and Common Pitfalls

These derivations condition on the prediction and require the stated conditional residual moments. They do not require that residuals from different observations be independent merely to calculate each observation's variance. Independence is an additional likelihood assumption often used for the joint observation model; if residuals are correlated, the marginal variance at each observation can still be as derived, but the joint covariance also needs specification.

The main alternatives encode different relationships between prediction and precision. Additive error keeps absolute variance fixed; proportional error makes variance grow quadratically with prediction; transformed error induces a variance determined by the local slope of the transformation. Choosing among them should reflect measurement behavior and the data, not just which formula is easiest to fit. A transformation can also change the conditional mean after returning to the observation scale, as the log example demonstrates.

• Pitfall: treating transformed-scale variance as observation-scale variance. If \(\operatorname{Var}(\epsilon_{ij}\mid F_{ij})=\tau^2\) on the transformed scale, it does not follow that \(\operatorname{Var}(Y_{ij}\mid F_{ij})=\tau^2\). The transformation's derivative scales the variance locally, and the exact result may require more than a first-order approximation.
• Pitfall: confusing variance with standard deviation. Proportional error gives variance \(F_{ij}^2\sigma_p^2\), but standard deviation \(|F_{ij}|\sigma_p\). The variance has squared observation units; the standard deviation has the original units.
• Pitfall: interpreting conditional residual variance as total variability. The function \(V(F_{ij})\) holds subject and occasion effects fixed. Variation in predictions contributes additional marginal variability.

The derivations identify what each observation model entails; they do not establish which model fits a given assay or data set. A local approximation can be poor when the transformed-scale error is large or the inverse transformation is strongly curved over the relevant range. In those situations, use the exact variance when available or evaluate the observation model directly rather than treating the delta-method expression as exact.