Worked Selection of a Residual Error Model

Why Select a Residual-Error Model?

A residual-error model describes how observed concentrations vary around the model-predicted concentrations after accounting for the structural model and the modeled subject or occasion effects. Its choice affects both the fit assigned to the data and the variability predicted for a future observation. A constant absolute spread may be reasonable near the assay’s lower range, while variation that grows with concentration may be more appropriate elsewhere.

Selection should not be based on a single fit statistic. A useful comparison combines likelihood-based fit summaries with residual patterns and the predictive implications of each candidate. The examples below use hypothetical concentration data and assume that each model converged successfully.

Definition: Let \(Y_{ij}\) be the observed concentration for subject \(i\) at observation \(j\), and let \(f_{ij}\) be its model-predicted concentration. The residual is \(r_{ij}=Y_{ij}-f_{ij}\). An additive error model has \(\operatorname{Var}(Y_{ij}\mid f_{ij})=\sigma_a^2\); a proportional model has \(\operatorname{Var}(Y_{ij}\mid f_{ij})=\sigma_p^2 f_{ij}^2\); and a combined model has \(\operatorname{Var}(Y_{ij}\mid f_{ij})=\sigma_a^2+\sigma_p^2 f_{ij}^2\). Here \(\sigma_a\) has concentration units and \(\sigma_p\) is dimensionless.

In the usual Gaussian versions, the corresponding conditional standard deviations are \(\sigma_a\), \(\sigma_p f_{ij}\), and \(\sqrt{\sigma_a^2+\sigma_p^2 f_{ij}^2}\), respectively. The proportional form presumes nonnegative predictions; a negative prediction would not give a meaningful proportional standard deviation. The combined form allows an absolute component and a concentration-dependent component to contribute at the same time.

These models describe residual variation, not structural misspecification. If predictions systematically miss the time course, changing the residual variance can make the fit statistic look better without fixing the underlying mean-model problem. The structural model, random-effects structure, included observations, and likelihood-fitting method should therefore be held constant when the aim is to compare residual-error structures.

What the Likelihood Comparison Measures

For a Gaussian residual model, the residual contribution to the negative twice log likelihood, at fixed predictions and variance parameters, has the form below. In a nonlinear mixed-effects fit, the reported likelihood also depends on how subject-level random effects are handled; the expression here isolates the role of the residual variance.

-2\ell_{\text{res}}=\sum_{i,j}\left[\log\{2\pi V_{ij}\}+\frac{r_{ij}^2}{V_{ij}}\right]

Here \(V_{ij}\) is the variance specified by the candidate error model. The squared residual is scaled by its variance, while the logarithm penalizes assigning a large variance everywhere. Thus, the best-fitting variance structure balances standardized residual size against the variance required to accommodate the data; it does not simply reward the model that makes every residual appear small.

For models fit by maximum likelihood to the same observations using the same likelihood basis, a smaller \(-2\ell\) indicates a higher maximized likelihood. A common fit-and-complexity summary is the Akaike information criterion, \(AIC=-2\ell+2k\), where \(k\) is the number of estimated parameters in the fitted model. Lower AIC is preferred among the candidates being compared, but an AIC difference is not a p-value and does not establish that one model is true.

Worked Example: Comparing Likelihood and AIC

Suppose 24 hypothetical concentrations are fit with the same structural and random-effects model under three residual structures. The reported maximized log likelihoods are \(-81.2\) for additive error, \(-74.5\) for proportional error, and \(-71.8\) for combined error. The first two fits each estimate seven parameters in total; the combined fit estimates eight.

Calculate \(-2\ell\): additive gives \(-2(-81.2)=162.4\); proportional gives \(-2(-74.5)=149.0\); combined gives \(-2(-71.8)=143.6\). The combined model has the highest maximized likelihood of the three.

Now calculate AIC. Additive: \(162.4+2(7)=176.4\). Proportional: \(149.0+2(7)=163.0\). Combined: \(143.6+2(8)=159.6\). In this hypothetical comparison, combined error has the lowest AIC, by 3.4 units relative to proportional error and 16.8 relative to additive error. This supports the combined model among these candidates, provided its residual diagnostics and predictive behavior are also acceptable.

The combined model is a more flexible extension of either one-component variance model: setting one component to zero recovers the other form. Because a variance component’s null value is on the boundary of its allowed range, a routine likelihood-ratio test using an ordinary one-degree-of-freedom chi-square reference distribution should not be applied automatically. AIC avoids that particular reference-distribution assumption, though it does not remove the need for diagnostics or validation.

Check Residual Spread Across Concentrations

A useful diagnostic is the residual divided by the model’s residual standard deviation. For a candidate with variance \(V_{ij}\), define the standardized residual as \(r_{ij}/\sqrt{V_{ij}}\). If the variance function is suitable and the Gaussian and independence assumptions are reasonable, these residuals should not show a systematic change in spread over predicted concentration. They should also be examined against time and other relevant predictors, since a concentration-only pattern may not reveal all problems.

The raw residual plot is still informative, but its expected spread differs under the candidate models. Under additive error, the spread should be roughly constant in concentration units. Under proportional error, it should increase with predicted concentration, while the coefficient of variation (residual standard deviation divided by the mean prediction) is approximately constant. Combined error permits a transition: the additive term can dominate at low concentrations and the proportional term at higher concentrations.

Worked Example: Reading Residuals at Low and High Concentrations

In a hypothetical fit, the additive estimate is \(\sigma_a=0.30\) mg/L and the proportional estimate is \(\sigma_p=0.12\). Consider one observation with prediction \(0.50\) mg/L and residual \(+0.28\) mg/L, and another with prediction \(10.0\) mg/L and residual \(+1.10\) mg/L. These two observations illustrate the scaling calculations; by themselves, they are not enough to select a model.

At \(0.50\) mg/L, additive error gives standardized residual \(0.28/0.30=0.933\). Proportional error has standard deviation \(0.12(0.50)=0.060\) mg/L, so its standardized residual is \(0.28/0.060=4.67\). Combined error has standard deviation \(\sqrt{0.30^2+0.060^2}=\sqrt{0.0936}=0.306\) mg/L and standardized residual \(0.28/0.306=0.915\).

At \(10.0\) mg/L, additive error gives \(1.10/0.30=3.67\). Proportional error has standard deviation \(0.12(10.0)=1.20\) mg/L, giving \(1.10/1.20=0.917\). Combined error has standard deviation \(\sqrt{0.30^2+1.20^2}=\sqrt{1.53}=1.237\) mg/L and standardized residual \(1.10/1.237=0.889\).

The example shows why a single raw residual can look large or small depending on predicted concentration and variance model. A convincing assessment uses all observations: plot raw and standardized residuals against predictions, inspect time-related patterns, and consider whether residual spread is roughly stable after scaling by the candidate standard deviation.

Predictive Consequences and a Selection Workflow

The residual model determines the spread around a predicted concentration, not just the fit score. For a Gaussian observation model, an approximate 95% observation range at a fixed prediction is \(f_{ij}\pm1.96\sqrt{V_{ij}}\). This is an observation-level range under the assumed model, not a confidence interval for a population parameter. It does not include additional uncertainty in parameter estimates unless that uncertainty is separately incorporated.

Worked Example: Comparing Observation Ranges

Suppose the fitted residual parameters are again \(\sigma_a=0.30\) mg/L and \(\sigma_p=0.12\), and consider a prediction of \(5.0\) mg/L. The additive standard deviation is \(0.30\) mg/L, giving the approximate range \(5.0\pm1.96(0.30)=5.0\pm0.588\), or 4.412 to 5.588 mg/L.

The proportional standard deviation is \(0.12(5.0)=0.60\) mg/L, giving \(5.0\pm1.96(0.60)=5.0\pm1.176\), or 3.824 to 6.176 mg/L. The combined standard deviation is \(\sqrt{0.30^2+0.60^2}=\sqrt{0.45}=0.671\) mg/L, giving \(5.0\pm1.96(0.671)=5.0\pm1.315\), or approximately 3.685 to 6.315 mg/L.

The combined model predicts the widest range at this concentration because both variance components contribute. Which range is appropriate depends on the supported error structure, not on a preference for wider or narrower predictions. Each range is conditional on the stated prediction and fitted variance parameters, and the Gaussian model can assign some probability to negative concentrations, especially near zero.

Set up a fair comparison.
Use the same observations, structural model, random-effects specification, likelihood method, and estimation settings. Verify that each fit converged and that the likelihoods are comparable.

Compare fit with complexity in view.
Review maximized likelihood and AIC, counting all estimated parameters consistently. Treat the ranking as evidence among candidate models, not as proof that the lowest-AIC model is correct.

Inspect scaled residual behavior.
Check raw and standardized residuals against predictions and time. Look for concentration-dependent spread, systematic trends, outliers, and unexplained correlation.

Assess predictive suitability.
Check that the selected variance function gives plausible observation variability across the concentration range that matters for the intended predictions.

The main alternatives have different strengths and limits. Additive error is simple and can represent a roughly fixed absolute assay or residual spread, but may understate variation at high concentrations. Proportional error represents approximately constant relative variability, but can become implausibly narrow as predictions approach zero. Combined error can accommodate both patterns, at the cost of an extra parameter and potential overfitting when data are sparse or cover a narrow concentration range.

A Gaussian additive component can also imply negative observations, and even a combined Gaussian model does not guarantee positivity. If that implication is important, the assumed observation distribution may need reconsideration; selecting the combined form alone does not solve it. Likewise, a residual model cannot repair serial correlation, an incorrect structural time course, or unmodeled occasion effects. Apparent excess residual spread may signal those issues rather than an inadequate variance function.

Common Mistakes and Failure Modes

Choosing solely by the largest likelihood. More flexible models generally can fit at least as well. Compare a complexity-adjusted criterion such as AIC, then check residuals and predictions.
Comparing non-comparable fits. Likelihood values from different observations or different likelihood approximations do not provide a fair direct ranking. Keep the analysis basis consistent.
Assuming combined error must be best. It can estimate a component near zero or be weakly identified. A lower training likelihood alone does not establish useful predictive improvement.
Checking only raw residuals. Raw residual spread is expected to vary under proportional error. Examine standardized residuals as well as raw residuals.
Using residual error to hide a mean-model problem. Time trends or systematic residual signs suggest that the predicted concentration profile may be inadequate, not merely that its variance function needs another component.

Key takeaway: Select a residual-error model using comparable likelihood-based fits, concentration-scaled residual diagnostics, and plausible predictive variability. AIC ranks trade-offs among specified candidates; it does not replace model checking or guarantee that the selected model is adequate.

Check Your Understanding

Answer each question, then compare with the solutions below.

At a predicted concentration of 4 mg/L, what residual standard deviation does a proportional model with \(\sigma_p=0.10\) imply?
Two models have \(-2\ell=120\) with \(k=6\), and \(-2\ell=115\) with \(k=9\). Calculate both AIC values and state which is smaller.
Why should standardized residuals be examined against predicted concentration when comparing additive and proportional error?
What is one limitation of interpreting a Gaussian observation range near zero concentration?

Solutions. 1. The standard deviation is \(0.10(4)=0.40\) mg/L. 2. The AIC values are \(120+2(6)=132\) and \(115+2(9)=133\); the first is smaller by 1, so it is preferred by AIC alone, without implying a decisive or definitive choice. 3. The models imply different raw-residual spread across concentration, so scaling residuals by their candidate standard deviations makes departures from the assumed spread easier to assess. 4. The Gaussian distribution can assign probability to negative concentrations, which may be implausible for a concentration measurement.