Why the Error Model Changes the Diagnostic
A residual is the difference between an observation and a model prediction. Its size is meaningful only in relation to the observation-error model: a difference of 2 concentration units may be large when the expected measurement error is 1 unit, but modest when the expected error is 10 units. In an NLME model, additive, proportional, and combined error models make different predictions about how that error size changes with the predicted value.
This tutorial focuses on that prediction-dependent spread. A residual-versus-prediction plot is not expected to have the same vertical scatter under all three error models. The useful question is not simply whether the raw residuals have constant spread, but whether their spread changes in the way the assumed variance function predicts. Standardized residuals help make that comparison, provided the prediction and variance used are appropriate.
For the variance comparisons below, assume that the conditional mean is correctly specified, the observation errors have conditional mean zero, and the stated variance function is correct. Unless the model explicitly includes dependence, observation errors are also assumed conditionally independent. These are assumptions about the observation-error component; they do not guarantee that the structural model, random-effects model, or predictions are adequate.
Three Variance Forms and Their Expected Patterns
Write \(m\) for a particular predicted value. An additive model assigns a constant standard deviation \(\sigma_{\text{add}}\), in the units of the observation. A proportional model assigns a standard deviation proportional to the magnitude of the prediction, with dimensionless coefficient \(\sigma_{\text{prop}}\). A combined model includes both contributions.
In the combined model, the additive and proportional error components are taken to have mean zero and to be independent of one another, with variances \(\sigma_{\text{add}}^2\) and \(\sigma_{\text{prop}}^2\), respectively. Their variances therefore add. If a software implementation uses a different parameterization, check its definition before interpreting parameter estimates or residuals.
These assumptions give distinct expectations for a plot of raw residuals against prediction. Additive error implies roughly constant vertical spread. Proportional error implies spread that grows in proportion to \(|m|\); its coefficient of variation is constant when \(m\) is positive. Combined error implies a floor in the spread at low predictions and increasing spread at higher predictions. Under any of the three models, the conditional mean residual remains zero if the conditional mean is correctly specified.
| Error form | Assumed residual SD at prediction \(m\) | Expected raw-residual spread |
|---|---|---|
| Additive | \(\sigma_{\text{add}}\) | Approximately constant across predictions |
| Proportional | \(\sigma_{\text{prop}}|m|\) | Increases in proportion to the magnitude of the prediction |
| Combined | \(\sqrt{\sigma_{\text{add}}^2+\sigma_{\text{prop}}^2m^2}\) | Nonzero low-prediction floor, with increasing spread at larger predictions |
For positive predictions, proportional error can also be understood through relative residuals: \(e_{ij}/m_{ij}\) has standard deviation \(\sigma_{\text{prop}}\) under the proportional model. This ratio is not useful near zero, where dividing by a small prediction magnifies noise. Combined error does not have constant relative spread across all predictions: the additive component matters most at low predictions, while the proportional component can dominate at high predictions.
Standardizing Residuals to Check the Assumed Spread
If the assumed error model is appropriate, dividing each raw residual by its model-assumed standard deviation should remove the prediction-dependent change in spread. The standardized residual should have conditional mean zero and variance approximately one under the stated model. “Approximately” matters: parameters and predictions are estimated from data, and the residuals are not generally independent simply because they have been standardized.
A useful diagnostic sequence is to examine raw residuals against predictions, then examine standardized residuals against predictions. Raw residual spread is compared with the shape implied by the variance function. Standardized residual spread is checked for remaining prediction dependence. Binned summaries of residual spread can make broad trends easier to see, especially when many points overlap; they do not replace looking at the individual observations and the sampling design.
Use a clearly specified prediction and form \(e_{ij}=y_{ij}-m_{ij}\). Interpret the residual conditionally: observation-error variance is defined around the conditional mean given the individual parameters.
For example, the combined model uses \(s_{ij}^2=\sigma_{\text{add}}^2+\sigma_{\text{prop}}^2m_{ij}^2\). This gives a concrete spread pattern to assess rather than an expectation of constant raw spread in every model.
Check whether raw spread changes as predicted, and whether dividing by \(s_{ij}\) leaves substantial spread trends. A funnel in raw residuals can be expected under proportional error; a funnel remaining in standardized residuals is more concerning.
Worked Calculations
Worked Example: Constant Spread Under Additive Error
Suppose hypothetical observations have predictions of 10, 20, and 40 mg/L. The fitted additive error standard deviation is \(\sigma_{\text{add}}=2\) mg/L. At these predictions, observed residuals are 1, \(-2\), and 3 mg/L. What spread does the model predict, and what are the standardized residuals?
The additive model assigns the same standard deviation at every prediction: \(s=2\) mg/L. Thus, the expected raw-residual spread is constant, even though the predicted concentrations differ. Standardizing gives \(1/2=0.50\), \(-2/2=-1.00\), and \(3/2=1.50\). These are in units of standard deviations and have no concentration units.
The three standardized residuals do not establish whether the variance model is adequate; the sample is far too small for that conclusion. They illustrate the calculation. If a larger residual-versus-prediction display showed raw spread increasing systematically with prediction, that pattern would not match the constant-spread implication of this additive model.
Worked Example: Proportional Error at Different Predictions
Consider hypothetical predictions of 5, 20, and 50 mg/L under a proportional error model with \(\sigma_{\text{prop}}=0.15\). Suppose the corresponding residuals are 1.5, 3.0, and \(-7.5\) mg/L. Calculate the model-assumed standard deviations and standardized residuals.
The standard deviation is \(0.15|m|\). At 5 mg/L it is \(0.15(5)=0.75\) mg/L; at 20 mg/L it is \(3.00\) mg/L; and at 50 mg/L it is \(7.50\) mg/L. The standardized residuals are therefore \(1.5/0.75=2.00\), \(3.0/3.00=1.00\), and \(-7.5/7.50=-1.00\).
The raw residuals have a larger absolute scale at larger predictions, as this model allows. The residual of 1.5 mg/L at a prediction of 5 mg/L is relatively large for the assumed model, while \(-7.5\) mg/L at 50 mg/L is one model-assumed standard deviation below the prediction. On the relative scale, the residuals are \(1.5/5=0.30\), \(3/20=0.15\), and \(-7.5/50=-0.15\); their magnitudes are not identical, but the proportional model specifies their variability, not that every observed ratio must be equal.
Worked Example: The Crossover in a Combined Error Model
Suppose hypothetical concentration data use a combined error model with \(\sigma_{\text{add}}=1.0\) mg/L and \(\sigma_{\text{prop}}=0.10\). For predictions of 10, 20, and 40 mg/L, the observed residuals are 1, \(-2\), and 4 mg/L. Calculate the standard deviations and standardized residuals, and identify where the two variance contributions are equal.
At a prediction of 10 mg/L, the variance is \(1.0^2+(0.10)^2(10)^2=1+1=2\;(\text{mg/L})^2\), so the standard deviation is \(\sqrt{2}=1.414\) mg/L. The standardized residual is \(1/1.414=0.707\).
At 20 mg/L, the variance is \(1+0.01(400)=5\;(\text{mg/L})^2\), giving a standard deviation of \(\sqrt{5}=2.236\) mg/L and standardized residual \(-2/2.236=-0.894\). At 40 mg/L, the variance is \(1+0.01(1600)=17\;(\text{mg/L})^2\), giving a standard deviation of \(\sqrt{17}=4.123\) mg/L and standardized residual \(4/4.123=0.970\).
The additive and proportional variances are equal when \(\sigma_{\text{add}}^2=\sigma_{\text{prop}}^2m^2\). For positive \(m\), this gives \(m=\sigma_{\text{add}}/\sigma_{\text{prop}}=1.0/0.10=10\) mg/L. Below that prediction the additive variance contribution is larger; above it the proportional contribution is larger. The crossover describes the model's variance components, not a guaranteed change point in individual residuals.
Choosing and Assessing the Variance Form
The main comparison is between a constant-variance additive model, a proportional model whose standard deviation scales with prediction magnitude, and a combined model that allows both a low-prediction floor and increasing spread. Choose among them by asking whether the observed residual patterns are compatible with the proposed variance function, while also considering scientific measurement characteristics and the data available. A visually attractive residual plot alone cannot prove that a variance model is correct.
For a practical assessment, inspect raw residuals against predicted values and use the assumed standard deviation to inspect standardized residuals against prediction. If residuals are binned, compare a spread summary within bins with the corresponding range of predictions; uneven numbers of observations per bin and clustering within individuals can make such summaries unstable. Where possible, examine whether the pattern appears across individuals rather than being driven by a small number of subjects.
The prediction used for a diagnostic matters. Observation-error variance is conditional on the model's conditional mean given individual parameters. A population prediction that does not condition on an individual's random effects is not the same quantity as an individual conditional prediction. Residuals based on different prediction definitions can therefore show different patterns. State which prediction is used, and do not interpret a prediction-based diagnostic as a pure check of observation error when the prediction also reflects other model components.
A pattern left in standardized residuals—such as spread increasing with prediction—suggests that the chosen variance function may not have removed the scale dependence. It can also arise from an incorrect conditional mean, an unsuitable prediction, dependence not represented by the model, influential observations, or limited information in the data. Residual diagnostics identify incompatibility to investigate; they do not, by themselves, identify its cause.
Common Pitfalls and Limitations
- Expecting constant raw spread under every model. That is the additive-error expectation, not the proportional or combined-error expectation. Compare raw spread with the specific variance function being assessed.
- Calling any funnel-shaped raw residual plot a failed fit. A funnel can be expected under proportional error. Check whether the spread follows the assumed prediction dependence and whether standardization removes it.
- Dividing by the wrong standard deviation. For combined error, use \(\sqrt{\sigma_{\text{add}}^2+\sigma_{\text{prop}}^2m^2}\), not the sum of the two standard deviations. The variances add under the stated independence assumption.
- Treating standardized residuals as independent, exact standard normal observations. Standardization targets a common variance under the model. Parameter estimation, repeated observations within individuals, and the use of estimated predictions can affect their distribution and dependence.
- Overinterpreting proportional error near zero. Its assumed standard deviation approaches zero as the prediction approaches zero. Ratios such as \(e/m\) are unstable near zero, and a purely proportional model may not suit measurements with a non-negligible absolute error floor.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- An additive model has \(\sigma_{\text{add}}=3\) mg/L. What standard deviation does it assign at predictions of 4 and 40 mg/L?
- A proportional model has \(\sigma_{\text{prop}}=0.20\). At a prediction of 25 mg/L, an observation is 30 mg/L. Find the raw and standardized residuals.
- A combined model has \(\sigma_{\text{add}}=2\) mg/L and \(\sigma_{\text{prop}}=0.10\). Find the model-assumed standard deviation at a prediction of 10 mg/L.
- A raw residual plot has wider scatter at larger predictions, but standardized residuals show approximately similar spread across predictions. Is that pattern necessarily evidence against the variance model? Explain.
Solutions. (1) The additive standard deviation is constant, so it is 3 mg/L at both predictions. (2) The raw residual is \(30-25=5\) mg/L. The assumed standard deviation is \(0.20(25)=5\) mg/L, so the standardized residual is \(5/5=1.00\). (3) The variance is \(2^2+0.10^2(10)^2=4+1=5\;(\text{mg/L})^2\), giving a standard deviation of \(\sqrt{5}=2.236\) mg/L. (4) No. Increasing raw spread can be expected under a prediction-dependent variance model; approximately stable spread after the model-based standardization is compatible with that expectation, although it does not prove the model is correct.