Why Residual Type Matters
A residual is a difference between an observation and a model prediction, but an NLME analysis can produce more than one relevant prediction for the same observation. A population prediction describes the typical individual at the population parameter values. An individual prediction uses that person's estimated random effects. These fit targets answer different questions, so residuals formed from them need not have the same values.
Weighting adds another distinction. A raw residual is measured in the response's units. A weighted residual scales the discrepancy using a residual-error variance. A conditional weighted residual can also account for dependence among the components of an individual's residual vector through a covariance-based transformation. These quantities are related, but they are not interchangeable.
The model may be written as \(y_{ij}=f_{ij}(\theta,\eta_i)+\epsilon_{ij}\), where \(\theta\) is a vector of population parameters, \(\eta_i\) represents individual random effects, and \(\epsilon_{ij}\) represents residual error. A common assumption is that, conditional on \(\eta_i\) and the model parameters, the residual errors have mean zero and a specified variance or covariance. Residual calculations use fitted values and estimated quantities, however, so they are diagnostic summaries rather than direct observations of the unobserved errors.
Raw Residuals: The Prediction Target Comes First
The population raw residual compares the observation with the prediction at the population level. The individual raw residual compares it with the prediction using the individual's estimated effects. Both remain in the response's units, such as mg/L or hours. Neither is divided by a standard deviation.
The individual prediction is commonly obtained using empirical Bayes estimates (EBEs) or another method for estimating individual effects. It is fitted using the individual's data, so it is not an independent prediction of those same observations. In many settings the individual prediction is closer to the observed data than the population prediction; that closeness alone does not establish that the population model describes the data adequately.
Worked Example: Population and Individual Raw Residuals
Suppose a hypothetical concentration observation is \(13.5\) mg/L. The fitted population prediction is \(10.0\) mg/L, and the individual prediction using the subject's estimated random effects is \(12.0\) mg/L.
The population raw residual is observed minus population prediction:
The individual raw residual is observed minus individual prediction:
Both differences are positive, so both predictions are below the observation. They differ because they use different fit targets, not because one observation has two different measured values. The smaller individual residual says that the individual prediction is closer for this observation; it does not, on its own, evaluate the population fit.
Individual Weighted Residuals
An individual weighted residual (IWRES) scales the individual raw residual using the residual-error standard deviation for that observation, evaluated under the fitted model's convention. For independent residual errors, the usual componentwise form is:
Here \(R_{ij}\) is the conditional residual-error variance, in squared response units. For example, if the residual standard deviation is \(2\) mg/L, dividing a residual in mg/L by \(2\) mg/L gives a dimensionless quantity. With an additive error model, the variance might be constant; with a proportional or combined error model, it can depend on the predicted response. The relevant variance must follow the fitted error model and the software's stated convention.
The IWRES is conditional in the practical sense that the individual prediction and its residual variance are evaluated using estimated individual effects. It is usually computed separately for each observation. In a simple model with conditionally independent residual errors, it does not decorrelate a vector of residuals if those residuals have a non-diagonal covariance structure.
Worked Example: Calculating an Individual Weighted Residual
Use the hypothetical observation and predictions from the previous example. Suppose the fitted additive residual error has a conditional standard deviation of \(2.0\) mg/L at this observation. The individual raw residual is \(1.5\) mg/L, so:
The IWRES is dimensionless. Its positive sign indicates that the observation is above the individual prediction. This calculation assumes that \(2.0\) mg/L is the appropriate conditional residual standard deviation for the fitted model; if the error model makes the variance depend on the prediction, that variance must be evaluated according to that model.
Conditional Weighted Residuals
A conditional weighted residual (CWRES) is formed from the conditional residual vector for an individual and a covariance-based weighting or whitening step. In a general formulation, let \(\mathbf{y}_i\) be the vector of observations for individual \(i\), and let \(\mathbf{f}_i(\hat{\theta},\hat{\eta}_i)\) be the corresponding vector of individual predictions. Define the conditional residual vector as \(\mathbf{r}_i^C=\mathbf{y}_i-\mathbf{f}_i(\hat{\theta},\hat{\eta}_i)\). If the conditional covariance used by the procedure is \(\mathbf{V}_i^C\), and \(\mathbf{V}_i^C=\mathbf{L}_i\mathbf{L}_i^\mathsf{T}\) is a Cholesky factorization, a whitened vector is:
This definition makes explicit that the weighting uses the covariance of a vector, not necessarily just a separate variance for each measurement. The covariance \(\mathbf{V}_i^C\) must be the one specified by the particular conditional-residual procedure. Depending on the software and method, its construction can involve a conditional residual-error covariance and a linearization of the model. Therefore, the label “CWRES” should not be assumed to imply one identical numerical algorithm in every program.
If \(\mathbf{V}_i^C\) is diagonal, the transformation divides each residual by its own conditional standard deviation, and the resulting components match the corresponding componentwise weighted residuals when the same residual vector and variances are used. If the covariance is not diagonal, whitening also changes the components to account for their dependence. A CWRES component can then depend on more than the corresponding raw residual alone.
Worked Example: Whitening a Two-Observation Residual Vector
Suppose a hypothetical individual's conditional residual vector, in concentration units, is \(\mathbf{r}^C=(2,3)^\mathsf{T}\) mg/L. For illustration, suppose the conditional covariance used in this calculation is:
This matrix is positive definite: its first leading principal minor is \(4>0\), and its determinant is \(4(9)-1(1)=35>0\). Its Cholesky factor is:
Solve \(\mathbf{L}\mathbf{z}=\mathbf{r}^C\). The first equation gives \(2z_1=2\), so \(z_1=1\). The second gives \(0.5z_1+\sqrt{8.75}z_2=3\), hence:
Thus the whitened residual vector is approximately \((1.000,0.845)^\mathsf{T}\). Componentwise division by the marginal standard deviations \(2\) and \(3\) would instead give \((1.000,1.000)^\mathsf{T}\). The difference arises from the off-diagonal covariance: whitening accounts for the specified dependence. These values are an illustration of the matrix operation, not a claim that all software uses this exact covariance construction for CWRES.
What the Calculations Assume
The residual quantities are only as appropriate as their predictions, variance model, and conditioning choices. Before interpreting them, check the following:
- Prediction target: Confirm whether the prediction is population-level or individual-level. Do not compare residuals as if the underlying predictions were the same.
- Residual-error model: The variance used for weighting must correspond to the fitted error model and the observation. A variance in squared response units gives a standard deviation in response units.
- Conditional covariance: A covariance-based transformation requires a valid covariance matrix, including symmetry and positive definiteness for the Cholesky calculation shown above.
- Conditioning and estimation: Individual effects are estimated, usually using the individual's observations. This conditioning can make individual-level residuals appear smaller and does not remove uncertainty about the estimated effects.
- Software definition: Record how the program calculates IWRES and CWRES. Procedures with the same label can differ in linearization, covariance construction, or handling of observations.
Derivation: Why the Covariance Transformation Is Called Whitening
Suppose, for this justification, that the conditional residual vector has mean zero and covariance \(\mathbf{V}_i^C\), and that \(\mathbf{V}_i^C=\mathbf{L}_i\mathbf{L}_i^\mathsf{T}\) with nonsingular \(\mathbf{L}_i\). Define \(\mathbf{z}_i=\mathbf{L}_i^{-1}\mathbf{r}_i^C\). Its covariance is:
The transformation therefore gives components with unit covariance under the stated conditional covariance model. If the residual vector is also multivariate normal with mean zero, then the transformed vector is multivariate normal with identity covariance. Unit covariance does not guarantee that a fitted model is correct: the result depends on the assumed and estimated covariance, and fitted residuals themselves are affected by parameter estimation.
Choosing and Interpreting a Residual
| Quantity | Compared with | Scaling | Main distinction |
|---|---|---|---|
| Population raw residual | Population prediction | None | Shows discrepancy from the population fit in response units |
| Individual raw residual | Individual prediction | None | Shows discrepancy after using estimated individual effects |
| IWRES | Individual prediction | Componentwise residual standard deviation | Scales an individual discrepancy; commonly computed observation by observation |
| CWRES | Conditional residual vector | Conditional covariance-based transformation | Can account for dependence among residual-vector components |
Use raw residuals when the discrepancy in original units is the quantity of interest. Use weighted residuals when scaling by the fitted residual error is appropriate. Use CWRES when the intended conditional procedure and its covariance construction are known and relevant. These summaries complement, rather than replace, inspection of model fit at the population and individual levels.
A frequent mistake is to say, “The individual residual is smaller, so the population model fits well.” That reasoning confuses fit targets. The individual prediction uses estimated subject-specific effects and is informed by the individual's data; it can be close even when the population prediction is systematically off. A second mistake is to call IWRES and CWRES synonyms. They may coincide under a diagonal covariance and matched conventions, but covariance-based conditional transformations need not equal componentwise scaling.
Residual diagnostics can also fail when the variance model is misspecified, the conditional covariance is poorly estimated, or the individual-effect estimates are unstable. An extreme standardized residual is not automatically evidence of a single bad observation; it may instead point to a model or estimation issue. Conversely, small residuals do not prove adequacy. Interpret them alongside the prediction target, the fitted error model, and the method used to construct them.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- An observation is \(8.0\) mg/L, its population prediction is \(6.5\) mg/L, and its individual prediction is \(7.0\) mg/L. What are the two raw residuals?
- If the individual raw residual is \(-0.6\) mg/L and the applicable conditional residual standard deviation is \(0.3\) mg/L, calculate the IWRES and interpret its sign.
- When does the covariance-based transformation reduce to componentwise standardization?
- Why does a smaller individual raw residual not, by itself, establish good population-level fit?
Solutions. (1) The population raw residual is \(8.0-6.5=1.5\) mg/L; the individual raw residual is \(8.0-7.0=1.0\) mg/L. (2) \(\mathrm{IWRES}=(-0.6\ \text{mg/L})/(0.3\ \text{mg/L})=-2.00\), a dimensionless negative residual, meaning the observation is below the individual prediction by two conditional residual standard deviations. (3) When the conditional covariance matrix is diagonal and its diagonal entries are the variances used for componentwise standardization, the Cholesky transformation divides each component by its standard deviation. (4) The individual prediction uses estimated individual effects informed by that individual's data, whereas population-level fit concerns predictions at the population parameter values; closeness to the individual prediction does not answer the population-fit question.