Why Conditional Residuals Can Be Reassuring for the Wrong Reason
Residual plots are often used to look for patterns that a model has failed to explain: trends over time, curvature against prediction, or changing variability. In a nonlinear mixed-effects model, however, a conditional residual is calculated after using each subject’s observations to estimate that subject’s random effects. The observations help determine the individual fit, and are then compared with it. This reuse can pull conditional residuals toward zero and make some misspecification harder to see.
Epsilon shrinkage summarizes how much the spread of standardized conditional residuals has contracted relative to the unit spread expected on their standardized scale. It is a warning about the diagnostic information in conditional residuals, not a correction that restores information lost through fitting. The amount of compression depends on the model, the subject’s observations, and how the residuals are defined and standardized.
This summary is most directly interpretable when the residuals are standardized so that unit spread is the relevant reference under the fitted model. It does not imply that every residual was multiplied by the same factor. Conditional residuals can be correlated, have unequal variances, and respond differently to the random-effect directions used in the model. In particular, a small pooled spread can coexist with an important residual pattern in a particular time range or subgroup.
How Individual Fitting Contracts Residual Components
The mechanism is clearest in a Gaussian linear mixed-effects model. For one subject, write the data after subtracting the population mean prediction as \(d=y-\mu\), where \(y\) is the observation vector and \(\mu\) is the population mean vector. Suppose \(d=Zb+e\), with \(b\sim N(0,G)\), \(e\sim N(0,R)\), and independent \(b\) and \(e\). Here \(Z\) maps random effects to observations, \(G\) is their covariance matrix, and \(R\) is the residual covariance matrix. Let \(V=ZGZ'+R\). Under these Gaussian assumptions, the conditional mean of the random effects is \(GZ'V^{-1}d\). The conditional residual vector is the observed deviation minus the fitted random-effect contribution.
The last equality follows because \(V-ZGZ'=R\), so \(I-ZGZ'V^{-1}=(V-ZGZ')V^{-1}=RV^{-1}\). Thus, components of the data deviation that the random effects can explain are partly used in \(\widehat b\), rather than left in \(r\). Under the same assumptions, the covariance of the conditional residual vector is
where \(R\) and \(V\) are symmetric. The residual covariance is therefore not generally \(R\). This derivation concerns the Gaussian linear model with known covariance parameters; in a nonlinear model, a local linear approximation may explain the same mechanism, but the exact conditional fitting and residual calculation can be more complex.
Worked Example: A Random Intercept Compresses the Subject Mean
Consider hypothetical data from one subject at four occasions. After subtracting the population predictions, the deviations are \(d=(3,1,1,3)'\). Assume a random intercept with variance \(\tau^2=4\), independent residual errors with variance \(\sigma^2=1\), and no other random effects. These assumptions make the random-intercept Gaussian calculation applicable.
For four observations, the posterior mean of the random intercept is \(\widehat b=\frac{4\tau^2}{\sigma^2+4\tau^2}\bar d\). The mean deviation is \(\bar d=(3+1+1+3)/4=2\), so
The conditional residuals subtract this same fitted intercept from each deviation: \(r=d-\widehat b=(19/17,-15/17,-15/17,19/17)'\), or approximately \((1.118,-0.8824,-0.8824,1.118)'\). The subject mean of the residuals is \(2-32/17=2/17\), compared with the original mean deviation of 2. The mean-direction component has been compressed to \(1/17\) of its original size, even though the within-subject contrasts are unchanged by this random-intercept fit.
This distinction matters for plots. A random intercept can absorb a subject-level offset, but it cannot by itself absorb a centered within-subject time trend. A random slope or another flexible individual parameter may, however, absorb a time pattern aligned with its fitted direction. Therefore, epsilon shrinkage is a clue about diagnostic sensitivity, not proof that every time pattern has been erased.
Calculating and Interpreting Epsilon Shrinkage
Worked Example: Estimating Epsilon Shrinkage From Standardized Residuals
Suppose eight hypothetical pooled standardized conditional residuals are \((-0.4,0.4,-0.6,0.6,-0.5,0.5,-0.3,0.3)\). Their mean is zero. Using the sample standard deviation, the sum of squared deviations is \(0.16+0.16+0.36+0.36+0.25+0.25+0.09+0.09=1.72\). Hence
On the stated convention, the pooled standardized conditional residuals have about half the unit reference standard deviation, giving an estimated epsilon shrinkage of about 0.5043. This does not mean that every residual is exactly half its unshrunk value. With only eight residuals, the estimate is also imprecise and potentially affected by their dependence and by the model’s standardization.
When interpreting a reported value, check which residuals were used, whether they were standardized, whether residuals were pooled across subjects, and whether the standard deviation used a sample or population denominator. A value above 1 can occur if the reported standard deviation exceeds 1; it signals dispersion above the reference scale, not “negative physical shrinkage.” The summary should not be read in isolation from plots and the residual construction.
When a Residual Plot Conceals a Time Pattern
Worked Example: A Random Slope Absorbs an Omitted Time Pattern
Consider a hypothetical subject measured at four occasions. Let the centered time-direction vector be \(q=(-1,-1,1,1)'\), and suppose a systematic, unmodeled time pattern contributes deviations \(d=2q=(-2,-2,2,2)'\). For illustration, assume the fitted model includes a random slope in direction \(q\), with prior variance \(\tau^2=4\), and independent residual variance \(\sigma^2=1\) per observation. The data pattern is entirely aligned with that random-effect direction.
Because \(q'q=4\), the conditional mean of the random-slope coefficient is \(\widehat b=\frac{\tau^2 q'd}{\sigma^2+\tau^2q'q}\). Here \(q'd=2q'q=8\), so
The conditional residual vector is \(r=d-q\widehat b=(2/17)q\), or approximately \((-0.1176,-0.1176,0.1176,0.1176)'\). Its systematic signal is only \(1/17\) of the original signal. The original squared signal size is \(d'd=16\); the remaining squared signal size is \((2/17)^2q'q=16/289\approx0.05536\). Thus a conditional residual-versus-time plot could show a very small version of this pattern, even though the fitted random slope has absorbed a systematic time effect. This example isolates the signal component and does not include random observation noise.
The plot alone cannot establish whether the time pattern represents genuine between-subject random slopes, a missing population time function, or another misspecification. That distinction requires checking the model and the data-generating design, not treating small conditional residuals as confirmation of adequacy.
Residual-versus-prediction plots have a related limitation. Conditional predictions use subject-specific estimates informed by the same observations whose residuals are plotted. This can make residuals appear close to zero and can weaken visible association with the fitted values. It does not follow that every such plot is misleading: a misspecification not aligned with the fitted random-effect structure may remain visible, and a well-designed residual calculation may account for some covariance. The relevant question is what information the conditional fit has used and what structure the plot can still reveal.
Practical Diagnostics and Alternatives
A useful diagnostic sequence makes the conditional fitting explicit rather than treating a near-zero residual spread as a pass/fail check.
Record whether it is raw conditional, weighted, or standardized conditional residual, and how individual effects entered its calculation.
Compare the spread of standardized conditional residuals with its reference scale. Treat substantial epsilon shrinkage as a reason to be cautious, not as a correction factor.
Plot residuals against time and prediction, and also examine population-level or marginal discrepancies and subject-level profiles. Look for systematic patterns that may be absorbed by fitted random effects.
Ask whether random intercepts, slopes, or flexible individual parameters overlap with the time or prediction direction under inspection.
Assess population predictions and, where suitable, simulation-based diagnostics. Agreement across checks is more informative than a clean conditional residual plot alone.
These approaches answer related but different questions. Conditional residuals assess what remains after individual fitting and can be useful for within-subject lack of fit, but shrinkage can reduce their sensitivity. Marginal or population-level residual comparisons do not condition on each subject’s empirical Bayes estimate in the same way; they preserve population discrepancies but may combine those with unexplained individual variation. Simulation-based checks compare observed features with features generated under the fitted model, offering a broader reference but depending on adequate model simulation and appropriately chosen summaries.
| Diagnostic | What it emphasizes | Main caution |
|---|---|---|
| Conditional residual plot | Deviations after fitting individual random effects | May look compressed when data strongly inform individual estimates |
| Population or marginal comparison | Agreement at the population level without relying on each subject’s fitted effect in the same way | Can obscure subject-specific patterns when aggregated |
| Simulation-based diagnostic | Whether observed data features are compatible with model-generated data | Conclusions depend on the model and the chosen diagnostic summaries |
Common Pitfalls and Limits
- Calling small conditional residuals evidence that the model is correct. A flexible individual fit can remove a pattern from the residuals. Check shrinkage and assess population-level fit as well.
- Assuming epsilon shrinkage is a universal multiplier. The contraction varies across directions, subjects, and residual definitions. Do not “inflate” every residual by dividing by \(1-\widehat S_{\epsilon}\).
- Assuming all time trends are hidden. A pattern orthogonal to the fitted random-effect directions may remain. Determine which components the individual model can absorb.
- Comparing unlike residual scales. A raw residual standard deviation cannot be directly interpreted against a unit reference unless the residual has been standardized appropriately.
- Ignoring dependence and small samples. Pooled conditional residuals may be dependent, and a sample standard deviation is uncertain when few observations or subjects contribute.
The Gaussian derivation gives a precise explanation of contraction under stated assumptions; nonlinear conditional fits, estimated variance parameters, sparse sampling, and non-Gaussian error structures can complicate its size and shape. Epsilon shrinkage does not diagnose which model component is wrong, estimate the amount of misspecification, or replace graphical and simulation-based assessment. Its practical value is to qualify the evidence supplied by conditional residuals.
Check Your Understanding
Answer each question before reviewing the solutions.
- In the Gaussian random-intercept example, which part of the deviations is strongly compressed: the subject mean or the within-subject contrasts?
- If the sample standard deviation of standardized conditional residuals is 0.70, what is the common descriptive epsilon-shrinkage estimate?
- Why might a random slope make a conditional residual-versus-time plot look reassuring even when the population time function is misspecified?
- Does an epsilon-shrinkage estimate of 0.50 mean every conditional residual is exactly half its corresponding unshrunk residual?
- Name one complementary diagnostic and one limitation of relying on it alone.
Solutions. (1) The subject mean is strongly compressed; a random intercept subtracts the same fitted value from every observation, leaving within-subject contrasts unchanged. (2) \(1-0.70=0.30\), or 30% on the stated descriptive convention. (3) If the omitted time pattern aligns with the fitted random-slope direction, individual fitting can absorb much of that pattern before residuals are plotted. (4) No. The summary describes pooled spread, not a common observation-by-observation multiplier. (5) A population-level comparison can reveal discrepancies not apparent after individual fitting, but aggregation can obscure subject-specific patterns; a simulation-based check is another option, but it depends on the fitted model and selected summaries.