Defining Epsilon Shrinkage For Residual Diagnostics

Why Residual Shrinkage Matters

An individual residual plot can look reassuringly flat even when it provides limited evidence about whether the model describes each subject well. One reason is that the individual predictions used to calculate residuals are informed by that subject’s own observations. When those predictions adapt to the data, the remaining residuals can have less variability than the model’s conditional residual distribution would predict. This reduction is called epsilon shrinkage, or residual shrinkage.

This is distinct from eta shrinkage. Eta shrinkage concerns the spread of individual empirical Bayes estimates of random effects relative to their population spread. Epsilon shrinkage concerns the spread of residuals after accounting for the model’s conditional predictions and residual error scale. The two can both affect individual-level interpretation, but they describe different quantities.

Definition: Epsilon shrinkage describes how much the observed variability of standardized individual residuals falls below their model-implied conditional reference variability. For residuals standardized to have reference standard deviation 1, the common estimate is \(1-S_r\), where \(S_r\) is their sample standard deviation. The estimate is often reported as a percentage by multiplying by 100%.

The definition requires specifying which residuals are used. In this tutorial, \(r_k\) denotes the \(k\)th standardized residual based on an individual prediction, such as an individual weighted residual, and \(N\) is the total number of residuals in the calculation. If the chosen residual is correctly standardized under its reference model, its target standard deviation is 1. The common estimate is then

\bar{r}=\frac{1}{N}\sum_{k=1}^{N}r_k,\qquad S_r=\sqrt{\frac{\sum_{k=1}^{N}(r_k-\bar{r})^2}{N-1}},\qquad \widehat{S}_{\epsilon}=1-S_r.

More generally, if the reference standard deviation is \(s_{\mathrm{ref}}\), the corresponding relative reduction is \(1-S_r/s_{\mathrm{ref}}\). A negative estimate is possible: it means the observed standard deviation exceeds the reference, not that shrinkage should be forced to zero. Report the residual definition and reference scale alongside the estimate, since different residual types need not have identical properties.

What the Calculation Does—and Does Not—Measure

Epsilon shrinkage is about spread, not average residual bias. The sample mean \(\bar{r}\) should therefore be inspected separately. Residuals can have a small standard deviation around a nonzero mean, or a mean near zero with a small standard deviation; these imply different diagnostic patterns. The usual standard deviation is calculated around the sample mean, so it does not by itself reveal a systematic offset.

The standardized scale matters. With a residual error model whose conditional standard deviation differs across observations, dividing each prediction error by its appropriate conditional standard deviation is necessary before comparing its variability with a reference of 1. Raw residuals measured in concentration units cannot be compared directly with that reference, and their pooled spread may be dominated by observations with larger expected errors.

The estimate is a summary over the residuals included in the calculation. It is not a subject-specific probability, a direct estimate of an individual’s residual variance, or a guarantee that every residual plot is compressed by the same amount. Dependence among residuals, unequal numbers of observations per subject, and the chosen residual construction can all affect the pooled summary.

Worked Example: Calculating Epsilon Shrinkage

Suppose a hypothetical fitted population PK model produces six standardized individual residuals: \(-0.8,-0.4,0,0.2,0.4,0.6\). They are already on a scale whose model-implied reference standard deviation is 1. Calculate the common estimate.

First, their mean is \(\bar{r}=0\). The sum of squared deviations from that mean is \(0.64+0.16+0+0.04+0.16+0.36=1.36\). Thus

S_r=\sqrt{\frac{1.36}{6-1}} =\sqrt{0.272} =0.5215, \qquad \widehat{S}_{\epsilon}=1-0.5215=0.4785.

The estimated epsilon shrinkage is therefore 0.4785, or 47.85%. In this small hypothetical set, the residual standard deviation is about 52.15% of its reference value. This summary suggests substantial compression of the residual spread, but six observations are far too few to treat the estimate as a precise characterization of a larger analysis.

Why Individual Fitting Can Reduce Residual Spread

A simple linear calculation illustrates the mechanism. Suppose \(n\) observations follow \(y=X\beta+\epsilon\), where the errors have mean zero, common variance \(\sigma^2\), and are independent. If the same observations are used to fit the \(p\) coefficients by least squares, the fitted residual vector is \(e=(I-H)y=(I-H)\epsilon\), where \(H\) is the projection matrix onto the columns of \(X\). Since \(H\) is symmetric and idempotent, the residual covariance is

\operatorname{Var}(e)=\sigma^2(I-H), \qquad E\left(\frac{e^\mathsf{T}e}{n}\right) =\frac{\sigma^2}{n}\operatorname{tr}(I-H) =\sigma^2\left(1-\frac{p}{n}\right).

The last equality follows because \(\operatorname{tr}(H)=p\). Fitting the observations uses some directions in the data to estimate the model, leaving less variation in the in-sample residuals. This result is an illustration of the mechanism, not a universal formula for epsilon shrinkage in nonlinear mixed-effects models: nonlinear individual predictions, conditional weighting, and the residual definition all matter.

The same broad idea applies when individual predictions use information from a subject’s observations. Those predictions can move toward the observed concentrations, reducing the deviations left over. Consequently, an individual residual plot may show less scatter than a residual plot based on population predictions. That apparent closeness is partly a consequence of how the individual predictions were obtained; it is not independent confirmation of individual predictive performance.

Worked Example: Why Standardization Changes the Summary

Consider a hypothetical set of four prediction errors, in mg/L, of \(1,-1,2,-2\). Their conditional residual standard deviations are \(2,2,4,4\) mg/L, respectively. Calculate the standardized residuals and their epsilon shrinkage, and explain why the raw errors alone are not the appropriate comparison.

Dividing each error by its corresponding conditional standard deviation gives \(0.5,-0.5,0.5,-0.5\). Their mean is 0, and the sum of squared deviations is \(4(0.5^2)=1\). Therefore,

S_r=\sqrt{\frac{1}{4-1}}=0.5774, \qquad \widehat{S}_{\epsilon}=1-0.5774=0.4226.

The standardized residuals give an estimated shrinkage of 42.26% relative to a reference standard deviation of 1. The raw errors have units of mg/L and combine observations with different expected conditional variability; their standard deviation cannot be compared directly with the standardized reference. Standardization makes the comparison meaningful under the stated conditional scales, although it does not remove dependence or other model problems.

Reading Individual Residual Plots in Context

A residual plot should still be examined for structure: trends with time, concentration, or another relevant predictor; systematic offsets; and unusual tails or clusters. Shrinkage does not prove that any such pattern is absent. Rather, it limits how much reassurance should be taken from a visually narrow cloud of individual residuals. A low-variability cloud can arise because the individual predictions have adapted to the same data being plotted.

This makes it useful to report the residual type, its standardization, the observations included, and the estimated epsilon shrinkage. When comparing residual plots based on population predictions with plots based on individual predictions, remember that they answer different questions. Population-prediction residuals show deviations from the population-level prediction; individual-prediction residuals show deviations after subject-specific information has been used. Neither plot alone settles all aspects of model adequacy.

Worked Example: A Quiet Plot with Compressed Residuals

In a hypothetical analysis, eight standardized individual residuals are \(-0.9,-0.7,-0.5,-0.3,0.3,0.5,0.7,0.9\). They are centered on zero and show no visible trend against time in the plotted observations. Calculate the shrinkage estimate and state what the plot supports.

The sum of squared residuals is \(2(0.9^2+0.7^2+0.5^2+0.3^2)=2(0.81+0.49+0.25+0.09)=3.28\). Since the mean is zero,

S_r=\sqrt{\frac{3.28}{8-1}} =\sqrt{0.4686} =0.6845, \qquad \widehat{S}_{\epsilon}=1-0.6845=0.3155.

The estimated shrinkage is 31.55%. The plotted residuals are centered and show no evident time trend, which is useful descriptive information. However, their standard deviation is only about 68.45% of the reference value, so their narrow spread is not, by itself, strong evidence that individual predictions would perform equally well on new observations. The calculation qualifies the interpretation; it does not erase the plot or establish that the model is misspecified.

Assumptions, Alternatives, and Failure Modes

The common estimate has a clear interpretation when the residuals are standardized appropriately, the reference standard deviation is known to be 1, and the set of residuals represents the diagnostic quantity of interest. Its numerical value is descriptive; a conventional uncertainty interval or universal threshold does not follow from the definition alone. In particular, the residuals may be correlated within a subject, so treating all \(N\) residuals as independent can overstate the information available.

A main alternative is to inspect residuals based on population predictions rather than individual predictions. These are not interchangeable estimates of epsilon shrinkage: the prediction used to construct a residual changes what that residual diagnoses. Another alternative is to use a different standardized residual, such as a conditional weighted residual, with its own construction and reference behavior. If the residual’s expected standard deviation is not 1, use its appropriate reference scale rather than applying \(1-S_r\) mechanically.

Common warning signs include applying the formula to unstandardized prediction errors, mixing residual definitions in one calculation, or interpreting a low value of \(S_r\) as evidence of a small mean bias. Check the residual definition and scale, examine the residual mean and relevant patterns separately, and consider whether repeated observations contribute unequal amounts to the pooled result. These checks clarify what the summary can support; they do not turn one shrinkage number into a complete goodness-of-fit assessment.

Key takeaway: Epsilon shrinkage is a reduction in standardized individual residual spread relative to its conditional reference. Calculate it only after identifying the residual and reference scale, and interpret a narrow individual residual plot with the possibility of data-adaptive individual predictions in mind.

Check Your Understanding

Try each question, then compare your reasoning with the solutions below.

Standardized residuals have sample standard deviation 0.70 and reference standard deviation 1. What is the common epsilon-shrinkage estimate?
Why should residual mean and residual standard deviation be considered separately?
A prediction error is 3 mg/L and its conditional residual standard deviation is 2 mg/L. What is its standardized residual?
Does a narrow individual residual plot prove that individual predictions will be accurate for new observations? Explain.

Solutions. (1) \(1-0.70=0.30\), or 30% estimated epsilon shrinkage. (2) The standard deviation measures spread around the sample mean, so a small spread does not rule out a nonzero mean; mean bias and variability describe different features. (3) The standardized residual is \(3/2=1.5\), dimensionless. (4) No. Individual predictions may use the same subject observations that produce the plotted residuals, reducing their spread; the plot should be interpreted with the residual definition, shrinkage, and other diagnostics in view.