Why a Residual-Error Estimate Can Reach a Bound
In a nonlinear mixed-effects model, residual error describes unexplained variation in observations around the model prediction, conditional on the individual’s parameters. If an estimated residual-error parameter reaches its lower or upper bound, the fit has not necessarily discovered that measurement error is exactly zero or that it is truly enormous. The estimate may instead reflect the chosen response scale, an unsuitable error model, a small number of influential observations, or difficulty separating residual variation from interindividual variation.
The useful first question is not “How do I move the estimate away from the bound?” It is “What feature of the data and model is pushing the estimate there?” The checks below address that question without treating a boundary estimate as proof of either a model failure or a valid scientific conclusion.
For individual \(i\) at observation time \(t_{ij}\), write the model as
where \(Y_{ij}\) is the observed response, \(f(t_{ij},\theta,\eta_i)\) is the model prediction, \(\theta\) denotes population parameters, and \(\eta_i\) denotes individual random effects. In the checks in this tutorial, residual errors are assumed to have mean zero conditional on the model prediction and individual effects. The assumed residual distribution and variance must also match the selected error model; conditional independence is commonly assumed unless the model specifies otherwise.
Start With the Error Model and Measurement Scale
With an additive error model, the conditional residual standard deviation is constant on the response scale:
Here \(\sigma_a\) has the same units as \(Y_{ij}\). Under a proportional error model, the standard deviation increases with the prediction:
The proportional parameter \(\sigma_p\) is dimensionless when \(Y_{ij}\) and \(f_{ij}\) use the same units. One common combined model is
Other implementations define combined error differently, so check the exact variance formula used by the software before interpreting its parameters. A residual parameter that is at a bound may be poorly scaled or expressed in unexpected units. For example, converting concentrations from mg/L to µg/L multiplies additive standard deviations by 1,000, while a proportional coefficient is unchanged if both observations and predictions are converted consistently.
Worked Example: Checking a Proportional Error Estimate
Suppose a hypothetical model has predictions of 10 and 20 mg/L, with observed residuals of 0.5 and -1.0 mg/L. As a simple scale check, calculate the root mean square of the relative residuals. This calculation describes these two residuals; it is not a full mixed-effects likelihood estimate.
The relative residuals are \(0.5/10=0.0500\) and \(-1.0/20=-0.0500\). Their root mean square is
The residuals correspond to a relative scale of 5.00% in this small illustration. If a fitted proportional parameter were pinned at a lower bound near zero despite residuals of this size across the data, investigate whether the fitted error model, data scale, or optimization setup matches the calculation. Two residuals alone cannot establish the correct model or estimate.
A log-transformed response requires particular care. Residuals on the log scale are not in the original response units, and a constant standard deviation on the log scale implies a different error pattern on the original scale. Do not compare a log-scale residual parameter directly with an additive error measured in the original units.
Check What the Residuals Say
Residual diagnostics should be considered against the error model that was actually fitted. Under an additive model, residuals should have roughly constant spread across prediction levels. Under a proportional model, relative residuals or residuals scaled by the prediction should have roughly constant spread. A funnel-shaped pattern, curvature, or systematic runs of positive and negative residuals may indicate that the assumed error structure or the mean model does not capture the data pattern.
A useful standardized residual divides a residual by its assumed conditional standard deviation. For example, under the combined variance formula above, this is \(r_{ij}/\sqrt{\sigma_a^2+(\sigma_p f_{ij})^2}\), where \(r_{ij}=Y_{ij}-f_{ij}\). Such diagnostics are meaningful only relative to the assumed model; a good-looking summary cannot establish that all modeling assumptions hold. Examine residuals across prediction level, time, individual, and relevant data-collection features.
Worked Example: How One Observation Can Inflate an Additive Scale Check
Consider six hypothetical residuals, in mg/L, from a model with fixed predictions for this diagnostic: \(0.1,-0.1,0.2,-0.2,0.1,1.0\). The root mean square residual is a descriptive scale check, not a degrees-of-freedom-adjusted estimate from a fitted likelihood.
The sum of squared residuals is
Thus the root mean square is \(\sqrt{1.11/6}=0.4301\) mg/L. The final residual contributes \(1.00/1.11=90.09\%\) of the total squared residual. If that observation is set aside solely to see its influence on this descriptive calculation, the remaining root mean square is
That large change flags the observation for review: check its recorded value, units, time, and relevant data handling, and inspect whether the model systematically misses that region. It does not justify deleting the observation. Any exclusion requires a defensible data or scientific reason, and the model should be refitted and reported transparently if the record is corrected or excluded.
Separate Residual Variation From Interindividual Variation
Both residual error and interindividual random effects can contribute to observed differences. When repeated observations are sparse, or when the sampling design offers little information to distinguish within-individual variation from between-individual variation, the two sources may be difficult to separate. A residual estimate at a lower bound can then occur because the model attributes much of the variation to individual effects; a large residual estimate can occur when the fitted random-effects structure does not explain observed differences.
A simple within-individual contrast illustrates what repeated observations can contribute. Suppose each individual has two observations with the same expected prediction after accounting for the model. Subtracting the individual mean removes a constant individual-level offset, leaving information about within-individual residual variation. This is a diagnostic simplification, not a replacement for fitting the NLME likelihood: nonlinear predictions, changing times, and random effects may make the real calculation more involved.
Worked Example: A Within-Individual Contrast Check
Four hypothetical individuals each have two observations after their individual means have been removed. Suppose the eight resulting deviations are \(0.2,-0.2,0.2,-0.2,0.2,-0.2,0.2,-0.2\) response units. The sum of squared deviations is \(8(0.2^2)=0.32\).
Each pair has one estimated individual mean, so the eight deviations have \(8-4=4\) within-individual degrees of freedom. The pooled within-individual variance estimate is therefore
response units. The divisor is four, not eight, because estimating each of the four individual means uses one degree of freedom. This calculation is an unbiased within-individual diagnostic under the simplified equal-mean, independent, constant-variance setup. It is not the residual-variance estimate from a general NLME fit.
For the same illustration, suppose the four individual means are \(1,2,3,4\), with an overall mean of 2.5. Their sample variance is
If each mean averages two independent observations, a simple method-of-moments subtraction gives an illustrative between-individual variance estimate:
response units. This illustrates how within-individual error can be separated from variation in individual means under restrictive assumptions. The subtraction is not a general estimator for nonlinear mixed-effects models; it is a check on whether the observed within-individual variation appears compatible with a residual estimate near a bound.
A Practical Investigation
Before changing a model, confirm the boundary itself: identify the parameterization, lower and upper limits, response units, and whether the reported value is exactly at the constraint or merely close after rounding. Then compare the residual pattern with the selected error structure and look for influential observations or records with inconsistent units. Finally, examine whether the data contain enough within-individual information to distinguish residual variability from random-effect variation.
Check the response transformation, units, error formula, and whether the constrained parameter is a standard deviation, variance, or coefficient of variation.
Use residuals on the appropriate scale and examine them against predictions, time, and individual. Look for changing spread or systematic structure.
Check data entry, units, and observation timing. Quantify how much unusual records affect a scale summary, but do not remove valid observations merely to change the estimate.
Ask whether repeated observations provide within-individual information and whether the assumed random effects plausibly account for between-individual variation.
Where scientifically justified, compare additive, proportional, or combined error structures. Check whether conclusions and diagnostics are stable, rather than choosing a structure only because its estimate is away from a bound.
Comparisons, Limitations, and Common Pitfalls
| Approach | What it assumes | When it may be informative | Important limitation |
|---|---|---|---|
| Additive error | Roughly constant residual spread on the response scale | Absolute deviations are similar across prediction levels | May fit poorly when variability grows with the prediction |
| Proportional error | Residual spread changes in proportion to the prediction | Relative deviations are more stable than absolute deviations | Can be unsuitable near zero predictions or when absolute noise matters |
| Combined error | Both an additive component and a prediction-dependent component contribute, under the specified variance formula | Data show a nonzero baseline spread as well as increasing spread | More parameters can be weakly informed by limited data |
These comparisons are diagnostic, not a rule to select the most flexible model. A combined structure cannot repair an incorrect mean model, erroneous records, or a mismatch between the model and the measurement process. Nor does a small residual estimate prove that the observations are nearly error-free: the estimate can be constrained by the lower limit or reflect variation assigned elsewhere in the model.
Common mistake: “The residual estimate is at its lower bound, so the data have no measurement error; I should fix it at zero.” The correction is to treat the boundary as a result that needs investigation. Check whether residuals remain relative to predictions, whether the scale and variance formula are appropriate, and whether random effects or the mean model explain variation that the residual term would otherwise capture. A bound is a constraint on an estimate, not direct evidence that the corresponding scientific quantity equals the bound.
A second pitfall is changing units or the error model simply to make the optimizer return an interior value. Rescaling can change the numerical value of an additive parameter without changing the underlying fit, and a different error model changes the assumptions. Make such changes only when justified by the measurement scale and residual behavior, then evaluate the resulting fit and document the rationale.
Check Your Understanding
Answer each question, then compare your reasoning with the brief solutions.
- An additive residual standard deviation is estimated in µg/L, but the response was fitted in mg/L. What should you verify before interpreting the estimate?
- A proportional-error fit has residuals that grow in absolute size with predictions but remain similar as fractions of predictions. Which error structure is more consistent with that pattern, and what is one caveat?
- Why do eight within-individual deviations from four estimated individual means have four, rather than eight, degrees of freedom?
- A lower-bound estimate occurs after one observation with a very large residual is found to have been recorded in the wrong units. What is an appropriate next step, and what should not be done automatically?
Solutions. (1) Verify the fitted response units and the parameter’s units: an additive standard deviation has the same units as the response, and 1 mg/L equals 1,000 µg/L. (2) A proportional structure is more consistent because the relative spread is stable; however, inspect the full residual pattern and confirm that proportional error is scientifically appropriate, especially near zero predictions. (3) Estimating each of four individual means uses one degree of freedom, leaving \(8-4=4\) within-individual degrees of freedom. (4) Correct the record using the source information, document the correction, and refit and reassess diagnostics. Do not delete the observation automatically or conclude that the bound is resolved without checking the refitted model.