Residual-Error Variance at a Boundary

Why a Residual-Error Boundary Deserves Attention

A residual-error variance estimate at its lower or upper bound is a diagnostic signal, not a complete explanation. It can indicate that the likelihood favors a value outside the allowed range, that the data contain little information about the parameter, or that another part of the model is accounting for variation that the residual model would otherwise describe. The same numerical result can therefore arise from quite different modeling situations.

The key is to combine the estimate with the likelihood’s behavior near the constraint and with residuals examined in context. A lower-bound estimate does not, by itself, establish that observations are measured without error. An upper-bound estimate does not establish that the true variance equals the software’s maximum. Both require checking the model, data, parameterization, and numerical estimation behavior.

Likelihood Behavior at a Constraint

Consider a simple diagnostic model in which predictions are treated as fixed and residuals \(e_i=y_i-f_i\) are independent normal random variables with mean zero and common variance \(v\). Here \(y_i\) is an observation, \(f_i\) its prediction, and \(v>0\) is the residual-error variance. This simplified setting gives a transparent way to understand likelihood direction; a pharmacometric model may also estimate predictions and other variance components.

The log likelihood combines a penalty for large variance through \(\log(v)\) with a penalty for residuals that are large relative to the variance. Differentiating with respect to \(v\) gives

For an unconstrained maximum with positive residual sum of squares, setting this derivative to zero gives \(\hat v=\sum e_i^2/n\). The derivative is positive below that value and negative above it, so the likelihood increases up to \(\hat v\) and decreases afterward. If the permitted range is \(v_{\min}\leq v\leq v_{\max}\), the constrained maximum in this simple model is the unconstrained value clipped to that range: it is \(v_{\min}\) if \(\hat v<v_{\min}\), \(v_{\max}\) if \(\hat v>v_{\max}\), and \(\hat v\) otherwise. This result follows from the derivative’s sign on either side of \(\hat v\); it does not imply that a complex nonlinear mixed-effects model has such a closed-form estimate.

At a lower bound, ask whether the likelihood would continue to improve as \(v\) decreases. At an upper bound, ask whether it would improve as \(v\) increases. In both cases, compare only evaluations made with the same data, objective, and estimation setup. The objective function value (OFV), when defined as \(-2\ell\), moves in the opposite direction to the log likelihood: a lower log likelihood means a higher OFV.

Reading a Boundary Estimate in a Pharmacometric Model

A pharmacometric residual model may use an additive error, a proportional error, or a combination. For example, a proportional model expresses error scale relative to the prediction, whereas an additive model expresses it in the observation’s units. Consequently, whether a parameter is a variance, standard deviation, or proportional coefficient matters. A bound of \(0.01\) on a standard deviation is not the same constraint as a bound of \(0.01\) on its variance: the corresponding variance is \(0.0001\).

Before interpreting the boundary, verify which quantity was constrained and in what units. Check that the recorded bound is sensible for the model’s parameterization and scale, that the reported value is not merely rounded to the boundary, and that the estimation run has credible convergence diagnostics. A boundary value from an unstable or failed run is not persuasive evidence about the data. Use gradient, objective-trace, and curvature checks as context rather than treating any one diagnostic as decisive.

Then examine residuals against predictions, time, and other relevant covariates. Look for systematic trends, changing spread, outliers, or clusters. If a residual variance is at its lower bound but the residual plot shows a pronounced pattern, the boundary is not evidence that the model describes the observations adequately. Conversely, small, pattern-free residuals are compatible with a small residual component, but still do not identify why that component is small.

Residual Patterns and Alternative Explanations

An estimate at a lower bound can arise because another model component accounts for variability that might otherwise appear as residual error. Random effects, for example, may explain between-individual differences, leaving a small estimated residual component. It can also arise when a chosen error structure is not well suited to the observations, when predictions are effectively overfitted, or when an implementation or data issue alters the residual scale. The boundary alone cannot distinguish these possibilities.

An upper-bound estimate calls for a different but overlapping set of checks. The model may leave substantial unexplained variability because its mean predictions are inadequate, because the data contain outliers or unmodeled heterogeneity, or because the residual error structure does not reflect how spread changes with prediction. A restrictive upper limit, a unit mismatch, or an inappropriate parameter scale can also force the estimate to the cap. These are hypotheses to investigate, not conclusions implied by the boundary.

Uncertainty, Comparisons, and Limitations

At a boundary, ordinary uncertainty summaries based on local curvature deserve caution. A standard error derived from a symmetric quadratic approximation presumes a sufficiently regular interior estimate. At a constraint, the likelihood may be asymmetric and its feasible direction is restricted; the curvature calculation may also be unstable. A small standard error or a successful covariance step does not, by itself, resolve the boundary diagnosis.

Useful comparisons include checking the fitted residual model against a scientifically plausible alternative error structure, inspecting residual plots under each fit, and comparing likelihoods only when the data and fitting setup make the comparison valid. Refit behavior across reasonable starting values and estimation settings can reveal numerical sensitivity. These checks help identify explanations, but none turns a constrained estimate into an unconstrained one. A bound should not be silently removed merely to obtain a more convenient estimate.

The simple Gaussian derivation assumes fixed predictions and independent, identically distributed normal residuals. In a mixed-effects model, predictions and other parameters are estimated, observations may have different conditional variances, and the likelihood can integrate over random effects. The derivative calculation is therefore a guide to the meaning of likelihood direction, not a substitute for evaluating the actual model. Nor does a boundary estimate alone establish a true zero residual effect or a unique cause. Report the constraint, parameter scale, diagnostics, and remaining uncertainty clearly.