Residuals Under Heteroscedastic PD Error

Why Residual Variance Needs Its Own Model

A PD model specifies more than how the expected response changes with concentration, time, or treatment. It also needs a description of how observations vary around that expected response. If the spread of observations changes with the expected response, a constant-variance error model can give misleading diagnostics and inefficient estimates. A residual that is ordinary in absolute size at a high response may be unusually large at a low response.

The key diagnostic question is therefore not simply whether residuals are large or small. It is whether their size is plausible relative to the expected error scale at that fitted response. Residual-versus-fitted plots show how raw residuals behave across the prediction range. Variance-standardized residuals put those residuals on a scale that accounts for the fitted variance function.

Here, the mean prediction and residual variance must refer to the same prediction level and error model. Conditional and population-level residuals answer different predictive questions, as explained in the earlier material on Conditional and Population-Level Residuals. A variance function cannot make those two residual definitions interchangeable.

Common PD Error Variance Functions

In the formulas below, \(\mu_i\) denotes the model-implied mean response for observation \(i\), before replacing unknown parameters with estimates. The formulas describe the conditional observation variance around that mean. They do not, by themselves, specify the mean-response model.

The combined formula follows when an additive error component and a proportional error component are independent, have mean zero, and have variances \(\sigma_a^2\) and \((\sigma_p\mu_i)^2\). The variance of their sum is the sum of their variances. The resulting standard deviation is the square root of that sum, not the sum of the two standard deviations.

An additive model predicts similar vertical spread at low and high fitted responses. A proportional model predicts a constant coefficient of variation: its error standard deviation divided by the magnitude of the mean is \(\sigma_p\). A combined model allows a substantial additive contribution near zero and increasing proportional contribution at larger mean responses. If a proportional model is used where the mean approaches zero, its predicted variance also approaches zero; that may be scientifically implausible or numerically troublesome. The combined model avoids that particular collapse when \(\sigma_a\) is positive.

These are variance assumptions, not guarantees that observed residuals will follow a normal distribution. Normal errors are often used for continuous PD likelihoods, but assessing whether spread changes with the fitted mean does not require asserting normality. Interpretation of extreme residuals does depend on the assumed tail behavior.

Reading Raw and Variance-Standardized Residuals

Plot raw residuals \(e_i\) against fitted means \(\hat f_i\). Under a suitable additive model, the vertical spread should be approximately stable over the fitted range. Under proportional error, raw residuals often fan out as fitted responses increase. Under combined error, the spread may be relatively stable at low fitted responses and widen at higher ones. The pattern should be judged across enough observations to distinguish a systematic change in spread from a few extreme values.

Then examine variance-standardized residuals, calculated with the fitted variance function. If that function adequately describes the scale, these residuals should have roughly similar spread across fitted values, without a systematic widening or narrowing. When the conditional error model has mean-zero errors and \(V_i\) is the correct variance, the standardized error \(\epsilon_i/\sqrt{V_i}\) has variance one. In an estimated model, fitted residuals will not generally be independent standard normal values: parameter estimation, repeated observations, and random effects can affect their distribution.

The standardization result is direct. If \(E(\epsilon_i\mid\mu_i)=0\) and \(\operatorname{Var}(\epsilon_i\mid\mu_i)=V_i\), then dividing by the positive square root of the variance gives:

This justifies comparing scale across fitted values after standardization. It does not prove that the mean model is correct, that errors are independent, or that their distribution is normal. A curve in standardized residuals can indicate mean misspecification, while clusters or sequences can reflect dependence rather than a variance-function problem.

Evaluating a Variance Model and Its Alternatives

A practical assessment pairs the raw residual-versus-fitted plot with the plot of variance-standardized residuals against fitted values. The first asks whether the observed scale changes; the second asks whether the proposed variance function has accounted for that change. A smooth trend in the center of residuals is not a pure variance signature: it can arise when the mean model is wrong. A remaining fan in standardized residuals suggests that the variance function may be inadequate, but a small data set may not provide enough information to distinguish competing functions reliably.

Compare plausible models using the same response definition and compatible likelihoods. A likelihood-based criterion can help compare additive, proportional, and combined error structures, but a lower criterion alone does not show that the chosen model is scientifically appropriate. Check whether fitted parameters are stable, whether the standardized residual spread is reasonably even, and whether conclusions depend strongly on the selected error structure. Replicate observations at similar fitted means are especially informative because they help separate spread from a changing mean.

Weighted fitting can be a computational way to account for unequal variances, but weighting is not a substitute for deciding what variance pattern is plausible. Likewise, transforming a response changes the scale on which errors and effects are interpreted. Neither approach automatically repairs a misspecified mean or dependence structure.

Diagnostics, Failure Modes, and Limitations

Several checks help prevent a variance pattern from being misdiagnosed. First inspect the center of the residual cloud: curvature or a systematic offset may indicate a mean-model problem. Second, consider whether observations are clustered within subjects or ordered in time; dependent residuals do not satisfy the independent-error assumption used by a simple variance function. Third, assess whether a few extreme observations are dominating the apparent spread. Finally, ensure that the fitted variance is positive and that its parameters are estimable over the observed range.

A variance function is most interpretable when the mean model is adequate, the residuals are defined at a clear prediction level, and the relevant observations are conditionally independent under the assumed model. A normal likelihood adds the assumption of normally distributed conditional errors. If the mean is near zero, a proportional-only function can imply almost no variability; if the response can be negative or takes values on a bounded scale, the simple functions may also be unsuitable. Those cases require care beyond choosing among these three basic patterns.

Common pitfall: “The high-response observations have larger raw residuals, so the mean model must be wrong.” Larger residuals in absolute units may be exactly what a proportional or combined variance model predicts. The correction is to compare the residual size with the expected standard deviation at that fitted response. Conversely, dividing by a variance function and seeing an even-looking plot does not prove the mean model, independence assumptions, or distributional tails are correct.