Diagnosing Residual Variance Misspecification

Why Compare Residual Spread Within Concentration Strata?

A residual variance model describes how much observations are expected to vary around their model-predicted means. If that variance is too small, residuals will tend to be more spread out than the model anticipates; if it is too large, they will tend to be less spread out. Because assay precision can depend on concentration, an overall residual spread can conceal a mismatch that is evident only in part of the concentration range.

The diagnostic question is not simply whether residuals are large. It is whether their spread around their stratum-specific mean is consistent with the residual variance predicted by the model. Their mean must be examined separately: a group of residuals consistently above zero indicates a directional discrepancy in the fitted means, not by itself an underestimated residual variance.

Definition: For observation $j$ from subject $i$, let $y_{ij}$ be the observed concentration, $\hat{\mu}_{ij}$ the fitted conditional mean, and $\hat{v}_{ij}$ the fitted residual variance for that observation. The standardized conditional residual is $$ z_{ij}=\frac{y_{ij}-\hat{\mu}_{ij}}{\sqrt{\hat{v}_{ij}}}. $$ Under a correctly specified conditional mean and variance model, these residuals should be centered near zero and have variance near one, in an idealized setting where the relevant parameters and individual effects are known. With estimated parameters and fitted individual effects, this is an approximate diagnostic target, not an exact finite-sample guarantee.

A concentration stratum is a group of observations in a chosen concentration range. Within each stratum, inspect both the signed residual mean and the residual spread around that mean. The mean helps reveal directional mean discrepancy; the centered spread is the key quantity for diagnosing variance mismatch. If the standardized residuals are appropriately calibrated, their within-stratum variance should be near one. A variance appreciably above one suggests that the model underestimates residual variance in that stratum; a variance appreciably below one suggests overestimation.

This comparison presumes that the standardized residuals are on a variance scale. For a Student-$t$ residual model with degrees of freedom $\nu>2$, the $t$ scale parameter is not its standard deviation: a $t$ variable with scale $a$ has variance $a^2\nu/(\nu-2)$. Standardize by the model-predicted standard deviation, rather than by $a$ alone. When $\nu\leq2$, the theoretical variance does not exist, so a variance-based comparison is not defined in the usual way.

Separate Mean and Spread

For a stratum containing $m$ standardized residuals $z_1,\ldots,z_m$, calculate the sample mean $\bar{z}$ and sample variance $s_z^2$. The sample variance measures spread around the stratum mean, not around zero. These answer different questions: $\bar{z}$ describes the signed offset, while $s_z^2$ describes dispersion after that offset is removed.

\bar{z}=\frac{1}{m}\sum_{k=1}^{m}z_k, \qquad s_z^2=\frac{1}{m-1}\sum_{k=1}^{m}(z_k-\bar{z})^2

A useful justification for keeping these summaries separate follows by expanding each squared residual around the mean:

\begin{aligned} \frac{1}{m}\sum_{k=1}^{m}z_k^2 &=\frac{1}{m}\sum_{k=1}^{m}\left[(z_k-\bar{z})+\bar{z}\right]^2\\ &=\frac{1}{m}\sum_{k=1}^{m}(z_k-\bar{z})^2+\bar{z}^{\,2}\\ &=\frac{m-1}{m}s_z^2+\bar{z}^{\,2}. \end{aligned}

The cross-product vanishes because $\sum_k(z_k-\bar{z})=0$. Thus the average squared residual combines a centered-spread contribution and a squared-mean contribution. A large average squared residual can arise from a nonzero mean even when the centered spread is small. Using only the average squared residual as a variance diagnostic can therefore confuse mean bias with variance underestimation.

Worked Concentration-Stratified Comparisons

Worked Example: Underestimated Variance in a Low-Concentration Stratum

Hypothetical data. In a low-concentration stratum, suppose five fitted standardized conditional residuals are $-2,-1,0,1,2$. The model predicts a standardized residual variance of one. Assume for this simple illustration that the observations can be treated as independent and that the fitted standard deviations are appropriate for the stratum.

The mean is $\bar{z}=(-2-1+0+1+2)/5=0$. The sum of squared deviations from the mean is $4+1+0+1+4=10$, so $s_z^2=10/(5-1)=2.5$. The observed sample variance is 2.5 times the model's target variance of one. The corresponding sample standard deviation is $\sqrt{2.5}\approx1.581$ standardized units, compared with a target standard deviation of one.

Because the mean is zero and the centered spread is above the target, this stratum shows a descriptive pattern consistent with underestimated residual variance. It does not establish the cause or statistical significance of the discrepancy; five observations provide a noisy estimate of spread.

Worked Example: Overestimated Variance in a Higher-Concentration Stratum

Hypothetical data. In a higher-concentration stratum, suppose three standardized conditional residuals are $-0.5,0,0.5$, again with a model target variance of one. Assume they are independent for this illustrative calculation and are standardized using the fitted residual standard deviations.

Their mean is zero. Their squared deviations sum to $0.25+0+0.25=0.50$, giving $s_z^2=0.50/(3-1)=0.25$. The sample standard deviation is $\sqrt{0.25}=0.5$. Thus the observed sample variance is one quarter of the target, and the observed standard deviation is half the target.

This is a pattern consistent with overestimated residual variance in this stratum: the model allows more residual spread than these data show. It is not correct to describe the model's variance as one quarter of the observed variance; the observed variance is one quarter of the model's target. As in the first example, the small sample makes this comparison uncertain.

Worked Example: Positive Mean With Little Centered Spread

Hypothetical data. A third stratum has standardized residuals $1.5,2,2.5,2,2$. Their mean is $\bar{z}=10/5=2$. The deviations from that mean are $-0.5,0,0.5,0,0$; their squared sum is $0.50$, so $s_z^2=0.50/4=0.125$.

The centered spread is below the target variance of one, while all residuals are positive and the mean is two standardized units. The average squared residual is $(2.25+4+6.25+4+4)/5=20.5/5=4.1$. The decomposition gives the same result: $(5-1)s_z^2/5+\bar{z}^{\,2}=4(0.125)/5+2^2=0.1+4=4.1$.

Looking only at the average squared residual of 4.1 might suggest greatly underestimated variance. But most of that value comes from the squared positive mean, $4$, rather than from centered spread, which is low. The pattern points to a substantial positive residual offset in this stratum, not to evidence of excess centered variability. A mean pattern calls for scrutiny of the fitted means; this variance diagnostic alone cannot determine its cause.

Conditions, Diagnostics, and Limits

These comparisons are most informative when the strata contain enough observations to estimate spread, the residual standard deviations are computed from the proposed variance model, and residual dependence has been accounted for or is negligible for the calculation. Repeated observations from one subject may be correlated. Treating a set of correlated residuals as independent can make a stratum look more informative than it is; an appropriate covariance structure or a subject-aware assessment may be needed.

In practice, use several complementary checks rather than relying on a single ratio. For each stratum, record the number of observations, $\bar{z}$, and $s_z^2$; compare the variance with one and the signed mean with zero. A plot of residuals within strata can reveal whether a summary is driven by one unusual observation. Examine whether conclusions persist under reasonable changes in bin boundaries, since different grouping choices can change the summaries. Small or unevenly sized strata deserve particular caution.

The ideal target of zero mean and unit variance is not exact after estimating population parameters and individual random effects from the same observations. Conditional residuals based on fitted individual effects may also be affected by how those effects are estimated. Consequently, a ratio above or below one is a diagnostic description, not automatically a formal test or proof of misspecification. Interpret its size alongside the data amount, the mean pattern, the dependence structure, and the fitting context.

Key takeaway: In each concentration stratum, examine the signed standardized-residual mean and the centered standardized-residual variance separately. A centered variance above one suggests underestimated residual variance; below one suggests overestimated variance. A nonzero mean is evidence of a directional discrepancy, not by itself evidence of variance mismatch.

Common Mistakes

Using residuals' average squares as if they were centered spread. The average square includes the squared mean. Calculate the variance around the stratum mean and report the mean separately.
Confusing variance and standard deviation ratios. A variance ratio of 2.5 corresponds to a standard-deviation ratio of $\sqrt{2.5}$, not 2.5.
Calling every large residual a variance problem. Residuals consistently on one side of zero indicate a mean offset. Check the signed mean before interpreting spread.
Treating a small stratum's variance as a precise estimate. Sample variance is unstable with few observations; report the stratum size and avoid strong conclusions from a handful of values.
Ignoring dependence or the residual distribution's scale. Correlated observations do not provide the same information as independent ones, and a Student-$t$ scale is not generally its standard deviation.

Check Your Understanding

Use the distinction between signed mean and centered spread to answer each question. The solutions are included below for checking your work.

A stratum has standardized residuals $-1,0,1$. Calculate its mean and sample variance, then compare the variance with the unit target.
In a stratum, every standardized residual equals $1.5$. What are the sample mean and sample variance? Which diagnostic feature is most evident?
A stratum's centered sample variance is $0.64$. What is its sample standard deviation, and what does the variance ratio suggest relative to a unit target?
Why can a large average squared standardized residual fail to demonstrate underestimated residual variance?

Solutions. (1) The mean is $(-1+0+1)/3=0$; the sample variance is $(1+0+1)/(3-1)=1$, matching the target descriptively. (2) The mean is $1.5$, and all deviations from it are zero, so the sample variance is zero. The prominent feature is a positive mean offset, not excess centered spread. (3) The sample standard deviation is $\sqrt{0.64}=0.8$. The observed variance is 0.64 times the unit target, a pattern consistent with overestimation, subject to sampling uncertainty. (4) The average squared residual includes the squared stratum mean, as well as the centered-spread contribution, so a nonzero mean can make it large even when centered variance is not high.