Why Examine Residuals Across Covariates?
A pooled residual display can look broadly centered even when the model misses a relationship with a covariate. For example, positive residuals in one range of body weight and negative residuals in another may cancel when summarized together. Similarly, residuals may be systematically higher in one category than another, although the pooled average is close to zero. Examining residuals across covariates helps reveal such patterns and can indicate where a model’s mean or variability may need further investigation.
This diagnostic is descriptive: it helps locate structure left in the residuals, but it does not by itself identify the cause or prove that a covariate should be added. The plotted residual must be defined consistently, and its sign must be understood. In the examples below, residual means the observed concentration minus its corresponding prediction, so a positive value indicates that the observation exceeds the prediction.
What Pattern Should a Well-Described Model Leave?
Let \(Y_i\) denote an observed response and \(X_i\) a covariate. Suppose the model specifies a conditional mean \(m(X_i;\theta)\), where \(\theta\) is the parameter vector. For this diagnostic’s basic justification, assume that the conditional mean is correctly specified and that the error has conditional mean zero:
If \(\theta\) were known and the residual were \(r_i=Y_i-m(X_i;\theta)\), then the residual’s conditional mean would be zero:
The final equality follows from the assumption that \(m(X_i;\theta)\) is the true conditional mean. Thus, a persistent nonzero residual location at particular covariate values or levels can suggest that the modeled conditional mean does not adequately describe the data. In practice, parameters are estimated, and residuals may be transformed or standardized. Their fitting constraints and scale therefore depend on the model and residual definition. The diagnostic is an empirical check for visible structure, not a claim that every fitted residual must be exactly zero on average at every covariate value.
For a continuous covariate, inspect the points across its observed range and look for location patterns such as a slope, curvature, or threshold-like change. For a categorical covariate, compare the residual distributions and centers across levels. In either case, also inspect spread: changing variability can be important even when the residuals remain centered near zero.
Use the recorded covariate values or levels and one residual definition consistently. State the residual sign convention and units.
For a continuous covariate, show residuals against the covariate. For a categorical covariate, show the individual residuals within each level, using modest horizontal jitter if points overlap.
Look for residual centers that change with the covariate, then look for changes in the vertical dispersion around those centers.
Note the number and range of observations, overlap between groups, and whether a few points dominate the apparent pattern.
Worked Examples
Worked Example: A Continuous Covariate and a Small Linear Trend
Suppose a hypothetical diagnostic data set contains five observations. The covariate is body weight in kilograms, and the residual is observed minus predicted concentration in mg/L. The recorded pairs are \((20,0.8)\), \((40,0.2)\), \((60,-0.3)\), \((80,-0.2)\), and \((100,0.7)\). Calculate the least-squares linear trend in residual against weight, then describe what the individual points show.
The mean weight is \(\bar{x}=(20+40+60+80+100)/5=60\) kg. The mean residual is \(\bar{r}=(0.8+0.2-0.3-0.2+0.7)/5=1.2/5=0.24\) mg/L. The slope is the sum of the cross-products of centered values divided by the sum of squared centered covariate values:
The fitted line is \(r=0.42-0.00300x\), because its intercept is \(\bar{r}-b_1\bar{x}=0.24-(-0.00300)(60)=0.42\) mg/L. It predicts a residual of \(0.36\) mg/L at 20 kg and \(0.12\) mg/L at 100 kg. This fitted slope is small and negative, but it is not a complete description of the points: the endpoint residuals are positive; the residual at 40 kg is also positive; and the observations at 60 kg and 80 kg are negative. A plot would show the individual pattern as well as the fitted summary. With only five observations, this calculation is descriptive and does not establish a reliable population trend.
Worked Example: Residual Location Across Categorical Levels
In a hypothetical analysis, renal-function category has three levels. Residuals, in mg/L, are: low, \(-0.4, 0.0, 0.2\); middle, \(0.1, 0.3, 0.5\); and high, \(-0.2, 0.0, 0.2\). Calculate the mean residual in each category and the pooled mean.
For the low category, the mean is \((-0.4+0.0+0.2)/3=-0.2/3=-0.0667\) mg/L. For the middle category, it is \((0.1+0.3+0.5)/3=0.9/3=0.300\) mg/L. For the high category, it is \((-0.2+0.0+0.2)/3=0/3=0\) mg/L. The pooled mean is the sum of all nine residuals divided by nine: \((-0.2+0.9+0)/9=0.7/9=0.0778\) mg/L.
The pooled mean is close to zero, but the middle category has a higher residual center than either of the other two categories. Its mean exceeds the low-category mean by \(0.300-(-0.0667)=0.367\) mg/L and the high-category mean by \(0.300-0=0.300\) mg/L. Plotting all observations by level would make this contrast visible without implying that it is precisely estimated: each category contains only three observations. The pattern motivates checking the model and data, not automatically adding a category effect.
Worked Example: Increasing Spread Without a Change in Group Means
A hypothetical set of residuals in mg/L is recorded at three values of a continuous covariate, clearance in L/h: at 10 L/h, \(-0.2, 0.2\); at 30 L/h, \(-0.6, 0.6\); and at 50 L/h, \(-1.0, 1.0\). Compare residual location and spread across these values.
The mean residual at each covariate value is zero: for example, at 30 L/h, \((-0.6+0.6)/2=0\) mg/L. The two observations at each value are increasingly far from that center. Their sample variances are \(0.08\), \(0.72\), and \(2.00\) \((\text{mg/L})^2\), respectively. For example, at 50 L/h the sample variance is \(\{(-1.0-0)^2+(1.0-0)^2\}/(2-1)=2.00\) \((\text{mg/L})^2\).
The example illustrates a spread pattern rather than a shift in the displayed residual means. It could point to a variance-model issue, but two observations at each covariate value are far too few to establish a general variance relationship. A residual plot helps separate these questions: are the centers changing, is the spread changing, or do both features change?
Reading the Plot and Comparing Alternatives
The most useful comparison is between a covariate-specific view and a pooled view. A histogram or normal-quantile plot, as covered in the earlier material on residual distribution and normal-quantile plots, summarizes the pooled residual distribution. It cannot show where in covariate space residuals tend to be positive or negative. A residual-versus-prediction display can reveal structure across predicted values, but it does not substitute for checking a scientifically relevant covariate: observations with similar predictions can have different covariate values, and vice versa.
For a continuous covariate, use points at the observed values; avoid replacing the scatter with a line that suggests more certainty than the data support. A smooth guide may help reveal broad curvature if it is clearly presented as descriptive. For a categorical covariate, show the raw residuals by level. A table of group counts and means can supplement the plot, but means alone conceal spread, skew, and isolated observations. If a level has very few observations, report or display that limitation rather than treating an unstable center as a settled pattern.
| Display | What it can reveal | What it does not establish |
|---|---|---|
| Residuals versus a continuous covariate | Location changes, curvature, abrupt changes, or changing spread across the observed range | A causal covariate effect or a reliable pattern where data are sparse |
| Residuals grouped by a categorical covariate | Differences in residual center or spread among the observed levels | That the category caused the discrepancy or that small groups are representative |
| Pooled residual distribution | Overall features of the residual distribution | Where in covariate values or levels a pattern occurs |
| Residuals versus predictions | Residual structure across predicted response values | Whether that structure is specifically associated with a covariate |
Interpret the pattern in light of the model, the residual definition, the observation scale, and the data collection. A trend can arise from a misspecified covariate relationship, but it may also reflect a correlated omitted variable, influential observations, sparse coverage, or a measurement issue. In repeated-measures data, residuals from the same participant may be dependent; a visual pattern should not be treated as a set of independent replications. A visible group difference also does not establish a causal effect of group membership.
Common Mistakes and Diagnostic Limits
- Relying on the pooled average. A pooled mean near zero can result from positive residuals in one covariate region and negative residuals in another. Inspect the conditional pattern, not just the overall center.
- Calling every pattern a mean trend. A widening band around zero suggests changing spread, not necessarily a changing conditional mean. Assess vertical location and dispersion separately.
- Reading a fitted straight line as the whole story. A slope compresses a potentially curved or irregular pattern into one number. Inspect the plotted points, including their signs and any influential observations.
- Treating categories as ordered without justification. If levels are labels rather than a meaningful scale, compare the levels directly; do not impose a linear trend across them merely because software can encode them as numbers.
- Inferring causation or certainty from a diagnostic plot. A plot can suggest where the model leaves structure, but it cannot identify the cause or quantify uncertainty on its own.
These displays are limited by the observed covariate range and the number of observations. A smooth-looking plot cannot validate extrapolation beyond that range. Sparse regions and rare categories make trends difficult to assess, while overlapping points can hide how many observations occupy a location. Keep the raw observations visible where possible, use consistent scales across comparable panels, and treat visual features as prompts for model and data review rather than as automatic instructions to modify the model.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- For a categorical level, residuals are \(0.2,-0.1,0.5\) mg/L. What is their mean residual?
- Residuals at successive covariate values are centered near zero, but their vertical spread increases. What feature is changing, and what should not be concluded from this appearance alone?
- Two equally sized categories have residuals \(-0.4,-0.2\) mg/L and \(0.2,0.4\) mg/L. What is the pooled mean, and why could a pooled display hide the group pattern?
- Why is it useful to show individual residuals across categorical levels rather than only a table of level means?
Solutions. (1) The mean is \((0.2-0.1+0.5)/3=0.6/3=0.200\) mg/L. (2) The spread, or dispersion, appears to increase while the centers remain near zero. This alone does not prove a general variance relationship, especially if observations are sparse, nor does it show that the conditional mean is changing. (3) The pooled mean is \((-0.4-0.2+0.2+0.4)/4=0/4=0\) mg/L. Opposing group centers cancel in the pooled average, so a pooled display can look centered while hiding differences by category. (4) Individual points show spread, overlap, and possible outliers that a mean does not reveal; they also make small group sizes apparent.