Residual Plots Against Covariates and Observation Order

Why Plot Residuals Against Predictors or Order?

A residual plot against fitted means asks whether discrepancies change with the model’s predicted response. A plot against a covariate asks a different question: do discrepancies vary with a particular predictor, or with a feature of the data collection process? A plot against time or observation order asks whether discrepancies occur in a sequence, such as a gradual drift or a run of unusually positive residuals.

These plots are exploratory diagnostics. They can point to a missing curve, a group difference, a shift over time, or possible dependence between nearby observations. A visible pattern is a reason to investigate the model and data collection; it is not, by itself, a formal test or proof of a particular explanation.

Definition: A residual plot against a covariate displays a chosen residual for observation \(i\) against a predictor value \(x_{ij}\) or predictor category. A residual plot against observation order displays that residual against the order in which observations were recorded. If the observations have genuine time stamps, plotting against time is usually more informative than using row number alone.

For a response residual, the vertical coordinate is \(y_i-\hat{\mu}_i\). A Pearson residual divides this discrepancy by an estimated standard deviation; a deviance residual is another scaled measure of the discrepancy. The choice affects the vertical scale, but all can help reveal structure. Use the same residual definition consistently within a plot, and label the axes so the scale and the meaning of the horizontal variable are clear.

Reading a Plot Against a Covariate

For a numeric predictor, look for persistent curvature, a slope or change in spread, or clusters of positive and negative residuals across the range of the predictor. A smooth curve can help summarize a broad pattern, but it should be read descriptively: a wavy curve through a small number of points is not compelling evidence on its own. Also check whether apparent structure is driven by a few observations or by a region with very little data.

For a categorical predictor, plot residuals by level, often with a small horizontal jitter to separate overlapping points. Compare the centres and spreads across groups. A group of residuals predominantly above zero suggests that observed responses in that group tend to exceed their fitted means; predominantly negative residuals suggest the reverse. Group differences can also be summarized by mean residuals, provided the summaries are shown alongside the individual observations or interpreted with appropriate caution.

A pattern against a predictor already in the model does not automatically mean that the predictor should be removed. For example, a straight-line mean structure can leave curvature, while a categorical coding can conceal an ordering among levels. Conversely, a pattern can arise from an omitted variable associated with the plotted predictor. The plot identifies a feature to investigate, not the cause.

Worked Example: A Pattern Across a Numeric Predictor

Suppose a Poisson GLM is fitted to counts \(y_i\), and its fitted means at four predictor values are as follows. Calculate the Pearson residuals and describe what their plot against \(x\) suggests.

\(x_i\)	Observed count \(y_i\)	Fitted mean \(\hat{\mu}_i\)
0	2	3
1	3	4
2	7	6
3	11	9

For a Poisson response, the variance at mean \(\mu\) is \(\mu\). Thus the Pearson residual is \(r_{Pi}=(y_i-\hat{\mu}_i)/\sqrt{\hat{\mu}_i}\). Substituting each row gives:

r_{P1}=\frac{2-3}{\sqrt{3}}\approx -0.577,\quad r_{P2}=\frac{3-4}{\sqrt{4}}=-0.500,\quad r_{P3}=\frac{7-6}{\sqrt{6}}\approx 0.408,\quad r_{P4}=\frac{11-9}{\sqrt{9}}\approx 0.667.

The first two residuals are negative and the last two are positive: the observed counts are below the fitted means at the lower predictor values and above them at the higher values. This directional pattern is worth investigating. It could be consistent with an inadequately represented predictor effect, but four points do not establish that explanation. The fitted means here are inputs to the calculation; the example does not claim they came from any particular fitting procedure.

Reading a Plot Against Time or Observation Order

Order is informative only when it has a meaningful connection to how the data were collected. If rows were arbitrarily sorted by patient identifier, a plot against row number has no inherent temporal interpretation. If observations were collected in sequence, however, persistent runs of positive or negative residuals, a gradual drift, a sudden shift, or repeated cycles can signal structure not represented by the fitted mean model.

Time and a numeric predictor are not interchangeable. Calendar time can itself be included as a predictor, but an order plot can still help reveal a sequence of residuals that is not summarized by that mean trend. When observations are clustered, repeated within a person, or collected in batches, the order plot may reveal the arrangement but cannot by itself identify the appropriate dependence model. A pattern may also reflect changing measurement procedures, missing covariates, or other features of the study.

Worked Example: A Run of Residuals in Collection Order

Six counts are recorded in time order and fitted with an intercept-only Poisson model. The observed counts are \(2,2,3,5,6,6\). Calculate the fitted mean and Pearson residuals, then interpret their order.

For an intercept-only Poisson model, the maximum-likelihood estimate of the mean is the sample mean. Here,

\hat{\mu}=\frac{2+2+3+5+6+6}{6}=\frac{24}{6}=4.

Each Pearson residual is \((y_i-4)/\sqrt{4}=(y_i-4)/2\), so the sequence is

-1,\quad -1,\quad -0.5,\quad 0.5,\quad 1,\quad 1.

The first three residuals are negative and the last three are positive. A plot against time would show an ordered shift from below to above the fitted mean, rather than an unsystematic alternation around zero. This is a descriptive warning that the constant-mean model may not capture the sequence adequately. It does not establish that the counts are dependent: a changing mean, a batch effect, or chance could also produce this short pattern.

Group Summaries and Residual Balance

A useful way to describe a categorical pattern is to compare the average response residual within each group. The following identity makes the interpretation exact, regardless of the GLM family: the group’s average response residual is its average observed response minus its average fitted mean.

Result: For a group \(G\) containing \(m\) observations, let \(r_i=y_i-\hat{\mu}_i\) be the response residual. Then

\frac{1}{m}\sum_{i\in G}r_i = \frac{1}{m}\sum_{i\in G}y_i - \frac{1}{m}\sum_{i\in G}\hat{\mu}_i.

Proof. Substitute the definition \(r_i=y_i-\hat{\mu}_i\) into the left side and use the distributive property of a finite sum:

\frac{1}{m}\sum_{i\in G}(y_i-\hat{\mu}_i) = \frac{1}{m}\left(\sum_{i\in G}y_i-\sum_{i\in G}\hat{\mu}_i\right) = \frac{1}{m}\sum_{i\in G}y_i-\frac{1}{m}\sum_{i\in G}\hat{\mu}_i.

Thus a positive group average means the group’s average observed response exceeds its average fitted mean. This is a summary of the observed residuals, not a guarantee that the group difference is statistically reliable. \(\square\)

Worked Example: Comparing Residuals Across Two Groups

A Gaussian identity-link model with an intercept only is fitted to four responses. Two observations are in group A and two are in group B; the responses are \(7,9\) in A and \(11,13\) in B. Calculate the fitted mean, residuals, and average residual in each group.

The overall response mean is \((7+9+11+13)/4=40/4=10\), so the intercept-only fitted mean is 10 for each observation. The response residuals are \(7-10=-3\), \(9-10=-1\), \(11-10=1\), and \(13-10=3\). The group averages are

\bar r_A=\frac{-3+(-1)}{2}=-2,\qquad \bar r_B=\frac{1+3}{2}=2.

Equivalently, group A has observed mean \((7+9)/2=8\), which is 2 below its fitted mean of 10; group B has observed mean \((11+13)/2=12\), which is 2 above its fitted mean. A plot by group would expose this contrast. It suggests that group membership may be relevant to the mean structure, but it does not establish why the groups differ or whether the pattern would persist in new data.

Why Residuals Can Balance and Still Show a Pattern

A fitted model can impose exact algebraic balance conditions on residuals. For example, under a canonical-link GLM, the likelihood score equations make certain sums of response residuals equal zero at an interior maximum. Those conditions do not require residuals to be evenly scattered across every predictor value or ordered sequence.

Result: Suppose the observations are independent and have densities from an exponential family \(f(y_i;\theta_i,\phi)=\exp\{[y_i\theta_i-b(\theta_i)]/a(\phi)+c(y_i,\phi)\}\), with common dispersion factor \(a(\phi)\). Let the canonical link give \(\theta_i=\eta_i=x_i^{T}\beta\), where \(x_i\) is the design vector. If the log-likelihood is differentiable and a finite maximum occurs at an interior value \(\hat{\beta}\), then

\sum_{i=1}^{n}x_i(y_i-\hat{\mu}_i)=0, \qquad \hat{\mu}_i=b'(x_i^{T}\hat{\beta}).

Proof. The part of the log-likelihood depending on \(\beta\) is

\ell(\beta)=\sum_{i=1}^{n} \left\{\frac{y_i x_i^{T}\beta-b(x_i^{T}\beta)}{a(\phi)}\right\} +\text{terms not depending on }\beta.

Differentiating with respect to \(\beta\), using the chain rule and \(b'(\theta_i)=E[Y_i]=\mu_i\), gives

\frac{\partial\ell(\beta)}{\partial\beta} = \frac{1}{a(\phi)}\sum_{i=1}^{n}x_i\{y_i-b'(x_i^{T}\beta)\} = \frac{1}{a(\phi)}\sum_{i=1}^{n}x_i(y_i-\mu_i).

At an interior differentiable maximum the score is zero. Since \(a(\phi)\) is common and positive, setting this expression to zero at \(\hat{\beta}\) yields the stated identity. \(\square\)

For an intercept, this result gives a zero sum of response residuals; for other design columns, it gives corresponding weighted sums. These are global constraints. Positive and negative residuals can still form a curve, separate into groups, or occur in long runs that cancel in the total. In addition, the result concerns response residuals under the stated canonical-link conditions; it is not a general claim that every type of residual has zero average within every group.

Common Pitfall and Practical Reading

Wrong reasoning: “The fitted model’s score equations balance residuals, so a residual plot against a predictor or time cannot show a real pattern.” The score equations constrain particular sums, not the arrangement of residuals across predictor values or across time. A positive run can cancel a negative run in the overall sum.

Correction: Read the plot for structured departures, and interpret any pattern in context. Check that the horizontal axis has a meaningful scale and order; compare individual points with any smooth or group summary; consider whether a few observations dominate; and ask whether another recorded feature could explain the pattern. Do not treat a plot as a formal test, and do not infer causation or dependence from its appearance alone.

Against a numeric predictor, look for sustained curvature, direction, or changes in spread.
Against a categorical predictor, compare residual distributions and group averages.
Against time or genuine collection order, look for drift, shifts, cycles, or runs.
Keep the distinction between an observed diagnostic pattern and its possible explanation explicit.

Check Your Understanding

Use the definitions and reasoning in this tutorial to answer the following questions.

What question does a residual plot against a covariate address that a plot against fitted means may not directly address?
A group has average observed response 14 and average fitted mean 12. What is its average response residual, and what does its sign mean?
Why can canonical-link score equations hold while residuals form a visible curved pattern against a predictor?
What information is lost if observations are plotted by row number after the data have been sorted by patient identifier?
A sequence plot shows a run of positive residuals followed by a run of negative residuals. Name two different explanations that could be investigated, and explain why the plot alone does not distinguish them.