Why Examine Residuals?
A model produces predictions. Measurements show what was observed. Comparing the two at the same times is a direct way to see whether the model is tracking the data well—and, importantly, whether its failures have a pattern.
A prediction can be wrong by a small amount because measurements are imperfect or because the model simplifies a complicated process. But if the prediction errors stay mostly on one side of zero, change direction at a particular time, or grow steadily, that pattern can point to a feature the model does not capture. It is a diagnostic clue, not by itself a proof of what caused the mismatch.
In this tutorial, observations will be denoted by \(y_i\), taken at times \(t_i\). A model gives a predicted quantity \(M(t)\). The comparison must use the model prediction at the same time as each observation.
Read the Pattern, Not Just the Size
The absolute size of a residual indicates how far one prediction is from one observation. The sequence of residuals, ordered by time, can reveal more. A nearly constant positive residual suggests a persistent offset. Residuals that are small at first and then drift upward suggest that the mismatch changes over time. Residuals that first fall and then rise can indicate that a simple trend does not capture the shape of the observations.
These interpretations are hypotheses to investigate. A pattern may also arise from measurement error, an unusual observation, or another feature not represented in the model. The residuals show where and how the model misses the observations; they do not uniquely identify the cause.
| Residual pattern over time | Possible diagnostic clue |
|---|---|
| Mostly positive or mostly negative, with similar size | A baseline offset may be present |
| Small before a time and then consistently larger | A regime change or other change in the process may be missing |
| Residuals grow or curve systematically | The model's rate or shape may be unsuitable |
| Mixed signs without a visible time pattern | The model may capture the broad trend, though individual errors remain |
Worked Example: A Persistent Baseline Offset
Worked Example: A Persistent Baseline Offset
Suppose a model predicts \(M(t)=5+2t\), and the observed quantities at times \(t=0,1,2,3\) are \(6,8,10,12\). Find the residuals and interpret their pattern.
At \(t=0\), the model predicts \(M(0)=5+2(0)=5\), so \(r_0=6-5=1\).
At \(t=1\), it predicts \(M(1)=5+2(1)=7\), so \(r_1=8-7=1\).
At \(t=2\), it predicts \(M(2)=5+2(2)=9\), so \(r_2=10-9=1\).
At \(t=3\), it predicts \(M(3)=5+2(3)=11\), so \(r_3=12-11=1\).
Every residual is positive and equal to \(1\). At these observation times, the model underpredicts by one unit each time. A baseline-shifted model \(M_1(t)=M(t)+1=6+2t\) gives predictions \(6,8,10,12\), so its residuals at the four sampled times are all zero.
Check: Subtracting each shifted prediction from the corresponding observation gives \(6-6=0\), \(8-8=0\), \(10-10=0\), and \(12-12=0\). This checks the fit at the listed times only. The observations do not establish that the quantity follows \(6+2t\) at times that were not measured.
Why a Constant Residual Suggests a Baseline Shift
The baseline of a model is its level before any time-dependent change is added. The calculation behind the previous example generalizes. Suppose a model is changed by adding the same constant \(c\) to each prediction. The new prediction at \(t_i\) is \(M(t_i)+c\). Its residual is therefore
Thus, if the original residuals are all exactly \(c\), adding \(c\) to the model predictions makes the residuals zero at those observation times. More generally, a cluster of residuals around a shared positive or negative level is a reason to check the baseline. It does not prove the baseline is the only problem: other errors or measurement variation could contribute to the observed pattern.
Worked Example: A Possible Change in Regime
Worked Example: A Possible Change in Regime
A model predicts \(M(t)=10+3t\). The observed quantities at integer times \(t=0,1,2,3,4,5\) are \(10,13,16,22,28,34\). Calculate the residuals and identify what changes in their pattern.
The model predictions are \(M(0)=10\), \(M(1)=13\), \(M(2)=16\), \(M(3)=19\), \(M(4)=22\), and \(M(5)=25\).
Subtract the predictions from the observations in time order:
\(r_0=10-10=0\), \(r_1=13-13=0\), \(r_2=16-16=0\), \(r_3=22-19=3\), \(r_4=28-22=6\), and \(r_5=34-25=9\).
The model matches the first three observations, but then the residuals become positive and increase by \(3\) at each successive time. This pattern is consistent with the original model failing to capture a faster increase after \(t=2\). For example, a candidate piecewise prediction that keeps the first segment and uses a rate of \(6\) after \(t=2\) is \(P(t)=10+3t\) for \(t\leq 2\), and \(P(t)=16+6(t-2)\) for \(t>2\).
For \(t=3,4,5\), this candidate gives \(P(3)=16+6(1)=22\), \(P(4)=16+6(2)=28\), and \(P(5)=16+6(3)=34\). Together with \(P(0)=10\), \(P(1)=13\), and \(P(2)=16\), it matches every listed observation.
Check: The revised predictions are \(10,13,16,22,28,34\); subtracting them from the observations gives zero at all six times. This is a possible model consistent with the data, not proof that the underlying process changed its rate at exactly \(t=2\). Other explanations could also produce the same sampled values.
Increasing Residuals and an Unsuitable Rate Model
A rate model can fail even when there is no sharp change at one identifiable time. If the residuals bend or grow steadily, the model's assumed trend may be too simple. For example, a model with a constant rate predicts equal increases over equal time intervals. If successive observations increase by larger amounts, that is a reason to question whether a constant rate is adequate.
Worked Example: Residual Curvature Against a Constant-Rate Model
Consider the constant-rate prediction \(M(t)=4+2t\) and observations \(4,7,12,19,28\) at \(t=0,1,2,3,4\). Calculate the residuals.
The predictions are \(M(0)=4\), \(M(1)=6\), \(M(2)=8\), \(M(3)=10\), and \(M(4)=12\).
The residuals are \(4-4=0\), \(7-6=1\), \(12-8=4\), \(19-10=9\), and \(28-12=16\).
The residuals are positive and grow faster than a constant offset. One expression consistent with these five residual values is \(t^2\), since \(0^2=0\), \(1^2=1\), \(2^2=4\), \(3^2=9\), and \(4^2=16\). Consequently, the observations are consistent at these times with the candidate expression \(4+2t+t^2\).
Check: At the five sampled times, \(4+2t+t^2\) gives \(4,7,12,19,28\), exactly the listed observations. This agreement does not establish that the underlying quantity equals that expression between observations. Although the derivative of the candidate expression is \(2+2t\), the sampled values alone do not establish that this is the underlying quantity's rate. The residual pattern only shows that the stated constant-rate model increasingly underpredicts the observations at the sampled times.
A Careful Diagnostic Process
Residual analysis is most useful when it proceeds in a consistent order. The goal is to describe what the comparisons show before proposing why the model misses.
Compare each observation \(y_i\) with \(M(t_i)\), the prediction at that same time, and confirm that both quantities use the same units.
Use \(r_i=y_i-M(t_i)\). Keep the sign, since it distinguishes underprediction from overprediction.
Look for persistent signs, a shift in level, or a steadily changing pattern rather than judging each number separately.
Describe the pattern and identify a model feature it may call into question. Do not present a possible cause as established solely by the residuals.
Recalculate its predictions at the observed times and verify the resulting residuals. Report clearly whether the check concerns only the sample or a broader claim.
Common Pitfall: Turning a Fit Into a Fact
A common error is to treat a formula that matches the observations as if it were proven to describe the quantity at every time. For instance, finding that \(3+2t+t^2\) gives \(3,6,11,18\) at \(t=0,1,2,3\) shows that this formula is consistent with those four values. It does not prove that the underlying quantity equals that formula between the observation times, or that its rate is \(2+2t\).
The correction is to distinguish the data from the model: say that the observations are consistent with the formula at the measured times, and state separately what the formula would imply if it were adopted as a model. Residuals diagnose disagreement between a chosen model and measured values; they do not turn a finite set of measurements into certainty about unobserved behavior.
A second practical error is reversing the subtraction. With the convention \(r_i=y_i-M(t_i)\), a positive residual means the observation is greater than the prediction. If \(y_i=12\) and \(M(t_i)=15\), then \(r_i=12-15=-3\), so the model overpredicts by \(3\), not underpredicts. Keeping the definition visible prevents sign interpretations from being reversed.
Why Residual Patterns Matter
A model may have been calibrated to match selected measurements, but other observed times can still reveal systematic failure. Residual patterns help distinguish a uniform level mismatch from a time-dependent mismatch and can indicate when a proposed rate model is not capturing the observed trend. This makes the comparison useful for refining predictions and deciding what additional measurements might clarify.
The key discipline is to separate three things: the residuals that were calculated, the model feature they may call into question, and the explanation that remains uncertain. A persistent offset is evidence of a baseline mismatch in the comparisons. A new drift may be evidence that one regime does not describe all times. Curvature may be evidence against a constant-rate prediction. None of these patterns alone proves the underlying mechanism.
Check Your Understanding
Use the residual convention \(r_i=y_i-M(t_i)\) to answer the following questions.
- If an observation is \(18\) units and the model prediction is \(21\) units at the same time, what is the residual, and what does its sign mean?
- What model failure is suggested when residuals are approximately equal and positive across several observation times?
- A model has residuals \(0,0,0,4,8,12\) at increasing times. Describe the pattern and give one cautious interpretation.
- Why does a candidate formula matching all sampled observations not establish the quantity's values between those times?
- What should be checked before interpreting a residual pattern if observations and predictions were recorded in different units?
