Separating Curvature from Changing Spread

Why Similar Residual Plots Can Have Different Causes

A residual plot can look troubling for more than one reason. A curved band may indicate that the fitted mean curve misses a systematic bend. A fan-shaped band may instead indicate that responses become more variable as the predictor changes. A few extreme points or repeated group-specific shifts can create still other patterns. These distinctions matter: changing the mean curve does not, by itself, fix unequal variance or dependence among observations.

The goal is not to diagnose a problem from one visual feature with certainty. It is to ask which part of the model the pattern challenges: the conditional mean, the conditional variance, the treatment of individual observations, or the assumption that errors are independent. A residual plot supplies evidence for that investigation, not a definitive explanation.

Definition: Let \(Y_i\) be the response for observation \(i\), and let \(x_i\) denote its predictor value. Write the true conditional mean as \(m(x_i)=E(Y_i\mid x_i)\). A fitted regression gives a predicted value \(\hat{y}_i\), and the residual is \(e_i=y_i-\hat{y}_i\). A residual is an observed difference between a response and its fitted value; it is not the unobserved error itself.

What Each Kind of Pattern Concerns

It helps to separate the mean from the spread. A suitable mean curve should represent how the average response changes with the predictor. A suitable variance description should represent how much individual responses vary around that mean. These are distinct features of the conditional distribution of \(Y\).

Model framework: A useful starting point is \(Y_i=m(x_i)+\varepsilon_i\), where \(E(\varepsilon_i\mid x_i)=0\). A constant-variance model additionally assumes \(\operatorname{Var}(\varepsilon_i\mid x_i)=\sigma^2\); a changing-variance model allows this conditional variance to depend on \(x_i\). For the usual independent-error regression interpretation, observations must also be independent, or the dependence must be represented in the model. A residual diagnostic is meaningful only with these assumptions in view.

If a straight line is fitted when the mean relationship bends, residuals can tend to be positive in some predictor regions and negative in others. That is a mean-pattern clue. If the mean curve is adequate but the vertical scatter grows with the predictor, residuals may form a fan. That is a variance-pattern clue. A single extreme residual may reflect an unusual response, a recording error, or a genuinely rare outcome; it does not automatically indicate curvature. If residuals tend to share a sign within batches, clinics, schools, or time periods, that grouping may indicate dependence or a missing group-level feature.

The distinction can be expressed directly. Suppose a candidate curve \(g(x)\) is used to predict the mean. Before accounting for the fact that the curve is estimated from data, the difference between a response and this candidate is

Y-g(x)=\{m(x)-g(x)\}+\varepsilon .

The first term is the mean-curve discrepancy; the second is random error around the true mean. Conditional on \(x\), the expected squared discrepancy is

E\{(Y-g(x))^2\mid x\} =\{m(x)-g(x)\}^2+\operatorname{Var}(\varepsilon\mid x).

To see this, square the sum: the cross-product has conditional expectation \(2\{m(x)-g(x)\}E(\varepsilon\mid x)=0\), while \(E(\varepsilon^2\mid x)=\operatorname{Var}(\varepsilon\mid x)\) because the conditional error mean is zero. Thus a large squared residual can arise from a mean mismatch, large variance, or both. In an actual fitted model, residuals also reflect estimation from the same data, so this identity explains the diagnostic logic but does not make any single residual a clean measurement of either component.

Read the Residual Plot by Asking Separate Questions

Start with residuals against the predictor, and, when relevant, against fitted values, observation order, or a group label. Keep the scale and the observational units in mind. A residual of 5 means five response-units above the fitted value, not a five-unit change in the predictor.

Look at the center.
Do residuals tend to lie above zero in one region and below zero in another, in an orderly shape? Repeated structure suggests the fitted mean may be missing a feature.

Look at the vertical spread.
Does the typical distance from zero increase or decrease across predictor values while the center remains roughly level? That is evidence to investigate unequal conditional variance.

Look for isolated points.
Check the observation and its measurement context. An extreme residual concerns an unusual response conditional on its fitted value; an unusual predictor value may also give the point leverage over the fitted curve.

Look for grouping or order.
Runs of similar residuals within a batch or adjacent times can indicate shared conditions or dependence. Do not treat such a pattern as ordinary independent scatter merely because residuals average near zero overall.

These questions are complementary. A residual plot can show both a changing center and changing spread, and clustered observations can also have a changing mean. Examine the sampling or measurement process, repeated observations, and the order in which data were collected when the design makes those features relevant.

Worked Example: A Curved Residual Center

Suppose a hypothetical laboratory records instrument response in units for five settings \(x=0,1,2,3,4\). A straight-line fit gives predicted responses \(10,12,14,16,18\), respectively. The observed responses are \(12,11,12,15,20\). Calculate the residuals and interpret their pattern.

For each setting, residual equals observed response minus fitted response:

e_i=y_i-\hat{y}_i: \quad (12-10,\;11-12,\;12-14,\;15-16,\;20-18) =(2,-1,-2,-1,2).

The residuals are positive at both ends and negative in the middle. Their sum is \(2-1-2-1+2=0\), and their predictor-weighted sum is \(0(2)+1(-1)+2(-2)+3(-1)+4(2)=0\). These are consistent with the stated least-squares line, yet the ordered U-shaped pattern remains. In context, the straight line tends to underpredict response at the low and high settings and overpredict it through the middle.

This is evidence that the fitted mean may miss curvature. It is not evidence that the response variance increases with the setting: the magnitudes are not steadily widening, and the pattern concerns which side of zero the residuals occupy. The data are hypothetical, and five points alone would not establish the correct mean form.

Worked Example: A Fan With Little Mean Pattern

A hypothetical process has four measurements at each of three settings. Suppose the fitted mean responses are 10, 12, and 14 units. The residuals at the first setting are \((-1,1,-1,1)\), at the second are \((-2,2,-2,2)\), and at the third are \((-4,4,-4,4)\). Compare the group centers and spreads.

The residual mean at each setting is zero: for example, at the third setting, \((-4+4-4+4)/4=0\). Thus these summaries do not show a systematic positive or negative residual center by setting. The sample residual variances, using the four residuals within each setting and the usual denominator \(n-1=3\), are:

s_1^2=\frac{1+1+1+1}{3}=\frac{4}{3}=1.333,\qquad s_2^2=\frac{4+4+4+4}{3}=\frac{16}{3}=5.333,\qquad s_3^2=\frac{16+16+16+16}{3}=\frac{64}{3}=21.33.

The residual center remains at zero, while the observed spread grows across the settings. A residual plot would tend to look like a fan. This is a clue to changing variance in response units squared, not a reason by itself to replace the mean curve with a more curved one. With only four observations per setting, the variance estimates are uncertain; more evidence and knowledge of the process are needed.

Unusual Points and Clustered Residuals

An unusual observation should be considered separately from a broad residual pattern. A large residual means the observed response is far from the model's prediction. It does not, on its own, say why. Verify units, transcription, eligibility, and measurement conditions before deciding whether to retain or exclude the observation. A real but extreme value may be scientifically important and should not be deleted simply because it changes a fitted result.

A point with an unusual predictor value can have high leverage: it is positioned where it may exert substantial influence on the fitted curve. A point can have a large residual without high leverage, or high leverage without a large residual. Therefore, neither a residual-versus-predictor plot nor the word “outlier” alone describes all aspects of its effect on a fit.

Clustered residuals raise a different concern. If observations from the same batch share an unmeasured condition, their errors may be related. Treating them as independent can make standard errors and uncertainty statements unreliable. A pattern tied to groups should prompt a check of the design and data collection; it is not necessarily fixed by selecting a more flexible curve in the predictor.

Worked Example: Repeated Shifts Within Batches

A hypothetical instrument is used in four batches, with three residuals (in response units) recorded per batch after fitting a predictor-based mean curve. The residuals are: batch A \((-3,-2,-1)\), batch B \((-3,-2,-1)\), batch C \((1,2,3)\), and batch D \((1,2,3)\). Calculate the batch means and overall mean.

The means are \((-3-2-1)/3=-2\) for each of A and B, and \((1+2+3)/3=2\) for each of C and D. Across all 12 observations, the total residual is \(3(-2)+3(-2)+3(2)+3(2)=0\), so the overall mean residual is \(0/12=0\).

Despite the zero overall mean, residuals are systematically negative in the first two batches and positive in the last two. This grouping could reflect batch-specific shifts or dependence among measurements within a batch. It is not the same visual signal as a smooth bend across predictor values, and the overall mean alone hides it. The residual pattern does not prove the cause; batch labels and process records are needed to investigate it.

Common Mistakes and a Practical Interpretation

Calling every curve curvature. A widening cloud around a level center points toward changing spread, not necessarily a misspecified mean.
Calling every large residual an outlier to remove. First check the observation and distinguish a valid unusual response from a data error. Do not remove a valid observation just to improve the plot.
Reading a zero overall residual mean as independence. Positive and negative group shifts can cancel, as in the batch example, while within-group dependence remains.
Treating a diagnostic pattern as proof. Small samples, uneven predictor coverage, and estimated fitted values can make patterns uncertain. Use the design and substantive knowledge alongside plots.
Changing the mean curve to solve every problem. A more flexible curve addresses a mean-shape concern; it does not automatically address unequal variance, a data error, or clustered errors.

A useful explanation names the feature that appears to change and the evidence for it. For example: “Residuals remain centered near zero, but their vertical spread is larger at higher settings, suggesting that conditional variance may increase.” This is more precise than saying simply that “the fit is bad.” If the center itself forms a sustained pattern, describe that separately as evidence that the mean curve may be inadequate.

Key takeaway: Separate residual location from residual spread, then inspect isolated observations and grouping. Curvature concerns systematic error in the conditional mean; a fan concerns conditional variance; an extreme point concerns an individual observation; and grouped runs may signal dependence or shared conditions. These clues guide investigation but do not identify a cause on their own.

Check Your Understanding

For each question, state what the residual pattern suggests and what additional caution or check is appropriate.

Residuals are mostly positive at low and high predictor values and mostly negative in the middle. Which aspect of the fitted model is most directly questioned?
Residuals remain centered near zero, but their typical magnitude increases steadily with the predictor. What feature is the main concern?
One observation has a residual of 9 response units while most others are between -2 and 2. Name two possible explanations or checks before deciding what to do with it.
Residuals average to zero overall but are mostly negative in some batches and mostly positive in others. Why does the overall mean not resolve the concern?
Why is a fan-shaped residual plot not, by itself, proof that the mean curve needs more curvature?