Why Examine Residuals?
A fitted concentration-effect curve summarizes how the model predicts response across concentrations. Its parameter estimates do not, by themselves, show whether the curve captures the observed pattern or whether the assumed residual error is plausible. Residual plots help assess these questions by displaying the differences between observed responses and fitted responses. They are diagnostic tools: a pattern can point to a problem worth investigating, but usually does not identify its cause uniquely.
For a direct-effect fit, the predicted response at concentration \(C_i\) can be written \( \hat{Y}_i=f(C_i;\hat{\theta}) \), where \(f\) is the chosen concentration-effect function and \(\hat{\theta}\) denotes its estimated parameters. For example, an Emax or sigmoid Emax function may be used for \(f\). The same diagnostic ideas apply to a linear concentration-effect function. The plots do not decide which model should be selected; their purpose here is to check how a fitted model behaves against the observations.
What the Plots Should Show
The basic reference pattern is a roughly even scatter of residuals around zero, without a systematic trend. This expectation follows from the model assumption that the conditional mean is correctly specified: at a given concentration, the average response across repeated observations is \(f(C_i;\theta)\). If that is so, the model error has conditional mean zero. A persistent run of positive or negative residuals across a range of concentrations is therefore a clue that the fitted mean curve may not capture the data pattern.
In practice, residuals use estimated parameters, so they are not independent observations of the model errors: fitting the curve has already used the responses, and parameter estimation can constrain the residuals. The reference pattern is consequently approximate, especially in small data sets. A residual plot is not a formal test that the model is true.
A residual-versus-concentration plot places \(C_i\) on the horizontal axis and \(e_i\), or \(r_i\) when appropriate, on the vertical axis. It is particularly useful for seeing whether discrepancies change across the concentration range. A residual-versus-fitted-value plot uses \(\hat{Y}_i\) on the horizontal axis. It can make changes in residual spread easier to see when variability is related to response magnitude. Since fitted values are determined by concentration in a simple direct-effect fit, these plots can look similar; neither replaces the other in every data set.
A normal quantile plot compares the ordered residuals with the quantiles expected from a normal distribution. It is most directly interpretable for variance-standardized residuals when a Gaussian residual-error assumption is being used. Points roughly along a straight line are compatible with normality; systematic curvature or a point far from the rest can signal departures worth checking. The plot does not establish normality, and mild departures do not automatically make every estimate unusable.
Reading Patterns Carefully
| Pattern in residuals | Possible diagnostic clue | What it does not prove |
|---|---|---|
| Curved or changing runs across concentration | The mean concentration-effect shape may not capture the observed pattern. | It does not identify which parameter or alternative model is responsible. |
| Fan-shaped spread as fitted response increases | The assumed residual variance may not match the data. | It does not establish a particular additive or proportional error model. |
| One unusually large residual | An observation may be influential or may reflect a recording, assay, or model issue. | It is not a justification for deleting the observation. |
| Same-sign residuals at high concentrations | The fitted curve may not capture the high-concentration response or plateau. | It does not prove that a specific plateau parameter is wrong. |
For an Emax-type curve, systematic high-concentration residuals deserve attention because the curve approaches a limiting response as concentration becomes large. If observations continue to rise while fitted values have flattened, high-concentration residuals may tend to be positive. If the fitted plateau is above the observed responses, they may tend to be negative. However, this interpretation is most useful when the data actually cover the high-concentration region. If all concentrations are below the apparent plateau, residuals cannot establish whether the limiting response has been adequately characterized.
Worked Examples
Worked Example: A Concentration-Dependent Residual Pattern
Suppose a hypothetical direct-effect fit predicts responses at five concentrations as shown below. Response is measured in arbitrary units and concentration in mg/L. The table gives the fitted values and observations; use \(e_i=Y_i-\hat{Y}_i\).
| Concentration \(C_i\) (mg/L) | Fitted response \(\hat{Y}_i\) | Observed response \(Y_i\) | Residual \(e_i\) |
|---|---|---|---|
| 0 | 10 | 10 | 0 |
| 1 | 20 | 22 | 2 |
| 2 | 30 | 31 | 1 |
| 4 | 40 | 35 | -5 |
| 8 | 45 | 40 | -5 |
For example, at \(C=4\) mg/L, \(e=35-40=-5\) response units. Reading the residuals in concentration order gives \(0,2,1,-5,-5\). They are mostly positive at the lower and middle concentrations and negative at the higher concentrations, rather than appearing as an unstructured scatter around zero. This is a clue that the fitted mean shape may miss a concentration-dependent pattern. The residuals alone do not establish whether the curve is too steep, has an inappropriate plateau, or has some other form of misspecification; the observed data and model assumptions require further scrutiny.
Worked Example: Distinguishing a Variance Pattern from a Mean Pattern
Consider hypothetical fitted responses of 10, 20, 40, and 80 response units, with residuals \(1,-2,4,-8\). The absolute raw residuals grow with fitted response, which could look like a fan in a residual-versus-fitted plot. Suppose, for illustration, that the assumed error model has additive standard deviation \(\sigma_a=1\) response unit and proportional standard deviation \(\sigma_p=0.10\), with conditional variance \(v_i=\sigma_a^2+(\sigma_p\hat{Y}_i)^2\).
At fitted response 10, the standard deviation is \(\sqrt{1^2+(0.10\times10)^2}=\sqrt{2}=1.414\), so \(r=1/1.414=0.707\). At 20, it is \(\sqrt{1+4}=\sqrt{5}=2.236\), so \(r=-2/2.236=-0.894\). At 40, it is \(\sqrt{1+16}=\sqrt{17}=4.123\), so \(r=4/4.123=0.970\). At 80, it is \(\sqrt{1+64}=\sqrt{65}=8.062\), so \(r=-8/8.062=-0.992\).
The raw discrepancies increase in magnitude, but the standardized residuals are similar in magnitude. Under the stated variance model, this pattern is less suggestive of an increasingly unusual discrepancy at high fitted responses. This calculation does not verify the assumed variance model: it shows why raw and standardized residual plots answer related but different questions. The variance form and its parameters must be justified for the analysis, not chosen solely because one plot looks more even.
Worked Example: Residuals Suggesting an Inadequate Plateau
Suppose a hypothetical fitted curve predicts 80 response units at each of four high concentrations: 20, 40, 80, and 160 mg/L. Observed responses are 89, 91, 88, and 90 units. The residuals are \(89-80=9\), \(91-80=11\), \(88-80=8\), and \(90-80=10\) response units. Their mean is \((9+11+8+10)/4=9.5\) response units.
All four residuals are positive, so the fit underpredicts these high-concentration observations. If lower-concentration residuals were scattered around zero, this repeated high-concentration pattern would be consistent with a curve that does not capture the high-concentration response adequately, including a possible plateau that is too low. It is a diagnostic clue, not proof of a particular parameter error: measurement issues, an unmodeled response feature, or other misspecification could also produce it. It is also important to check that the high concentrations are supported by observations rather than relying on extrapolation.
Worked Example: Inspecting a Normal Quantile Plot
Suppose a hypothetical fit produces eight variance-standardized residuals, ordered from smallest to largest: \(-2.1,-1.2,-0.4,0.1,0.3,0.5,1.1,2.7\). For a simple normal quantile plot, compare these with standard normal quantiles at plotting positions \((i-0.5)/8\), for \(i=1,\ldots,8\). Approximate expected quantiles are \(-1.53,-0.89,-0.49,-0.16,0.16,0.49,0.89,1.53\).
Most ordered values are reasonably close to their corresponding expected values, but the largest residual, 2.7, is notably above 1.53. This suggests an unusually large upper-tail discrepancy relative to the assumed Gaussian error model. With only eight residuals, the plot is imprecise; the point should prompt checks of the observation and model, not an automatic deletion or a definitive conclusion that the errors are nonnormal.
Assumptions, Diagnostics, and Alternatives
The interpretation of these plots depends on the assumptions being checked. For mean-shape diagnostics, the fitted function should represent the conditional mean response, concentrations and responses should be correctly paired and measured, and observations should be treated in a way consistent with the fitting model. For variance-standardized plots, the assumed conditional variance \(v_i\) must be appropriate. A normal quantile plot has its usual interpretation when the conditional residuals are assumed to be approximately Gaussian after standardization. If the data are repeated or otherwise dependent, a simple independent-residual interpretation may not apply.
A useful short justification for looking for zero-centered patterns is that, under the specified model, \(E(Y_i\mid C_i)=f(C_i;\theta)\). If the true parameter value were known, the error \(Y_i-f(C_i;\theta)\) would therefore have conditional expectation zero. A systematic trend in residuals against concentration is evidence to investigate because it conflicts with that expected pattern. In an actual fit, \(\theta\) is estimated from the same data, so residuals are not exactly the unobserved errors; the diagnostic is a practical approximation rather than a proof.
Residual plots complement, rather than replace, other checks. A raw residual plot preserves discrepancies in response units and is easy to interpret. A standardized residual plot accounts for a specified variance pattern and can make observations comparable when their expected error scales differ. A normal quantile plot focuses on distributional shape, not the mean curve across concentration. Formal goodness-of-fit procedures may summarize evidence differently, but they also depend on assumptions and can be insensitive to specific patterns. No single display can diagnose every failure.
Plot residuals against concentration and fitted response. Look for smooth trends, runs of one sign, or systematic departures near high concentrations.
Look for changing spread. Compare raw residuals with residuals standardized using the variance model actually assumed for the fit.
Confirm the data and measurement context, and assess whether the point is influential. Do not remove it simply because its residual is large.
Describe what the pattern suggests and what remains uncertain. A plot indicates possible model failure; it does not uniquely identify its cause.
A common mistake is to treat a residual plot with no obvious pattern as proof that the model is correct. The correction is that visual checks have limited power, particularly with few observations or sparse concentration coverage. Conversely, one unusual residual is not sufficient reason to discard an observation. Check data integrity and context, and consider how conclusions change with the point included and excluded only as a sensitivity analysis. Finally, a fitted curve that looks smooth is not necessarily adequate: systematic residual patterns can remain hidden when only the observed and fitted response curves are viewed.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- An observation has response 54 units and fitted response 60 units. What is its raw residual, and what does its sign mean?
- Residuals are near zero at low concentrations but mostly positive at the highest concentrations. Give one plausible diagnostic interpretation and one reason not to treat it as proof.
- Raw residuals grow in magnitude with fitted response, but standardized residuals using the stated variance model show similar spread across the range. What does this suggest?
- What assumption makes a normal quantile plot of standardized residuals useful, and what should a single extreme point prompt?
Solutions. (1) \(e=54-60=-6\) response units; the model overpredicts this observation by 6 units. (2) The curve may underpredict the high-concentration response, possibly because it does not capture the plateau. Other data or model problems could produce the same pattern, so the plot does not identify the cause uniquely. (3) The raw spread may increase with response while remaining broadly consistent with the specified variance scaling; this is a diagnostic clue, not confirmation that the variance model is correct. (4) The conditional residuals are assumed to be approximately Gaussian after standardization. An extreme point should prompt checks of the observation and model, not automatic deletion.