Why concentration-range residuals matter
A saturable exposure–response model can fit observations near the middle of the concentration range while still misrepresenting the low-concentration response or the maximum effect. A single overall fit statistic can hide that structure. Plotting residuals against concentration, with the observed and predicted responses considered together, helps reveal where the model’s curve departs from the data.
This tutorial focuses on diagnosing the shape of a steady exposure–response relationship. It does not assume that a residual pattern proves which biological mechanism is wrong: an apparent Emax shape error can also arise from an incorrect baseline, exposure measurement error, or unaccounted-for subject structure. The diagnostic task is to identify a pattern worth investigating and then check whether the data actually contain information to distinguish competing explanations.
The Emax curve and its shape parameters
For a stimulatory response at concentration \(C\), a common Hill Emax model is
Here \(E_0\) is the response at zero concentration, \(E_{\max}\) is the maximum increase above \(E_0\), \(EC_{50}\) is the concentration producing half of that maximum increase, and \(\gamma\) is the Hill slope. Concentration and \(EC_{50}\) must use the same units; \(E_0\) and \(E_{\max}\) use the response units. For \(C=EC_{50}\), the fraction is \(1/2\), regardless of \(\gamma\). As \(C\) approaches zero the fraction approaches zero, and as \(C\) increases without bound it approaches one.
The simple Emax model is the special case \(\gamma=1\). For an inhibitory response, the same saturating fraction may be subtracted from a baseline rather than added; the residual signs must then be interpreted using that model’s direction of effect. The examples below use a stimulatory response and the raw-residual convention above.
A useful diagnostic separates three questions: is the high-concentration limit wrong, is the curve shifted along the concentration axis, or is the transition between low and high response too gradual or too steep? These correspond mainly to \(E_{\max}\), \(EC_{50}\), and \(\gamma\), respectively. They can be strongly correlated in finite data, so a residual pattern is evidence to examine, not a unique parameter verdict.
Why the parameters leave different signatures
Let \(q=C^\gamma/(EC_{50}^\gamma+C^\gamma)\), the fraction of the maximum increase achieved at concentration \(C\). The local sensitivities of the predicted response show where each parameter has the greatest influence:
The first sensitivity grows from near zero at low concentration toward one at the plateau. Thus, an \(E_{\max}\) error is most visible at high concentrations. The magnitude of sensitivity to \(\log(EC_{50})\) is greatest at \(q=1/2\), which occurs at \(C=EC_{50}\); changing potency shifts the curve most around its middle and has little effect on either asymptote. The slope sensitivity is zero at \(C=EC_{50}\) and changes sign on either side. Changing \(\gamma\) therefore alters the transition’s steepness, with opposite directions of change below and above \(EC_{50}\).
These are local derivatives: they describe the immediate change in predictions for a small parameter change while other parameters are held fixed. In a fitted model, parameters may move together. For example, raising \(E_{\max}\) while raising \(EC_{50}\) can preserve predictions over part of the observed range. This is why the concentration range and residual pattern must be assessed alongside the estimates.
Worked Example: Recognizing an Incorrect Maximum Effect
Suppose hypothetical observations follow a curve with \(E_0=10\) response units, \(E_{\max}=100\) response units, \(EC_{50}=2\) mg/L, and \(\gamma=1\). A fitted model instead uses \(E_{\max}=80\) response units, with the other parameters correct. Calculate the predictions and raw residuals at 1, 2, and 8 mg/L.
At 1 mg/L, the observed-curve value is \(10+100(1/(2+1))=43.33\) response units. The fitted prediction is \(10+80(1/(2+1))=36.67\), so \(r=43.33-36.67=+6.667\) response units.
At 2 mg/L, the observed-curve value is \(10+100(2/(2+2))=60.00\), and the fitted prediction is \(10+80(2/(2+2))=50.00\). The residual is \(+10.00\) response units.
At 8 mg/L, the observed-curve value is \(10+100(8/(2+8))=90.00\), and the fitted prediction is \(10+80(8/(2+8))=74.00\). The residual is \(+16.00\) response units.
The model increasingly underpredicts as concentration approaches the plateau. That is consistent with an underestimated maximum increase when the baseline and other shape parameters are correct. In real data, noise and parameter re-estimation can blur this pattern; it should prompt a check of high-concentration calibration and plateau coverage, not an automatic decision to increase \(E_{\max}\).
Worked Example: Recognizing a Potency Shift
Now let the hypothetical observed curve have \(E_0=10\), \(E_{\max}=80\), \(EC_{50}=2\) mg/L, and \(\gamma=1\). The fitted curve has the correct \(E_0\), \(E_{\max}\), and \(\gamma\), but \(EC_{50}=3\) mg/L. Calculate raw residuals at 0.5, 2, and 8 mg/L.
At 0.5 mg/L, the observed response is \(10+80(0.5/(2+0.5))=26.00\) response units. The fitted response is \(10+80(0.5/(3+0.5))=21.43\), giving \(r=+4.571\) response units.
At 2 mg/L, the observed response is \(10+80(2/(2+2))=50.00\). The fitted response is \(10+80(2/(3+2))=42.00\), giving \(r=+8.000\) response units.
At 8 mg/L, the observed response is \(10+80(8/(2+8))=74.00\). The fitted response is \(10+80(8/(3+8))=68.18\), giving \(r=+5.818\) response units.
Because the fitted \(EC_{50}\) is too high, its curve is shifted to the right and underpredicts these responses. The discrepancy is largest around the middle of the curve and shrinks toward the plateau, where both curves approach the same maximum. With observations spanning both sides of the transition, residuals concentrated around the middle, together with reasonable low and high asymptotes, support investigating a potency shift.
Worked Example: Recognizing a Hill-Slope Error
Suppose the observed curve has \(E_0=10\) response units, \(E_{\max}=80\) response units, \(EC_{50}=2\) mg/L, and \(\gamma=2\). A fitted curve uses the same \(E_0\), \(E_{\max}\), and \(EC_{50}\), but \(\gamma=1\). Find residuals at 1, 2, and 4 mg/L.
At 1 mg/L, the observed fraction is \(1^2/(2^2+1^2)=0.2\), so the observed response is \(10+80(0.2)=26.00\) response units. The fitted fraction is \(1/(2+1)=0.3333\), so its prediction is \(36.67\). Thus \(r=26.00-36.67=-10.67\) response units.
At 2 mg/L, both fractions equal \(1/2\), so both predictions are \(10+80(0.5)=50.00\) response units and \(r=0\).
At 4 mg/L, the observed fraction is \(4^2/(2^2+4^2)=0.8\), giving \(74.00\) response units. The fitted fraction is \(4/(2+4)=0.6667\), giving \(63.33\). Thus \(r=+10.67\) response units.
The fitted curve is too gradual: it predicts too much effect below \(EC_{50}\) and too little above it, with agreement at \(EC_{50}\). This opposite-sign pattern across the transition is consistent with a slope that is too small. It is informative only if concentrations on both sides of the transition are represented; a one-sided concentration range may not reveal the sign change.
When the plateau is not sampled
The maximum effect is the limiting response as concentration becomes very large, not simply the largest observed response. If all concentrations are near or below \(EC_{50}\), the data may not show enough curvature to determine both \(E_{\max}\) and \(EC_{50}\) reliably. In the low-concentration region, when \(C\) is small relative to \(EC_{50}\), the curve is approximately
This approximation makes clear that observations in this region principally inform a combination of parameters, rather than the plateau separately. A good fit in that region does not establish that the estimated \(E_{\max}\) is well determined.
Worked Example: Detecting Weak Plateau Information
Consider two hypothetical curves with \(E_0=10\) response units and \(\gamma=1\). Curve A has \(E_{\max}=80\) response units and \(EC_{50}=2\) mg/L. Curve B has \(E_{\max}=120\) response units and \(EC_{50}=4\) mg/L. Compare their predictions at concentrations that might be the only available observations, 0.5, 1, and 2 mg/L.
At 0.5 mg/L, curve A predicts \(10+80(0.5/(2+0.5))=26.00\); curve B predicts \(10+120(0.5/(4+0.5))=23.33\) response units, a difference of 2.667.
At 1 mg/L, curve A predicts \(10+80(1/3)=36.67\), while curve B predicts \(10+120(1/5)=34.00\), a difference of 2.667 response units. At 2 mg/L, both predict 50.00 response units.
The two curves can therefore look similar over this limited range, despite different maximum effects and potencies. At 8 mg/L, however, curve A predicts \(10+80(8/10)=74.00\), while curve B predicts \(10+120(8/12)=90.00\) response units. A high-concentration observation would help distinguish the predicted plateaus. If the study has no such observations, the correct diagnostic conclusion is that plateau information is weak—not that one of the two maximum effects has been established.
A practical diagnostic sequence
Check the direction of the modeled effect, the baseline, units, and whether the displayed residual is observed minus predicted. A sign convention or response transformation can reverse the apparent pattern.
Show individual observations and, where useful, a smooth trend around zero. Mark the concentration range and predicted \(EC_{50}\). Consider standardized residuals if residual spread changes with concentration.
A discrepancy that persists or grows near the high-concentration end raises concern about the maximum effect. A middle-range discrepancy that diminishes near both ends suggests a potency shift. Opposite residual signs below and above \(EC_{50}\) can indicate a slope error.
Look for observations below, around, and above the transition, including enough high concentrations to assess whether responses level off. If those regions are absent, report the ambiguity rather than treating a parameter estimate as a demonstrated plateau.
Compare predictions and residuals under plausible alternative parameterizations, and check whether baseline, variability, subject clustering, or exposure measurement could explain the same pattern. Do not infer a biological mechanism from a residual plot alone.
Comparisons, diagnostics, and limitations
A residual-versus-concentration plot is direct and easy to interpret, but it is sensitive to uneven sampling: many observations in one region can dominate the visual impression, while a sparse plateau may conceal systematic error. A sensitivity plot or local sensitivity calculation clarifies which concentration regions inform each parameter, but it depends on the model and fitted parameter values. Prediction checks across concentration strata can show whether the fitted model reproduces the observed response distribution in each region; they still cannot create information about an unobserved plateau.
| Candidate issue | Typical raw-residual pattern when the fitted model is wrong | Most informative region |
|---|---|---|
| Maximum effect too low | Positive residuals that remain or increase toward high concentrations | Well-sampled high concentrations showing a plateau |
| Potency too low in the fit (fitted \(EC_{50}\) too high) | Positive residuals around the transition, tending toward zero near the shared asymptotes | Concentrations around the response transition |
| Slope too shallow in the fit | Often negative residuals below \(EC_{50}\), near-zero residuals at \(EC_{50}\), and positive residuals above it | Observations on both sides of \(EC_{50}\) |
| Insufficient plateau sampling | No unique residual signature; different curves may fit the observed range similarly | Additional observations well above the transition |
These signatures assume the baseline and response direction are appropriate, concentrations are measured adequately, and the plotted observations can reasonably be compared under the same exposure–response relationship. Repeated observations from one person are not automatically independent; subject-level structure and serial dependence can make a large cluster of residuals look like stronger evidence than it is. Residual variability may also change with concentration, so a fan-shaped spread should not be mistaken for a mean-curve shape error.
The most common mistake is to call the largest observed response the estimated maximum effect and conclude that any lower fitted asymptote must be wrong. The largest observation is affected by residual variation and need not represent a plateau. The correction is to ask whether responses level off across adequately sampled high concentrations and whether uncertainty in the asymptote is acceptably constrained. If the data do not reach the plateau, report that limitation explicitly.
A second pitfall is to treat a residual pattern as proof of a specific parameter error. Similar patterns can arise from a wrong baseline, a misspecified response scale, or unmodeled population differences. Use concentration-stratified plots alongside subject-level checks and prediction comparisons. A pattern can prioritize what to investigate; it cannot by itself establish a causal explanation or uniquely identify the correct curve.
Check Your Understanding
Answer each question, then compare with the brief solutions.
- Using \(r=Y-\hat{Y}\), what does a positive residual mean?
- A fitted curve has the correct plateau but underpredicts mainly around the middle of the concentration range. Which parameter would you investigate first, and why?
- For a stimulatory Hill Emax curve, what happens to the effect of changing \(\gamma\) at \(C=EC_{50}\)?
- Why can observations only at low concentrations fail to determine \(E_{\max}\) separately from \(EC_{50}\)?
- A fitted curve is too shallow, while the true curve is steeper and has the same \(E_0\), \(E_{\max}\), and \(EC_{50}\). What residual signs are expected below and above \(EC_{50}\)?
Solutions. 1. The model underpredicts: the observed response exceeds its prediction. 2. Investigate \(EC_{50}\), because a potency shift primarily changes predictions around the transition while leaving the asymptotes unchanged. 3. The first-order sensitivity is zero because \(\log(C/EC_{50})=0\); all Hill slopes give one-half of the maximum increase at that concentration. 4. At low concentrations, the response increase is approximately \(E_{\max}(C/EC_{50})^\gamma\), so the data mainly constrain a parameter combination and may not reveal the plateau. 5. Residuals tend to be negative below \(EC_{50}\) and positive above it: the shallow fitted curve predicts too much below the transition and too little above.