Why Curve Choice Needs More Than a Good-Looking Fit
Dose-response data relate a dose, concentration, or exposure level to an outcome. A response may increase at a roughly constant rate, rise quickly and then level off, or bend in another way. Choosing a curve is therefore not just a matter of drawing a line through the points. The choice should reflect the observed pattern, how well candidate curves predict new observations, and what is scientifically plausible.
These sources of evidence can disagree. A flexible curve may fit the observed data closely but predict poorly at held-out doses. A simple curve may predict well but imply an impossible response at high doses. The goal is to compare candidates systematically and explain the trade-offs, not to treat any single plot or score as decisive.
Candidate Curves and the Conditions for Comparing Them
Let \(x_i\) be the dose for observation \(i\), and let \(Y_i\) be its measured response. A candidate regression model describes the conditional mean response as a function of dose, with an error around that mean. Three useful shapes to compare are a straight line, a saturating curve, and a quadratic curve.
For each candidate, one common regression setup is \(Y_i=\mu(x_i)+\varepsilon_i\), where the errors have conditional mean zero. For ordinary least-squares fitting and standard model-based inference, assumptions such as independence and constant error variance also matter; normal errors are often used for small-sample inference. They do not make a particular curve shape correct. For prediction comparisons, the data used to evaluate errors must be held out from fitting, and the training and validation observations should represent the prediction setting of interest.
Before comparing models, check that the dose and outcome are recorded consistently, that the observed dose range is relevant to the intended use, and that observations are not treated as independent when the design makes them dependent. A model comparison cannot repair poor measurement or an unsuitable validation split. In particular, a dose-response pattern in observational data alone does not establish that changing the dose causes the observed response.
A Practical Comparison
Begin with a scatterplot of response against dose, showing the individual observations and the dose range with data support. Overlay the candidate fitted curves on the same axes and in the original outcome units. Then examine residuals—the observed responses minus the fitted responses—for systematic structure. A smooth run of positive residuals followed by negative residuals, for example, can indicate that a straight line misses curvature. A pattern is a clue to investigate, not by itself a final model-selection rule.
Next check each candidate against scientific constraints. Ask whether the response should be nondecreasing, whether it has a known upper or lower bound, and whether a proposed turning point is plausible within the dose range being used. Finally, compare out-of-sample prediction errors using the same outcome scale and loss function. For squared-error loss, a standard summary is root mean squared error (RMSE).
Worked Examples
Worked Example: Reading the Data Pattern and Comparing Validation Errors
Suppose a hypothetical experiment measures response units at five doses. A scatterplot rises throughout the observed range but appears to flatten at higher doses. The investigators fit linear, saturating, and quadratic candidates on training data, then evaluate each at the same five held-out doses. The held-out responses are 5, 9, 13, 16, and 18 units. The prediction errors, defined as observed minus predicted response, are:
| Candidate | Held-out prediction errors (response units) |
|---|---|
| Linear | -1.0, -0.5, 0, 0.5, 1.0 |
| Saturating | -0.4, 0.3, 0.2, -0.2, 0.1 |
| Quadratic | -0.2, 0.1, -0.1, 0.2, -0.1 |
For the linear model, the squared errors sum to \(1.00+0.25+0+0.25+1.00=2.50\). Thus
For the saturating model, the squared errors sum to \(0.16+0.09+0.04+0.04+0.01=0.34\), so its RMSE is \(\sqrt{0.34/5}=\sqrt{0.068}=0.2608\) response units. For the quadratic model, the sum is \(0.04+0.01+0.01+0.04+0.01=0.11\), giving \(\sqrt{0.11/5}=\sqrt{0.022}=0.1483\) response units.
The quadratic has the smallest RMSE on these held-out observations, and the linear model has the largest. This is evidence about prediction for this validation set, not proof that the quadratic is the true dose-response relationship. The apparent flattening in the plot makes the saturating form scientifically worth considering; the investigators should also check whether the quadratic's behavior is plausible over the dose range they intend to use.
Worked Example: Interpreting a Close Validation Comparison
In a second hypothetical analysis, held-out prediction errors from repeated validation are summarized on the same outcome scale. The saturating model has RMSE 2.217 response units, and the quadratic has RMSE 2.250 response units. Their displayed difference is
The saturating model has the lower reported RMSE, but the advantage is only 0.033 response units. That difference should not be inflated into a claim that the saturating model is decisively better. Consider how variable the rankings are across validation splits, whether this difference matters for the prediction task, and whether either curve violates scientific constraints. If both are plausible and their predictive performance is practically similar, a simpler or more interpretable choice may be reasonable. State the evidence and uncertainty in the ranking rather than presenting a close numerical winner as certain.
Worked Example: Checking Saturation and a Polynomial Turning Point
Suppose hypothetical dose-response data show an increasing response that appears to level off. One fitted saturating candidate is \(\mu(x)=2+10x/(3+x)\), with response measured in units and dose \(x\geq0\). It is increasing because its derivative is \(30/(3+x)^2\), which is positive for every allowed dose, and it approaches 12 response units as dose increases. For example, at dose 3 its predicted response is \(2+10(3)/(3+3)=7\) units; at dose 12 it is \(2+120/15=10\) units.
Now consider a hypothetical fitted quadratic, \(\mu(x)=2+1.4x-0.05x^2\). Its derivative is \(1.4-0.1x\), which is zero at dose 14. The quadratic therefore increases below dose 14 and decreases above it. At dose 12 it predicts \(2+16.8-7.2=11.6\) response units, but at dose 30 it predicts \(2+42-45=-1\) response unit. If the response cannot be negative and the dose-response mechanism is expected to level off rather than reverse, that high-dose behavior is scientifically implausible.
The plotted data and validation errors still matter: a scientific preference for saturation does not excuse poor prediction over the observed range. But if the saturating and quadratic candidates predict similarly within that range, the saturating curve may be more defensible when a nondecreasing response and an upper plateau are supported by subject-matter knowledge. The polynomial's unsupported downturn is a reason not to interpret its behavior beyond the data, not evidence that a real downturn occurs.
Justifying the Choice
A disciplined choice follows from combining distinct kinds of evidence. The plot describes the shape and coverage of the observed data. Residual patterns can reveal whether a candidate systematically misses parts of that shape. Scientific constraints rule out forms that imply impossible or unsupported behavior. Validation error estimates how well the fitted candidates predict observations not used to fit them.
These checks answer different questions. A curve can have small training residuals and still predict poorly. A good validation score does not guarantee sensible predictions at doses far outside the validation range. Nor does a plausible scientific form automatically fit the data well. Keep the comparison within the dose range that has support, and describe extrapolation as uncertain when evidence is sparse.
Show the outcome against dose and identify where observations are concentrated or sparse.
Compare their shapes and predictions on common axes and in the original outcome units.
Look for systematic departures, then assess monotonicity, bounds, and turning points against scientific knowledge.
Use the same validation observations and loss measure for each candidate, and treat close rankings cautiously.
Explain which evidence supports the chosen form and where its limitations remain.
Common Mistake: Choosing by One Criterion
A common mistake is to select whichever curve follows the observed points most closely and then assume it is also the best scientific explanation. The wrong reasoning is: “The quadratic bends to match these points, so the response must follow a quadratic law.” A close in-sample fit alone does not establish that claim. The correction is to compare predictions on held-out data, inspect residual patterns, and check plausible behavior over the dose range of interest. A selected regression form is a useful approximation supported by evidence, not proof of the underlying biological mechanism.
Another mistake is to treat a small difference in validation RMSE as decisive without checking its stability or practical relevance. RMSE is tied to a particular validation design and squared-error loss; a different prediction task or loss can change which errors matter most. Report the scale, the comparison, and any limitations so that the choice can be evaluated in context.
Check Your Understanding
Try these questions using the ideas in this tutorial.
- What shape does the saturating model \(\mu(x)=\beta_0+A x/(K+x)\) represent when \(A>0\), \(K>0\), and \(x\geq0\)?
- A candidate has held-out errors of 1, -1, and 2 response units. Write the RMSE calculation and give the result to three significant figures.
- Why can a quadratic curve with the smallest validation RMSE still be a poor scientific choice?
- What might a smooth sequence of positive residuals at low doses and negative residuals at high doses suggest about a fitted straight line?
- Why should a small difference in validation RMSE be reported cautiously rather than treated as proof that one model is superior?