Choosing a Curve Is More Than Finding the Closest Fit
When a straight line misses a relationship’s curvature, several nonlinear forms may be reasonable alternatives. A polynomial, a regression spline, or another scientifically motivated curve may each describe the observed data adequately. The choice matters because a curve can fit the current observations well yet behave implausibly, predict poorly for new observations, or be difficult to explain in the terms relevant to the analysis.
There is no single numerical score that settles every comparison. The appropriate choice depends on the analysis goal: explaining a relationship, predicting new outcomes, or communicating a scientifically meaningful pattern. A defensible decision considers residual behavior, subject-matter credibility, performance on data not used to fit the curve, and interpretability together.
A Common Framework for Comparing Candidate Forms
A useful working model is \(Y_i=m(x_i)+\varepsilon_i\), where \(m(x_i)\) is the conditional mean response at predictor value \(x_i\), and \(\varepsilon_i\) is the deviation from that mean. For the usual regression interpretation, errors are assumed to have conditional mean zero, \(E(\varepsilon_i\mid x_i)=0\), and observations are independent. Constant conditional variance is also a common working assumption for standard regression summaries, although comparing candidate forms does not by itself require normally distributed errors.
These assumptions do not guarantee that a particular curve is scientifically appropriate. They clarify what the curve is intended to represent: the mean response, rather than each individual observation. Residuals can reveal that the chosen mean form leaves systematic structure, but they are not a substitute for subject-matter reasoning or evaluation on new data.
Four questions help organize a comparison:
- Residual structure: Do residuals show a systematic pattern that suggests the candidate mean curve misses part of the relationship?
- Scientific credibility: Does the curve’s behavior over the range relevant to the analysis make sense in context?
- Prediction: Does the fitted form predict well for observations that were not used to estimate it?
- Interpretability: Can the curve be explained in a way that supports the purpose of the analysis?
The criteria can disagree. A more flexible form can reduce training error, while a simpler candidate may be easier to explain or may predict better on held-out data. The best-supported choice is therefore not automatically the curve with the smallest training SSE.
Worked Example: Residual Structure Reveals a Poor Linear Description
Suppose a hypothetical experiment records a response \(y\), in response units, at five exposure levels \(x=0,1,2,3,4\), in exposure units. The constructed data are \(y=1,2,5,10,17\). Compare a straight line with a quadratic curve. For this illustration, assume observations at the five levels are independent and that the conditional mean is the feature of interest. With only five constructed observations, those assumptions cannot be established from the data; no inference about a real process is intended.
The sample means are \(\bar{x}=2\) and \(\bar{y}=7\). The least-squares line has slope \(b_1=\sum (x_i-\bar{x})(y_i-\bar{y})/\sum (x_i-\bar{x})^2\). Here the numerator is \(12+5+0+3+20=40\), and the denominator is \(4+1+0+1+4=10\). Thus \(b_1=4\), and the intercept is \(b_0=\bar{y}-b_1\bar{x}=7-4(2)=-1\). The fitted line is \(\hat{y}=-1+4x\).
At \(x=0,1,2,3,4\), its fitted values are \(-1,3,7,11,15\). The residuals, observed minus fitted, are \(2,-1,-2,-1,2\). Therefore,
The residuals are mostly positive at the ends and negative in the middle, rather than scattered without an obvious pattern around zero. The quadratic \(m(x)=1+x^2\) matches all five constructed values, so its training SSE is zero. In these data, curvature clearly matters to the description; the line leaves a pronounced pattern. But the exact quadratic fit is not proof that a real relationship follows this formula. The example demonstrates how residual structure can challenge a candidate form, not how to establish a scientific law.
Scientific Credibility and Interpretability
A candidate curve should be assessed over the range relevant to the question, in the units and context of the variables. Ask whether its direction and overall shape are scientifically credible there, and whether its parameters or fitted behavior can be communicated accurately. A curve that fits observed points closely may still represent a relationship in a way that conflicts with reliable subject knowledge.
Interpretability is purpose-dependent. For an explanatory analysis, a compact description of how the mean response changes may be more useful than a curve with many local adjustments. For prediction, a less direct parameterization might still be useful if it predicts accurately and its limitations are clear. In either case, do not describe a nonlinear coefficient as though it were automatically a constant slope: its meaning depends on the form and on the other terms in the model.
Worked Example: A Tiny Training Improvement May Not Change the Scientific Summary
Imagine a hypothetical dose-response analysis in which \(x\) is dose in milligrams and \(y\) is a measured response in response units. Subject knowledge supports a smooth, gradually accelerating increase over the observed dose range. Assume independent observations with conditional mean zero errors, and suppose measurement variation is about one response unit. Two fitted candidates give the following rounded training predictions:
| Dose \(x\) (mg) | Observed \(y\) | Quadratic fit | Flexible fit |
|---|---|---|---|
| 0 | 2.0 | 2.1 | 2.0 |
| 1 | 4.0 | 3.8 | 4.0 |
| 2 | 7.0 | 7.2 | 7.0 |
| 3 | 11.0 | 10.8 | 11.0 |
| 4 | 16.0 | 16.1 | 16.0 |
For the quadratic fit, the residuals are \(-0.1,0.2,-0.2,0.2,-0.1\) response units. Its training SSE is \(0.01+0.04+0.04+0.04+0.01=0.14\) response-units squared. The flexible fit has rounded residuals of zero at these five points, so its displayed training SSE is zero. The small advantage in training fit is \(0.14\) response-units squared; its square-root scale is \(\sqrt{0.14/5}\approx0.167\) response units per observation, smaller than the stated one-unit measurement variation.
Both fitted summaries are compatible with the stated gradual increase over the observed range. If the goal is to describe smooth acceleration compactly, the quadratic may communicate the pattern more directly; its second-degree term represents curvature, not a constant slope. The displayed data alone do not show that the flexible curve is scientifically wrong, nor do they establish that the quadratic is correct. The decision should reflect the analysis goal and the subject knowledge, not a claim that zero training error settles the comparison.
Use New Data to Assess Prediction
When the goal is prediction, assess candidate curves on observations not used to estimate them. For \(n_{\mathrm{test}}\) held-out observations, the test mean squared error is the average squared difference between each held-out response and the prediction made without using that response to fit the curve. It is expressed in squared response units; its square root is on the response scale.
Worked Example: Comparing Two Curves on Held-Out Observations
Suppose two candidate forms have been fitted using a training set in a hypothetical study. Four new, independent observations from the intended prediction setting are held out; assume their response scale and measurement process match the training data. Their observed responses are \(4,7,11,16\) response units. Candidate A predicts \(5,7,9,11\); candidate B predicts \(3,7,12,18\). Neither candidate was fitted using these four responses.
For candidate A, observed minus predicted values are \(-1,0,2,5\) response units. Thus
For candidate B, the differences are \(1,0,-1,-2\), giving
Candidate B has the lower test MSE on this held-out set. Its test root mean squared error is \(\sqrt{1.5}\approx1.225\) response units, compared with \(\sqrt{7.5}\approx2.739\) for candidate A. These results favor B for prediction in settings like the one represented by the held-out observations. Four observations provide limited evidence, however; the comparison does not guarantee that B will perform better on every future sample or under a different prediction setting.
A Short Justification: Training Fit and Model Flexibility
A basic mathematical fact helps explain why training SSE should not be the only selection criterion. Let \(\mathcal{M}_1\) and \(\mathcal{M}_2\) be two candidate sets of curves, with \(\mathcal{M}_1\subseteq\mathcal{M}_2\). Define their best achievable training losses using infima, so the statement remains valid even if a best-fitting curve is not attained:
To see why, every curve \(m\) in \(\mathcal{M}_1\) is also in \(\mathcal{M}_2\). Therefore the infimum over \(\mathcal{M}_2\) is no greater than the loss for any particular \(m\in\mathcal{M}_1\). It is consequently a lower bound for all losses in \(\mathcal{M}_1\), and cannot exceed their infimum. If both best-fitting curves exist, the infima are minima and the same inequality applies to those minima.
This result says only that enlarging a candidate set cannot worsen its best achievable training SSE. It does not say that the added flexibility improves prediction, scientific credibility, or communication. Those questions require the other criteria.
Common Mistake and a Practical Decision
A common mistake is to reason, “This curve has the smallest training SSE, so it is the best curve.” The correction is that training SSE measures closeness to the data used for fitting; it does not assess prediction on new data or whether the form makes scientific sense. A second mistake is to select a candidate from a single validation score without considering whether the held-out data represent the intended setting. Treat a score as evidence for a particular goal and target population, not as a universal ranking.
Decide whether the main purpose is explaining the mean relationship, predicting new outcomes, or both.
Use residual behavior to notice whether a candidate leaves systematic structure, and ask whether its shape is scientifically credible over the relevant range.
Compare predictions on observations kept out of fitting, using data that represent the intended prediction setting.
State which criteria support the selected form, which limitations remain, and why the choice matches the analysis goal.
A sound conclusion need not claim that one curve is best in every respect. It should say which candidate is better supported for the stated purpose and acknowledge relevant trade-offs. This makes the choice transparent and prevents a convenient fit statistic from standing in for scientific judgment.
Check Your Understanding
Use the criteria and definitions in this tutorial to answer the following.
- Why can a candidate with lower training SSE still be a poor choice for prediction?
- What does a systematic pattern in residuals suggest about a fitted mean curve?
- In a held-out prediction comparison, why must the held-out responses not be used to fit the candidate curves?
- Two candidate forms have similar predictive performance. What additional questions about scientific credibility and interpretability could help distinguish them?
- Explain why set inclusion guarantees an inequality for the infimum of training losses even when a minimum may not exist.