Using Out-of-Sample Error to Choose Curve Shape

Why Check a Curve on Data It Has Not Seen?

A flexible curve can follow bends in the data used to fit it, including bends caused by random variation. Its training error may therefore be smaller than that of a simpler curve even when it does not predict new observations better. To assess whether an apparent gain persists beyond the fitting data, compare the candidate curves on observations that played no part in fitting them.

This comparison is useful when several plausible functional forms have been proposed. It gives evidence about prediction for new observations from a specified target population. It does not, by itself, prove that one curve is the true relationship or explain why the relationship has its shape.

Training Error and Out-of-Sample Error

Definition: A training set is the data used to fit a regression curve. A validation set is a separate set of observations withheld from fitting and used to compare candidate curves. For a fitted curve \(\hat f\), the squared prediction error for an observation with predictor value \(x_i\) and outcome \(y_i\) is \((y_i-\hat f(x_i))^2\). The validation mean squared error is the average of these squared errors over the validation set.

For \(m\) validation observations, write the validation mean squared error as

\mathrm{MSE}_{\mathrm{val}}=\frac{1}{m}\sum_{i=1}^{m}\left(y_i-\hat f(x_i)\right)^2.

The root mean squared error is \(\mathrm{RMSE}_{\mathrm{val}}=\sqrt{\mathrm{MSE}_{\mathrm{val}}}\). MSE is expressed in squared outcome units, while RMSE is expressed in the original outcome units. With squared-error loss, the curve with the smaller validation MSE also has the smaller validation RMSE. The MSE is convenient for calculation; the RMSE is often easier to interpret in context.

Training MSE is calculated in the same way, but using the observations that fitted the curve. It describes how closely the curve fits those particular data. Validation MSE measures its errors on held-out observations. A curve with lower training MSE but higher validation MSE has fit the training sample more closely without showing better prediction on the validation sample.

Conditions for a Fair Comparison

A validation score is informative only if the validation observations represent the prediction setting of interest. The central conditions are that the validation outcomes were not used to fit or tune the curves, and that the validation observations reflect the population and circumstances for which predictions are intended. In the usual random-split setting, observations are treated as independent draws from the same target population. If the intended predictions concern later time periods, for example, a split that respects time order may be more appropriate than a random split.

To isolate the choice of curve shape, fit each candidate using the same training set and evaluate each on the same validation set. Use the same outcome scale and the same error measure for every candidate. Any data-dependent decisions, such as estimating knot locations or selecting predictors, must be made without using validation outcomes. Otherwise, the validation set has influenced the curve being evaluated and is no longer a clean check on its prediction error.

No normality assumption is needed merely to calculate and compare these squared prediction errors. The scores are descriptive estimates of predictive performance, not hypothesis tests. Their interpretation depends on the validation data being relevant to the future prediction task.

Why Held-Out Error Estimates Prediction Performance

The justification can be stated directly. Let \(D\) denote the training data, so that after fitting, \(\hat f_D\) is fixed. Suppose the validation observations \((X_i,Y_i)\) are independent draws from the same target population and were not used to construct \(\hat f_D\). Conditional on the fitted curve, each validation squared error has the same expected value: the expected squared prediction error for a new observation from that population.

\begin{aligned} E\left[\mathrm{MSE}_{\mathrm{val}}\mid D\right] &=E\left[\frac{1}{m}\sum_{i=1}^{m}\left(Y_i-\hat f_D(X_i)\right)^2\mid D\right]\\ &=\frac{1}{m}\sum_{i=1}^{m}E\left[\left(Y_i-\hat f_D(X_i)\right)^2\mid D\right]\\ &=E\left[\left(Y-\hat f_D(X)\right)^2\mid D\right]. \end{aligned}

The last equality holds because each validation observation follows the same target distribution. Thus, conditional on the training data, validation MSE estimates the prediction error of the curve fitted on that training set. It is not a guarantee that the score equals future error exactly: a finite validation set can give a noisy estimate. Nor does it automatically describe a curve refitted on a different amount of data or used in a different population.

Worked Example: A Lower Training Error Does Not Guarantee Better Predictions

Suppose two hypothetical curves were fitted to the same training set to predict a chemical concentration in milligrams per liter. The training MSE was \(1.70\) for a linear curve and \(0.80\) for a quadratic curve. On a separate validation set, the observed concentrations were \(21\), \(22\), and \(23\) mg/L at predictor values \(5\), \(6\), and \(7\), respectively. The fitted curves predict \(\hat y=10+2x\) and \(\hat y=10+2x+0.5(x-2)^2\).

For the linear curve, the predictions at \(x=5,6,7\) are \(20,22,24\) mg/L. Its validation squared errors are \((21-20)^2=1\), \((22-22)^2=0\), and \((23-24)^2=1\). Therefore,

\mathrm{MSE}_{\mathrm{val,linear}}=\frac{1+0+1}{3}=0.6667\;(\mathrm{mg/L})^2, \qquad \mathrm{RMSE}_{\mathrm{val,linear}}=\sqrt{0.6667}=0.8165\;\mathrm{mg/L}.

For the quadratic curve, the predictions are \(24.5\), \(30.0\), and \(36.5\) mg/L. Its squared errors are \((21-24.5)^2=12.25\), \((22-30)^2=64\), and \((23-36.5)^2=182.25\). Thus,

\mathrm{MSE}_{\mathrm{val,quadratic}}=\frac{12.25+64+182.25}{3}=86.17\;(\mathrm{mg/L})^2, \qquad \mathrm{RMSE}_{\mathrm{val,quadratic}}=\sqrt{86.17}=9.282\;\mathrm{mg/L}.

The quadratic curve had the smaller training MSE, but on these held-out observations its prediction errors were much larger. For squared-error prediction in this setting, the validation results favor the linear curve. This is evidence about prediction, not proof that the underlying relationship is linear.

Compare Every Candidate on the Same Validation Set

The comparison should be made as a controlled calculation: keep the observations and scoring rule fixed, and vary the candidate curve. The following example uses a hypothetical validation set of four outcomes measured in mg/L. Curve A is a more flexible candidate than curve B. Their training MSEs are \(0.90\) and \(1.20\;(\mathrm{mg/L})^2\), respectively.

Worked Example: Calculating and Comparing Validation MSE

The observed validation outcomes are \(10,14,16,20\) mg/L. Curve A predicts \(12,13,18,18\) mg/L; curve B predicts \(11,15,17,19\) mg/L. For curve A, the errors are \(-2,1,-2,2\) mg/L, so the squared errors sum to \(4+1+4+4=13\;(\mathrm{mg/L})^2\). Its validation MSE is \(13/4=3.25\;(\mathrm{mg/L})^2\), and its RMSE is \(\sqrt{3.25}=1.803\) mg/L.

For curve B, the errors are \(-1,-1,-1,1\) mg/L. Its squared errors sum to \(1+1+1+1=4\;(\mathrm{mg/L})^2\), giving

\mathrm{MSE}_{\mathrm{val,B}}=\frac{4}{4}=1.00\;(\mathrm{mg/L})^2, \qquad \mathrm{RMSE}_{\mathrm{val,B}}=\sqrt{1.00}=1.000\;\mathrm{mg/L}.

Curve B has the higher training MSE but the lower validation MSE. If the goal is to choose between these candidates for squared-error prediction under the conditions represented by the validation set, curve B is preferred by this comparison. The difference in validation scores is not a formal significance test, and it does not guarantee that curve B will perform better on every future sample.

Selection and Final Evaluation Are Different Tasks

Once validation outcomes have been used to choose a curve, the validation score is useful for selection but should not be treated as a fully independent final assessment of the selected curve. Choosing the smallest of several validation errors tends to favor a candidate that happened to score well on that particular set. If an unbiased final evaluation is needed, keep a separate test set aside: do not use its outcomes to fit curves, tune them, or choose among them. Evaluate the selected procedure on that test set only after the selection decisions are complete.

Worked Example: Why Reusing the Selection Set Can Mislead

Consider three hypothetical candidate curves, A, B, and C, compared using squared-error loss. On a two-observation validation set, their squared errors are as follows:

Candidate	Validation squared errors	Validation MSE
A	1, 1	1.00
B	0, 4	2.00
C	1, 4	2.50

The arithmetic gives \( \mathrm{MSE}_{\mathrm{val,A}}=(1+1)/2=1.00\), \( \mathrm{MSE}_{\mathrm{val,B}}=(0+4)/2=2.00\), and \( \mathrm{MSE}_{\mathrm{val,C}}=(1+4)/2=2.50\). Curve A is selected because it has the smallest validation MSE.

Suppose a separate, untouched test set later gives MSEs of \(9.00\) for A, \(1.00\) for B, and \(4.00\) for C. These test scores do not change the fact that A was the validation-selected curve; they show that its validation advantage did not carry over to this test set. If the test set were now used to switch the selection to B, its score would no longer be an independent final assessment. This small example illustrates the roles of the sets; its two-observation validation sample is not a recommendation for a sample size.

Common Mistake: Choosing by Training Error Alone

Wrong reasoning: “The flexible curve has the smallest training error, so it must be the best curve.”
Correction: Training error describes fit to the observations used to estimate the curve. Compare candidate curves on observations withheld from fitting to see whether the gain persists out of sample.
Wrong reasoning: “I used the validation outcomes to decide which curve to fit, but the validation score is still an independent check.”
Correction: If the outcomes influenced fitting or tuning, they are no longer untouched validation data. Use data not involved in those choices for an independent final evaluation.

Key takeaway: For a fair comparison of candidate curve shapes, fit them using the same training data and calculate the same prediction loss on the same untouched validation data. A lower held-out error supports better prediction for the represented setting; it does not establish a true functional form or guarantee future performance.

Check Your Understanding

Answer each question using the definitions and procedures in this tutorial.

A curve has training MSE \(2.1\) and validation MSE \(3.4\), both in squared outcome units. What does each score describe?
For validation errors of \(-2,1,0\) outcome units, calculate the validation MSE and RMSE.
Why should candidate curves be compared on the same validation observations and with the same error measure?
A researcher uses validation outcomes to choose knot locations. Can the resulting validation score still be treated as an untouched check? Explain.
After choosing a curve using validation MSE, what is the purpose of evaluating it on a separate test set?