Leave-One-Out Cross-Validation

Why Leave One Observation Out?

A single train-test split can give a useful estimate of prediction error, but its result may depend on which observations happened to land in the test set. Cross-validation addresses this by evaluating a procedure on multiple held-out subsets. Leave-one-out cross-validation (LOOCV) takes this idea to the limit: each observation is held out in turn, and the model is fitted using all the others.

LOOCV is appealing when the data set is small because each fit uses nearly all available observations. The trade-off is that it requires one fit per observation. Its estimate can also be sensitive to individual cases, so using the largest possible training set in each fit does not automatically make LOOCV the most stable choice.

Definition: For \(n\) observations, LOOCV forms \(n\) training sets. Training set \(i\) contains every observation except observation \(i\); observation \(i\) is used only to evaluate the prediction from that fit. If \(\hat{y}_{(-i)}\) is the prediction for the held-out outcome \(y_i\), the held-out error is \(e_{(-i)}=y_i-\hat{y}_{(-i)}\). The LOOCV mean squared error is \(\operatorname{LOOCV\ MSE}=\frac{1}{n}\sum_{i=1}^n e_{(-i)}^2\), and the LOOCV root mean squared error is its square root.

What the Estimate Measures

The procedure being evaluated includes the rule used to fit the regression model and produce predictions. For example, the procedure might be ordinary least squares with a specified predictor and outcome. In each LOOCV round, the procedure is applied to the \(n-1\) training observations, then used to predict the omitted outcome. Each observation contributes exactly one held-out error.

A predictive interpretation needs the observed cases to be relevant to the future cases of interest. In particular, the observations should represent the same prediction setting, and one observation should not improperly reveal information about another held-out case. These are requirements of the data and evaluation design, not a special property guaranteed by the LOOCV calculation. For ordinary least-squares regression, each training set must also permit the requested fit; for simple linear regression, that means its training predictor values cannot all be identical.

LOOCV MSE estimates average squared prediction error for the procedure trained on \(n-1\) observations from the available data. It is not the training error, and it is not exactly the error of a final model fitted on all \(n\) observations. The RMSE is in the outcome's units, while MSE is in squared outcome units. For example, if the outcome is measured in minutes, RMSE is in minutes and MSE is in minutes squared.

Worked Example: Hand-Calculating LOOCV for Simple Linear Regression

Suppose a hypothetical data set records four pairs: \((x,y)=(1,2),(2,3),(3,5),(4,6)\), where \(x\) is a study-duration measure in hours and \(y\) is a score in points. Use simple linear regression with an intercept and calculate LOOCV MSE and RMSE. Each training set has three distinct predictor values, so all four least-squares fits are defined.

For each fit, the slope is \(b_1=S_{xy}/S_{xx}\), where \(S_{xy}=\sum (x_j-\bar{x})(y_j-\bar{y})\) and \(S_{xx}=\sum(x_j-\bar{x})^2\), with the sums taken over that fit's training cases. The intercept is \(b_0=\bar{y}-b_1\bar{x}\). The held-out prediction is \(\hat{y}_{(-i)}=b_0+b_1x_i\).

Omit case 1. The training means are \(\bar{x}=3\) and \(\bar{y}=14/3\). Here \(S_{xy}=3\) and \(S_{xx}=2\), so \(b_1=3/2\) and \(b_0=14/3-(3/2)(3)=1/6\). At \(x_1=1\), the prediction is \(1/6+3/2=5/3\), giving error \(2-5/3=1/3\).

Omit case 2. For the training pairs \((1,2),(3,5),(4,6)\), \(S_{xy}=19/3\) and \(S_{xx}=14/3\), so \(b_1=19/14\). The means are \(\bar{x}=8/3\) and \(\bar{y}=13/3\), giving \(b_0=13/3-(19/14)(8/3)=5/7\). The prediction at \(x_2=2\) is \(5/7+(19/14)(2)=24/7\), so the error is \(3-24/7=-3/7\).

Omit case 3. For \((1,2),(2,3),(4,6)\), \(S_{xy}=19/3\), \(S_{xx}=14/3\), \(\bar{x}=7/3\), and \(\bar{y}=11/3\). Thus \(b_1=19/14\) and \(b_0=11/3-(19/14)(7/3)=1/2\). The prediction at \(x_3=3\) is \(1/2+(19/14)(3)=32/7\), so the error is \(5-32/7=3/7\).

Omit case 4. For \((1,2),(2,3),(3,5)\), the training means are \(\bar{x}=2\) and \(\bar{y}=10/3\), with \(S_{xy}=3\) and \(S_{xx}=2\). Therefore \(b_1=3/2\) and \(b_0=1/3\). The prediction at \(x_4=4\) is \(1/3+(3/2)(4)=19/3\), so the error is \(6-19/3=-1/3\).

The four squared errors are \(1/9,9/49,9/49,1/9\). Therefore,

\operatorname{LOOCV\ MSE} =\frac{1}{4}\left(\frac{1}{9}+\frac{9}{49}+\frac{9}{49}+\frac{1}{9}\right) =\frac{1}{4}\left(\frac{2}{9}+\frac{18}{49}\right) =\frac{65}{441} \approx 0.1474.

The RMSE is \(\sqrt{65/441}\approx 0.3839\) points. In this hypothetical example, the estimated typical prediction error on the score scale is about \(0.384\) points, according to this RMSE summary. This is an evaluation of the fitting procedure, not a guarantee about any one future prediction.

Calculating the Held-Out Error Summary

The calculation follows the same pattern no matter which regression procedure is being evaluated: refit without one case, predict that case, record the error, and repeat until every case has been held out. Then calculate the chosen summary from those \(n\) errors. Squaring prevents positive and negative errors from cancelling in MSE, while taking the square root returns the summary to the outcome scale.

Another common measure is mean absolute error, \(\operatorname{LOOCV\ MAE}=\frac{1}{n}\sum_{i=1}^n |e_{(-i)}|\). It summarizes absolute prediction errors in the outcome's units. MSE penalizes larger errors more heavily because errors are squared. The evaluation measure should match the purpose of the prediction task; one measure is not universally best.

Worked Example: LOOCV for an Intercept-Only Regression

Suppose a hypothetical set of four outcomes, measured in points, is \(2,4,4,6\). Use an intercept-only regression, which predicts the mean outcome in its training data. This model has no predictor, and each fit is defined as long as its training set has observations.

When the outcome \(2\) is held out, the training mean is \((4+4+6)/3=14/3\), so the error is \(2-14/3=-8/3\). Holding out the first \(4\) gives prediction \((2+4+6)/3=4\) and error \(0\). Holding out the second \(4\) also gives prediction \((2+4+6)/3=4\) and error \(0\). Finally, holding out \(6\) gives prediction \((2+4+4)/3=10/3\) and error \(6-10/3=8/3\).

The squared errors are \(64/9,0,0,64/9\), so

\operatorname{LOOCV\ MSE} =\frac{1}{4}\left(\frac{64}{9}+0+0+\frac{64}{9}\right) =\frac{32}{9} \approx 3.556, \qquad \operatorname{LOOCV\ RMSE} =\sqrt{\frac{32}{9}} \approx 1.886\text{ points}.

The result illustrates why the held-out prediction must use only the training outcomes. For example, when \(2\) is held out, the prediction is the mean of the other three observations, \(14/3\), not the mean of all four outcomes.

A Short Justification of the LOOCV Formula

The target of a prediction procedure is its average loss on future observations from the prediction setting of interest. Squared error for a case is the square of its observed outcome minus the prediction made for it. In LOOCV, each available case takes its turn as that evaluation case, and the remaining cases provide the training data. Averaging the resulting \(n\) squared errors gives the formula directly:

\operatorname{LOOCV\ MSE} =\frac{1}{n}\left(e_{(-1)}^2+e_{(-2)}^2+\cdots+e_{(-n)}^2\right) =\frac{1}{n}\sum_{i=1}^{n}e_{(-i)}^2.

This is an average of observed held-out losses, not a proof that it exactly equals future prediction error. The fits share much of their training data, and each uses \(n-1\) observations rather than all \(n\). Thus the result is best understood as an estimate for the specified procedure and training-set size under the assumed prediction setting.

Computational Cost and Variability

LOOCV requires \(n\) model fits, each using \(n-1\) observations. In \(K\)-fold cross-validation, the data are split into \(K\) folds; each fit holds out one fold, so the usual procedure requires \(K\) fits. Thus, LOOCV can be substantially more computationally demanding when \(n\) is large. The precise time depends on the fitting procedure, not only on the number of fits.

The held-out cases also differ. In LOOCV, each test set contains one observation, so the estimate averages individual losses. In \(K\)-fold cross-validation, each test set contains a group of observations, and the final estimate averages across the held-out losses from all folds. The LOOCV training sets are extremely similar to one another: any two share \(n-2\) observations. Consequently, the individual error contributions are dependent. LOOCV can have higher variability than a K-fold alternative in some settings, although this is not a universal guarantee; variability depends on the data and the learning procedure.

LOOCV does have an important practical feature: each model is trained on almost all the available observations. That can matter when a smaller training set would poorly represent the final fitting situation. But more training data in each fit do not by themselves settle which validation method is preferable. Consider computational capacity, sample size, the stability of the fitting procedure, and the purpose of the evaluation.

Worked Example: Counting the Fits

A hypothetical data set contains \(n=120\) observations. Suppose one model fit takes \(0.4\) seconds, with roughly similar fitting time across training-set sizes. LOOCV requires \(120\) fits, for an estimated fitting time of \(120(0.4)=48\) seconds.

For comparison, ten-fold cross-validation requires 10 fits. If the folds are balanced, each test fold contains \(120/10=12\) observations and each training set contains \(120-12=108\) observations. The estimated fitting time is \(10(0.4)=4\) seconds. Under these simplifying assumptions, LOOCV uses 12 times as many fits and takes about 44 seconds longer. Actual time may differ if fit duration changes with training-set size or other computational steps are involved.

This comparison counts fits; it does not show that one error estimate must be better. LOOCV trains on 119 cases per fit, while ten-fold cross-validation trains on 108. Which training size better reflects the intended use, and how stable the resulting estimate is, are separate considerations.

Common Pitfalls

Using the held-out outcome in its own fit. That no longer evaluates a prediction made without that case. Leave the observation out of every data-dependent part of the fit for that round.
Calling LOOCV training error. The \(i\)th prediction comes from a fit that excludes case \(i\); it is a held-out prediction. Training error instead measures fit on observations used to estimate the model.
Confusing MSE and RMSE units. MSE has squared outcome units; RMSE has the original outcome units.
Assuming LOOCV is always preferable because it uses \(n-1\) training cases. It requires \(n\) fits and its estimate can be variable. More observations per fit do not guarantee lower variability or a more useful estimate.
Interpreting the estimate as a guarantee for a future case. LOOCV summarizes held-out errors across the available cases under a prediction setting; it does not promise that a particular new prediction will be close.

Key takeaway: LOOCV fits the specified regression procedure \(n\) times, each time holding out one observation. Its MSE or RMSE summarizes the resulting held-out errors. The method uses nearly all cases for each fit, but its computational cost and potential variability should be weighed against K-fold alternatives.

Check Your Understanding

Answer these questions using the definitions and calculations above.

For a data set of 75 observations, how many fits does LOOCV require, and how many observations are in each training set?
Why does the held-out prediction for an observation need to come from a model fitted without that observation?
If the outcome is measured in dollars, what are the units of LOOCV MSE and LOOCV RMSE?
In the intercept-only example, why is the prediction for the held-out outcome \(2\) equal to \(14/3\), rather than the mean of all four outcomes?
Give one reason LOOCV may be computationally expensive and one reason it may be more variable than a K-fold alternative.

What You'll Learn