Why Use Cross-Validation to Select a Regression Model?
A regression model can fit its training data well and still predict new observations poorly. This is especially likely when the candidate models differ in complexity or when a model has been chosen after considering many alternatives. Cross-validation compares candidates on observations that were not used to fit the corresponding fold-specific model. It therefore provides a practical estimate of how well each candidate’s fitting procedure predicts observations similar to those in the study.
The comparison is meaningful only when the candidates are evaluated on the same folds, with the same response scale and loss measure. The selected candidate is the one with the smallest estimated validation error for the stated objective. Cross-validation is a selection tool, not a guarantee of future performance and not a substitute for a final test-set evaluation when one is available.
Fold Errors and Their Aggregation
For a quantitative response, a common criterion is squared prediction error. Let \(y_j\) be the observed response for validation observation \(j\), and let \(\hat y_j^{(-k)}\) be the prediction for that observation from a model fitted without fold \(k\). If fold \(k\) contains \(n_k\) observations, its mean squared error (MSE) is
where \(\mathcal{F}_k\) is the set of observations in fold \(k\). A pooled cross-validation MSE weights each fold by its number of validation observations:
Here \(n=\sum_k n_k\) is the total number of modeling observations. If the folds are equal in size, the pooled MSE equals the simple average of the fold MSEs. If fold sizes differ, an unweighted average gives a small fold the same influence as a much larger fold, even though it contains fewer validation predictions.
The root mean squared error is \(\operatorname{CV\ RMSE}=\sqrt{\operatorname{CV\ MSE}}\). Because the square root is increasing, ranking candidates by pooled MSE gives the same ranking as by pooled RMSE. MSE is in squared response units; RMSE is in the original response units and may be easier to interpret. The chosen measure should match the prediction objective. For example, squared error penalizes large errors more heavily than absolute error does.
A Practical Cross-Validation Comparison
A fair comparison uses the same folds for every candidate. Then each candidate is judged on the same held-out observations, reducing the chance that a model appears better merely because it received easier validation cases. The fitting process must also be repeated within each fold: fit the model on the training portion, predict the held-out fold, and record the validation errors.
Specify the response, candidate models, prediction loss, and the data eligible for cross-validation. Keep a final test set separate if one is intended for final evaluation.
Partition the modeling data into \(K\) folds. Use the same fold assignments for each candidate so the comparison is paired by validation observations.
For each fold and each candidate, fit using only the other folds. Apply any data-dependent preprocessing or predictor selection using only that training portion, then predict the held-out fold.
Calculate the fold losses, pool them with the correct observation weights, and choose the candidate with the smallest cross-validation error for the specified loss.
After selection, fit the chosen modeling procedure on all available training data. This refitted model is distinct from any fold-specific model used to calculate cross-validation error.
The reasoning behind pooled MSE follows directly from the definition of an average over observations. Fold \(k\) has total squared error \(n_k\operatorname{MSE}_k\). Adding these totals across folds gives the sum of all validation squared errors, because each observation belongs to exactly one fold. Dividing by the total number of observations gives the average squared error per validation prediction. This also explains why the fold MSEs must be weighted by \(n_k\) when fold sizes differ.
Worked Example: Unequal Fold Sizes Change the Summary
A hypothetical insurer is comparing two least-squares regression candidates for predicting a numeric claim-cost measure, in thousands of dollars. The response is evaluated by squared error. There are no interest or mortality calculations in this prediction problem; no parametric distribution for the response or errors is assumed. Three shared validation folds contain 4, 4, and 2 observations. Candidate A has fold MSEs \(2,2,9\), and candidate B has fold MSEs \(3,3,3\). Select a candidate using pooled cross-validation MSE, and compare that result with the unweighted mean of fold MSEs.
Candidate A. Its fold squared-error totals are \(4(2)=8\), \(4(2)=8\), and \(2(9)=18\). Thus its pooled MSE is
The unweighted mean of its fold MSEs is \((2+2+9)/3=13/3\), approximately \(4.333\).
Candidate B. Its fold squared-error totals are \(4(3)=12\), \(4(3)=12\), and \(2(3)=6\). Its pooled MSE is
The unweighted mean is also \((3+3+3)/3=3.0\). Since \(3.0<3.4\), pooled cross-validation MSE selects candidate B. This example does not reverse the ranking, but it shows why the two summaries differ for candidate A: the small fold with MSE 9 receives only two observations’ weight in the pooled calculation.
Method and trap. Weight each fold MSE by its validation size before dividing by the total size. Treating the unweighted average as the pooled observation-level MSE would incorrectly report candidate A’s error as \(4.333\), rather than \(3.4\). The rankings happen to agree here; that is not guaranteed.
Worked Example: Selecting Among Three Candidates
A hypothetical energy cooperative predicts monthly site-level electricity use, measured in hundreds of kilowatt-hours. Three regression candidates are assessed on the same three folds, each containing two sites. The fold squared-error totals are as follows: candidate L has \(8, 2, 8\); candidate Q has \(4, 6, 2\); and candidate R has \(2, 12, 4\). Use pooled MSE to select a candidate. No interest or mortality assumptions apply, and the fold totals do not rely on a specified parametric distribution.
Each candidate has six validation observations in total. For candidate L, the total squared error is \(8+2+8=18\), so its pooled MSE is \(18/6=3.0\). For candidate Q, the total is \(4+6+2=12\), so its pooled MSE is \(12/6=2.0\). For candidate R, the total is \(2+12+4=18\), so its pooled MSE is \(18/6=3.0\).
Candidate Q has the smallest pooled MSE, \(2.0\), and is selected under the stated squared-error objective. Its pooled RMSE is \(\sqrt{2.0}\), approximately \(1.414\) hundreds of kilowatt-hours. Candidates L and R each have RMSE \(\sqrt{3.0}\), approximately \(1.732\) hundreds of kilowatt-hours. These RMSE values are rounded to three decimals.
Check. All three candidates have the same total validation size, so averaging their three fold MSEs gives the same ranking as pooling their squared errors. Candidate Q’s error is lower than either alternative. The method avoids selecting a candidate by inspecting only its best fold or by comparing fold errors calculated on different observations.
What Cross-Validation Estimates—and What It Does Not
Cross-validation estimates predictive performance for a modeling procedure trained on a fraction of the available data. In \(K\)-fold cross-validation, each fold-specific fit uses approximately \((K-1)/K\) of the modeling observations. The estimate is therefore not exactly the error of a model trained on the full data set, and it can vary with the particular partition. Repeating the procedure with different partitions can reveal sensitivity to fold assignment, but it does not remove all uncertainty.
For the usual interpretation, observations should be sampled in a way that makes the validation cases reasonably representative of the future cases to which predictions will be applied. If rows are dependent, clustered, or ordered in time, randomly distributing related or future observations into training folds can yield an overly optimistic estimate. The fold construction should respect the data structure and intended prediction task. Cross-validation does not fix a mismatch between the validation design and the way the model will actually be used.
A further condition is that information from a validation fold must not affect its prediction. Suppose a candidate uses a selected subset of predictors, or predictors are standardized using sample means and standard deviations. If those choices or quantities are calculated once from all observations before cross-validation, the held-out responses or predictors can influence the fold model. This leakage makes validation errors less independent of the held-out data and can make performance look better than it is. Repeat such data-dependent operations using the training portion of each fold only.
Worked Example: Keeping the Final Test Set Untouched
A hypothetical pension administrator has 120 records available for predicting an annual service-cost response in dollars. The records are divided before model selection into 90 training records and 30 final test records. On the 90 training records, five-fold cross-validation gives candidate A a pooled MSE of 25,000 and candidate B a pooled MSE of 21,000, both in squared dollars. The administrator then refits candidate B on all 90 training records. On the 30 test records, its squared errors sum to 720,000. Explain the selection and calculate the final test RMSE. No interest or mortality assumption is involved; no response distribution is specified.
Selection using training data. The cross-validation comparison is \(21{,}000<25{,}000\), so candidate B is selected. The 30 test records played no role in that choice.
Final test evaluation. The test MSE is the total test squared error divided by the number of test records:
Therefore the test RMSE is \(\sqrt{24{,}000}\), approximately \(\$154.92\), rounded to the nearest cent. This estimates prediction error for the final refitted candidate on this test sample; it is not another criterion to use repeatedly to choose among candidates.
Method and trap. Select with cross-validation on the training data, refit the selected candidate on all training records, and evaluate it once on the test set. If the administrator instead used the test error to choose between A and B, the test set would become part of model selection and would no longer provide an untouched final assessment.
Common Exam Traps
- Comparing different folds. If candidates are evaluated on different partitions, some apparent advantage may come from easier validation observations. Use common folds for a direct comparison.
- Confusing fold MSE and pooled MSE. Average fold MSEs directly only when fold sizes are equal, or when the intended summary explicitly gives folds equal weight. For observation-level pooled error, weight each fold by its size.
- Leaking information. Any preprocessing, predictor screening, or tuning that uses the data must be performed within each training fold. A validation observation must not influence the fitted prediction it is used to assess.
- Choosing by the test set. Repeatedly checking test performance while changing the model turns the test set into validation data. Reserve it for the final assessment after selection.
- Interpreting a small difference too strongly. A lower cross-validation error is a comparison result, not proof that one model is universally better. Fold assignment, sampling variability, the prediction target, and the chosen loss all matter.
Check Your Understanding
For each problem, identify the appropriate cross-validation calculation or interpretation before checking the answer.
- A four-fold comparison has equal-sized folds. Candidate P has fold MSEs \(5,3,4,4\), and candidate Q has \(4,4,4,5\). Which candidate has lower pooled MSE, and what is that MSE for each? (Answer: P, with MSE \(4.0\); Q has MSE \(4.25\).)
- Two folds contain 6 and 2 observations. A model’s fold MSEs are 2 and 8. Calculate the pooled MSE and explain why the unweighted mean is not the pooled value. (Answer: \((6(2)+2(8))/8=3.5\); the unweighted mean is 5, which gives the smaller fold excessive weight.)
- Three equal-sized folds give candidate C squared-error totals of \(6, 12, 18\), and candidate D totals of \(10, 10, 10\). Each fold has five observations. Which candidate has lower pooled MSE? (Answer: C has \(36/15=2.4\); D has \(30/15=2.0\), so D is lower.)
- A modeling team standardizes each predictor once using all observations, then runs cross-validation. What is the issue, and what is the correction? (Answer: The held-out folds influence preprocessing; calculate standardization using each fold’s training portion and apply those training estimates to that fold’s validation observations.)
- A model has the smallest cross-validation MSE among four candidates. The team then repeatedly revises the model until the final test MSE is smallest. Can that test MSE still be reported as an untouched final evaluation? (Answer: No; repeated model changes based on test performance use the test set for selection. A new untouched test set or another appropriate evaluation design is needed for a final assessment.)