From Folds to a Cross-Validation Score
K-fold cross-validation estimates how well a specified modeling procedure predicts observations not used to fit that particular model. The calculation is straightforward only if the roles of the folds remain clear: for each fit, one fold is held out and all the other folds form the training data. The held-out observations contribute prediction errors, not coefficient estimates.
For elastic net, the candidate being evaluated includes a fixed mixing parameter \(\alpha\) and penalty parameter \(\lambda\). This tutorial focuses on computing the score for such a fixed candidate. It does not choose a validation loss or compare tuning rules.
A common elastic net convention fits an intercept \(b_0\) without penalizing it and estimates coefficients \(\boldsymbol{\beta}\) by minimizing the following objective on a training set of size \(m\):
Here \(\lambda\geq 0\), and \(0\leq\alpha\leq1\). The same candidate values of \(\alpha\) and \(\lambda\) are used for each fold; the estimated intercept and coefficients are refit using that fold's training observations. The response prediction for held-out observation \(i\in I_k\) is \(\hat y_i^{(-k)}=\hat b_{0,k}+x_i^\mathsf{T}\hat{\boldsymbol{\beta}}_k\). The superscript indicates that the fit did not use fold \(k\).
For squared-error loss, each held-out observation contributes \((y_i-\hat y_i^{(-k)})^2\). Since every observation is held out once, the pooled K-fold cross-validation mean squared error is:
The Computation in Order
A reliable calculation follows the same sequence for every fold. Do not fit once on all the data and then call the resulting in-sample residuals validation errors. Each held-out prediction must come from a fit that excluded that observation's fold.
Specify \(K\), the fold membership of every observation, and the candidate \((\alpha,\lambda)\). Do not change the candidate between folds when computing its score.
For fold \(k\), fit the elastic net to \(T_k\), the observations outside \(I_k\). This produces fold-specific coefficient estimates.
Use that fold's fitted coefficients to calculate one prediction for each observation in \(I_k\). Do not use its response to obtain the prediction.
Calculate each held-out loss, add the losses across all folds, and divide by \(n\) for pooled mean squared error.
Worked Calculations
Worked Example: Pooling Errors From Three Equal Folds
A hypothetical insurer is evaluating a fixed elastic net candidate with \(\alpha=0.6\) and \(\lambda=0.15\) for predicting a claim-severity index. Six observations are divided into three folds of two observations each. After fitting on the four training observations for each fold, the held-out values and predictions are as shown. Compute the three fold mean squared errors and the pooled cross-validation mean squared error.
| Fold | Held-out responses | Held-out predictions |
|---|---|---|
| 1 | 4, 6 | 3, 7 |
| 2 | 5, 8 | 6, 6 |
| 3 | 3, 9 | 4, 7 |
Fold 1. The residuals are \(4-3=1\) and \(6-7=-1\). Their squared errors are \(1^2=1\) and \((-1)^2=1\). Thus the fold sum is \(1+1=2\), and the fold mean squared error is \(2/2=1\).
Fold 2. The residuals are \(5-6=-1\) and \(8-6=2\). The squared errors are \(1\) and \(4\). The fold sum is \(5\), so its mean squared error is \(5/2=2.5\).
Fold 3. The residuals are \(3-4=-1\) and \(9-7=2\). The squared errors are \(1\) and \(4\). The fold sum is \(5\), so its mean squared error is \(5/2=2.5\).
The total squared error is \(2+5+5=12\) over \(n=6\) held-out observations. Therefore the pooled cross-validation mean squared error is \(12/6=2\). Because the folds have equal sizes, this also equals the unweighted average of the fold means: \((1+2.5+2.5)/3=2\). The method is to score only held-out predictions; the mistake avoided is treating training residuals as validation errors. Check: the mean of all six squared errors \(1,1,1,4,1,4\) is \(12/6=2\), as required.
Worked Example: Refit a Simple Elastic Net in Every Fold
Consider six hypothetical records with a single predictor \(x\). Each of three folds contains one observation with \(x=-1\) and response \(y=8\), and one with \(x=1\) and response \(y=12\). Use squared-error loss and the elastic net with an unpenalized intercept, \(\alpha=0.5\), and \(\lambda=0.2\). Compute the K-fold cross-validation mean squared error.
For each fold, the four training observations consist of two copies of \((-1,8)\) and two copies of \((1,12)\). Thus the training predictor mean is \(0\), the training response mean is \(10\), and the unpenalized least-squares slope before the penalty is \(2\). The same training configuration occurs for all three folds.
Derive the penalized slope. With \(m=4\), balanced \(x\), and intercept \(b_0=10\), the derivative of the squared-error part of the objective with respect to \(\beta\) is \(\beta-2\): the mean of \(x^2\) is \(1\), and the unpenalized slope is \(2\). Since the minimizing slope is positive, the derivative of the penalty is \(\lambda\alpha+\lambda(1-\alpha)\beta\). Setting the total derivative equal to zero gives:
The intercept is the training response mean, \(\hat b_0=10\). For a held-out observation with \(x=-1\), the prediction is \(10-19/11=91/11\); for \(x=1\), it is \(10+19/11=129/11\). The corresponding responses are \(8\) and \(12\), so the residuals are \(8-91/11=-3/11\) and \(12-129/11=3/11\). Each squared error is \(9/121\).
Every fold has this same pair of held-out errors. Across all six observations, the pooled sum of squared errors is \(6(9/121)=54/121\), and division by \(n=6\) gives \(\operatorname{CV}_3=9/121\approx0.0744\), rounded to four decimal places. Check: each fold mean squared error is also \(\{2(9/121)\}/2=9/121\). The key is that the slope was re-estimated from the four training records for each fold, even though symmetry makes the three fitted results identical here.
Worked Example: Unequal Fold Sizes Require Weighted Aggregation
A seven-record hypothetical portfolio data set is split into three folds with sizes \(2,2,\) and \(3\). For a fixed elastic net candidate, the held-out residuals are \(1,3\) in fold 1; \(0,2\) in fold 2; and \(-1,1,2\) in fold 3. Compute the pooled mean squared error and compare it with the unweighted average of fold mean squared errors.
In fold 1, the squared errors are \(1\) and \(9\), with sum \(10\) and fold mean \(10/2=5\). In fold 2, the squared errors are \(0\) and \(4\), with sum \(4\) and fold mean \(4/2=2\). In fold 3, the squared errors are \(1\), \(1\), and \(4\), with sum \(6\) and fold mean \(6/3=2\).
The pooled squared-error sum is \(10+4+6=20\). There are \(2+2+3=7\) held-out observations, so the pooled cross-validation mean squared error is \(20/7\approx2.8571\), rounded to four decimal places. Equivalently, weight each fold mean by its size:
The unweighted average of the three fold means would instead be \((5+2+2)/3=3\), which is not the pooled observation-level mean. That average gives the two-observation fold and the three-observation fold equal weight. The method is to add the individual losses and divide by the total number of held-out observations. Check: the weighted numerator is \(10+4+6=20\), matching the direct sum of squared errors.
Why the Aggregation Works, and What Can Go Wrong
The pooled formula follows directly from the partition. Because the folds are disjoint and cover every observation, each index \(i\) appears in exactly one set \(I_k\). Consequently, summing the held-out losses over folds counts every observation's out-of-fold loss once and only once. Dividing by \(n\) therefore gives the arithmetic mean loss per observation.
If fold \(k\) has size \(n_k\) and fold mean loss \(L_k\), its total loss is \(n_kL_k\). Thus pooled mean loss is \(\sum_k n_kL_k/n\). It reduces to the unweighted mean of the \(L_k\) values only when fold sizes are equal (or when the particular fold means happen to make the two calculations coincide).
A cross-validation calculation is only interpretable when the partition and fitting steps match the intended evaluation. Every fold must be held out in turn, the same candidate \((\alpha,\lambda)\) must be used when scoring that candidate, and each held-out prediction must come from the corresponding training fit. Keep the loss definition and its units explicit: squared errors have squared response units, so the mean squared error does as well. The square root would be a different summary, not the mean squared error computed here.
These mechanics matter because an incorrectly aggregated score can change the apparent ranking of candidates, while training predictions can make performance look better than genuinely held-out predictions. For a fixed candidate, a clear record of fold membership, training fit, held-out prediction, loss, and final denominator makes the calculation auditable and reduces indexing errors.
Check Your Understanding
For each problem, first identify the held-out errors and then apply the stated aggregation rule.
- A six-observation data set has three equal folds. Their held-out squared-error sums are \(6,4,\) and \(8\). What is pooled cross-validation mean squared error? (Answer: \(18/6=3\))
- Two folds contain \(2\) and \(4\) observations, with fold mean squared errors \(1.5\) and \(3\). Find the pooled mean squared error. (Answer: \(\{2(1.5)+4(3)\}/6=2.5\))
- For one held-out fold, the observed responses are \(7,10,4\), and predictions are \(8,8,5\). Find the fold's sum and mean of squared errors. (Answer: squared errors \(1,4,1\); sum \(6\), mean \(2\))
- A five-observation data set has fold sizes \(1,2,2\), with fold means of squared error \(4,1,\) and \(2.5\), respectively. Compute the pooled mean and explain why the unweighted average is not the required calculation. (Answer: \(\{1(4)+2(1)+2(2.5)\}/5=2.2\); the fold sizes differ.)
- For a fixed elastic net candidate, each of four folds has the same held-out squared-error sum of \(3\), and each fold has two observations. What is the pooled mean squared error? (Answer: total \(12\) divided by \(8\), or \(1.5\))