Pooling Losses Across Validation Folds

Why Pooling Validation Results Requires Care

In cross-validation, each fold supplies predictions for observations withheld from the model fitted for that fold. After validation, those predictions can be summarized in two ways: calculate a metric separately within each fold and average the fold metrics, or combine the held-out predictions and outcomes and calculate one metric from the combined set. These approaches need not give the same answer.

The distinction matters because folds may contain different numbers of observations, and many performance metrics are nonlinear functions of their underlying quantities. A reported score should therefore make clear whether it is an average across folds or a metric calculated from pooled held-out predictions. Neither summary is automatically correct for every purpose; the choice depends on the target summary.

Definition: Let validation fold \(k\) contain \(n_k\) observations, with held-out outcomes \(y_{ki}\) and predictions \(\hat{y}_{ki}\), for \(i=1,\ldots,n_k\). A fold-level metric is calculated using only the observations in that fold. A pooled metric is calculated after combining the held-out outcomes and predictions from all folds. In ordinary \(K\)-fold validation, each observation contributes one held-out prediction, so the total number of held-out predictions is \(N=\sum_{k=1}^{K}n_k\).

For regression, write the held-out prediction error as \(e_{ki}=y_{ki}-\hat{y}_{ki}\). The fold mean squared error (MSE) is the average squared error in that fold. The pooled MSE is the average squared error over all held-out observations. When fold sizes differ, the pooled MSE gives more weight to a fold containing more observations.

\operatorname{MSE}_k=\frac{1}{n_k}\sum_{i=1}^{n_k}e_{ki}^2, \qquad \operatorname{MSE}_{\mathrm{pool}} =\frac{\sum_{k=1}^{K}\sum_{i=1}^{n_k}e_{ki}^2}{\sum_{k=1}^{K}n_k} =\sum_{k=1}^{K}\frac{n_k}{N}\operatorname{MSE}_k.

Why the Pooled MSE Is a Weighted Average

The pooled MSE follows directly from its definition. Each fold’s MSE is its sum of squared errors divided by its fold size. Rearranging gives the fold’s total squared error. Adding those fold totals and dividing by the total number of observations yields the pooled average.

Recover each fold’s error total.
Multiplying \(\operatorname{MSE}_k\) by \(n_k\) gives \(\sum_{i=1}^{n_k}e_{ki}^2\).

Add the error totals.
Across all folds, the total squared error is \(\sum_{k=1}^{K}n_k\operatorname{MSE}_k\).

Divide by the total number of observations.
Since \(N=\sum_{k=1}^{K}n_k\), the pooled MSE is \(\sum_{k=1}^{K}(n_k/N)\operatorname{MSE}_k\). The weights sum to one.

Thus, when fold sizes are equal, the pooled MSE equals the ordinary arithmetic mean of the fold MSEs. When fold sizes are unequal, it generally does not. The same reasoning applies to mean absolute error (MAE), because pooled MAE is the total absolute error divided by the total number of observations.

Worked Example: Unequal Fold Sizes and Mean Squared Error

A hypothetical pricing model is evaluated on two held-out folds. Fold 1 has two squared prediction errors, \(0\) and \(4\). Fold 2 has four squared prediction errors, \(1\), \(1\), \(16\), and \(16\). Calculate the unweighted average of the fold MSEs and the pooled MSE. State which is the pooled observation-level error.

Method. Calculate each fold’s squared-error total and divide by its size to obtain its MSE. Then compare the unweighted fold average with the pooled total divided by the total number of held-out observations.

Fold 1. Its squared-error total is \(0+4=4\), and its size is \(n_1=2\). Therefore, \(\operatorname{MSE}_1=4/2=2\).

Fold 2. Its squared-error total is \(1+1+16+16=34\), and its size is \(n_2=4\). Therefore, \(\operatorname{MSE}_2=34/4=8.5\).

Unweighted fold average. Giving the two folds equal weight gives \((2+8.5)/2=5.25\).

Pooled MSE. There are \(2+4=6\) observations, and the total squared error is \(4+34=38\). Thus, \(\operatorname{MSE}_{\mathrm{pool}}=38/6=6.3333\) to four decimal places. The weighted calculation checks this result: \((2/6)(2)+(4/6)(8.5)=6.3333\).

The pooled MSE is \(6.3333\), because it averages squared errors over all six held-out observations. The unweighted fold average, \(5.25\), instead gives the two-observation fold and the four-observation fold equal influence. The common mistake is to call the unweighted fold average the pooled MSE; the correction is to weight each fold MSE by its share of the observations.

Nonlinear Metrics: RMSE and Classification Scores

A metric can be nonlinear even when the underlying error totals combine simply. Root mean squared error (RMSE) is the square root of MSE. Consequently, pooled RMSE is the square root of the pooled MSE; it is not generally the average of the fold RMSE values. Taking a square root after pooling and averaging square roots across folds are different operations.

\operatorname{RMSE}_{\mathrm{pool}} =\sqrt{\sum_{k=1}^{K}\frac{n_k}{N}\operatorname{MSE}_k} \quad\text{whereas}\quad \overline{\operatorname{RMSE}}_{\mathrm{fold}} =\frac{1}{K}\sum_{k=1}^{K}\sqrt{\operatorname{MSE}_k}.

Classification metrics have a similar issue. For a specified positive class, each fold has counts of true positives (TP), false positives (FP), false negatives (FN), and true negatives (TN). Accuracy, precision, recall, and the F1 score are ratios formed from these counts. To calculate a pooled version, first add the corresponding counts across folds, then calculate the ratio. In general, an average of fold-level ratios is not the ratio from pooled counts.

Definition: For a specified positive class, accuracy is \((\mathrm{TP}+\mathrm{TN})/n\), precision is \(\mathrm{TP}/(\mathrm{TP}+\mathrm{FP})\) when its denominator is positive, and recall is \(\mathrm{TP}/(\mathrm{TP}+\mathrm{FN})\) when its denominator is positive. The F1 score is \(2\mathrm{TP}/(2\mathrm{TP}+\mathrm{FP}+\mathrm{FN})\) when its denominator is positive. A pooled version of one of these metrics uses the sums of the relevant counts from all folds.

Worked Example: Pooled RMSE Is Not the Average Fold RMSE

Two validation folds for a hypothetical reserve-estimation model have these results: fold 1 contains three observations with total squared error \(12\), and fold 2 contains one observation with squared error \(9\). Calculate each fold’s RMSE, the arithmetic mean of the fold RMSEs, and the pooled RMSE.

Method. Divide each fold’s squared-error total by its size, take the square root to obtain its RMSE, and calculate pooled RMSE from the combined error total.

Fold 1. Its size is \(3\), so \(\operatorname{MSE}_1=12/3=4\) and \(\operatorname{RMSE}_1=\sqrt{4}=2\).

Fold 2. Its size is \(1\), so \(\operatorname{MSE}_2=9/1=9\) and \(\operatorname{RMSE}_2=\sqrt{9}=3\).

Arithmetic mean of fold RMSEs. This is \((2+3)/2=2.5\).

Pooled RMSE. The total squared error is \(12+9=21\), and the total number of observations is \(3+1=4\). Hence, the pooled MSE is \(21/4=5.25\), and the pooled RMSE is \(\sqrt{5.25}=2.2913\) to four decimal places. As a check, the pooled MSE also equals \((3/4)(4)+(1/4)(9)=5.25\).

The pooled RMSE is \(2.2913\), not \(2.5\). The latter is a valid arithmetic average of fold RMSEs but answers a different question: it weights each fold equally before taking account of how many observations each contains. The common mistake is to apply the square root fold by fold and then assume the resulting numbers can be averaged to obtain pooled RMSE.

Worked Example: Pooling Confusion Counts Before Calculating F1

A hypothetical claims-screening classifier is evaluated on two held-out folds. In fold 1, the counts are TP \(=1\), FP \(=1\), FN \(=2\), and TN \(=0\). In fold 2, they are TP \(=4\), FP \(=0\), FN \(=0\), and TN \(=4\). Calculate the mean fold accuracy and pooled accuracy, then compare the mean fold F1 score with the pooled F1 score.

Method. Calculate the requested metrics within each fold, then add the confusion counts and recalculate the metrics from the combined counts.

Fold sizes and accuracies. Fold 1 has \(1+1+2+0=4\) observations. Its accuracy is \((1+0)/4=0.25\). Fold 2 has \(4+0+0+4=8\) observations. Its accuracy is \((4+4)/8=1\). The unweighted mean fold accuracy is \((0.25+1)/2=0.625\).

Pooled accuracy. The combined counts are TP \(=1+4=5\), FP \(=1+0=1\), FN \(=2+0=2\), and TN \(=0+4=4\), for \(12\) observations. The pooled accuracy is \((5+4)/12=9/12=0.75\). This also equals the size-weighted accuracy, \((4/12)(0.25)+(8/12)(1)=0.75\).

Fold F1 scores. In fold 1, the F1 score is \(2(1)/(2(1)+1+2)=2/5=0.4\). In fold 2, it is \(2(4)/(2(4)+0+0)=8/8=1\). Their unweighted mean is \((0.4+1)/2=0.7\).

Pooled F1 score. From the combined counts, it is \(2(5)/(2(5)+1+2)=10/13=0.7692\) to four decimal places. The pooled F1 score is therefore \(0.7692\), rather than the mean fold F1 of \(0.7\). The common mistake is to assume that every ratio should be averaged across folds; the correction is to pool the counts first when the desired quantity is the F1 score over all held-out predictions.

Choosing and Reporting the Summary

The pooled metric describes performance over the combined set of held-out predictions, with observations contributing according to their number. The unweighted fold average describes the average fold-level result, with each fold contributing equally, regardless of its size. These are distinct summaries, and a question may specifically ask for either one.

For an additive per-observation loss, such as squared error or absolute error, pooling the loss totals and dividing by the total observation count gives the pooled metric. Equivalently, it is a fold-size-weighted mean of the fold metrics. For a nonlinear metric, the required aggregation depends on its definition: for RMSE, pool squared errors before taking the square root; for accuracy, pool correct and total counts; for precision, recall, or F1, pool the relevant confusion counts before forming the ratio.

When fold sizes are equal, the unweighted fold average of an additive per-observation metric agrees with its pooled value. Equal sizes do not generally make an average of nonlinear fold metrics equal to the corresponding pooled metric. For example, the average of fold RMSE values need not equal pooled RMSE even with equal fold sizes, because the square root is applied at a different stage.

In reporting a validation result, specify the metric and the aggregation rule. If repeated validation produces more than one held-out prediction for an observation, do not silently treat those repeated predictions as distinct observations: state whether the calculation pools prediction occasions or first summarizes predictions for each observation. The intended target determines the appropriate summary.

Key takeaway: Identify what is being averaged before calculating a validation score. Pool per-observation losses by adding their totals and dividing by the total number of observations. For nonlinear metrics, combine the underlying quantities required by the metric first, then calculate the metric.

Check Your Understanding

For each problem, distinguish the requested pooled metric from an unweighted average across folds when both are relevant.

A regression validation has two folds: \(n_1=2\) with \(\operatorname{MSE}_1=1\), and \(n_2=3\) with \(\operatorname{MSE}_2=5\). Find the pooled MSE and the unweighted mean fold MSE. (Answer: pooled MSE \(=3.4000\); unweighted mean \(=3.0000\))
A model has two MAE folds: fold 1 has \(n_1=4\) and \(\operatorname{MAE}_1=1.5\); fold 2 has \(n_2=2\) and \(\operatorname{MAE}_2=4\). Find pooled MAE. (Answer: \(14/6=2.3333\))
Two folds have sizes \(2\) and \(6\), and their MSE values are \(4\) and \(9\), respectively. Find pooled RMSE and the unweighted mean fold RMSE. Round to four decimal places. (Answer: pooled RMSE \(=\sqrt{7.75}=2.7839\); mean fold RMSE \(=(2+3)/2=2.5000\))
In fold 1, TP \(=2\), FP \(=2\), FN \(=0\), TN \(=0\). In fold 2, TP \(=1\), FP \(=0\), FN \(=3\), TN \(=4\). Find the unweighted mean fold F1 score and the pooled F1 score. (Answer: mean fold F1 \(=(0.6667+0.4000)/2=0.5333\); pooled F1 \(=6/11=0.5455\))
A fold has five squared errors with total \(10\), and a second fold has three squared errors with total \(24\). Find the two fold MSEs, the unweighted mean fold MSE, and the pooled MSE. (Answer: fold MSEs \(2\) and \(8\); unweighted mean \(=5\); pooled MSE \(=34/8=4.2500\))

What You'll Learn