Pooling Predictions Versus Averaging Fold Metrics

Why the Aggregation Rule Matters

In cross-validation, each held-out observation receives a prediction from a model that did not use that observation for fitting. After the folds have been evaluated, there are two common summaries: pool the out-of-fold predictions and calculate one metric, or calculate a metric within each fold and average those fold metrics. These summaries do not always answer the same question.

For accuracy and per-observation loss, the difference is usually a weighting choice: pooling gives each observation equal weight, while a simple average gives each fold equal weight. AUC is more delicate. A pooled AUC compares scores from observations in different folds, even though those scores came from different fitted models. A fold-average AUC does not make those cross-fold comparisons.

The distinction matters in genomic prediction, where folds may differ in size, class balance, cohort composition, or the scale of model scores. A reported performance number should make clear both the metric and the aggregation rule used to obtain it.

Setup and Definitions

Suppose a binary outcome \(y_i\) is recorded for each of \(n\) observations. Each observation belongs to one held-out fold \(k(i)\), and \(\hat p_i^{(-k)}\) denotes its predicted probability of the positive outcome from the model fitted without fold \(k\). Write \(s_i\) for a prediction score used to rank observations. If a predicted class is needed for accuracy, use a fixed threshold \(\tau\): the predicted class is positive when \(\hat p_i^{(-k)}\geq\tau\), and negative otherwise. The threshold is treated as already specified, not selected from these held-out outcomes.

This setup presumes that each observation has one out-of-fold prediction, that the outcome and prediction are paired to the same observation, and that all data-dependent preprocessing and model fitting were confined to the corresponding training data. The fold design must also respect the intended evaluation unit: for example, related individuals or samples that must be kept together should not be split across training and held-out folds. These conditions make the predictions suitable for evaluation; they do not guarantee that every fold metric is stable or that folds represent the eventual target population.

For accuracy, let \(a_k\) be the number of correct classifications among the \(n_k\) observations in fold \(k\). The fold accuracy is \(A_k=a_k/n_k\). Pooled accuracy is the total number correct divided by the total number of observations. These two definitions already show why fold sizes matter.

Thus, pooled accuracy is the fold-size-weighted mean of fold accuracies. It equals the unweighted fold mean when fold sizes are equal, or in special cases where the fold accuracies happen to make the two averages equal. The same weighted-average identity applies to any loss that is calculated per observation and then averaged.

Worked Examples

For binary outcomes, empirical AUC can be defined as the proportion of positive-negative pairs in which the positive observation has the higher score, counting a tied pair as one-half. If fold \(k\) contains \(n_{k+}\) positives and \(n_{k-}\) negatives, it contributes \(n_{k+}n_{k-}\) such pairs to its fold AUC.

A pooled AUC uses the same pairwise rule but compares every positive observation with every negative observation, including pairs drawn from different folds. Consequently, it is not generally an average of the fold AUCs. Even a pair-count-weighted average of fold AUCs includes only within-fold pairs, whereas pooled AUC also includes cross-fold pairs.

For a per-observation loss, let \(\ell_i=\ell(y_i,\hat p_i^{(-k)})\). The fold mean loss is \(L_k=(1/n_k)\sum_{i:k(i)=k}\ell_i\); the pooled mean loss is the average over all observations. Log loss is one example, defined for binary outcomes by \(-y_i\log(\hat p_i^{(-k)})-(1-y_i)\log(1-\hat p_i^{(-k)})\). Smaller values indicate predictions that assign greater probability to the observed outcomes, with large penalties for confident errors.

Why the Summaries Differ

The accuracy and loss identities follow from partitioning a sum by fold. For any per-observation quantity \(q_i\), define the fold mean \(\bar q_k=(1/n_k)\sum_{i:k(i)=k}q_i\). Then \(\sum_i q_i=\sum_k\sum_{i:k(i)=k}q_i=\sum_k n_k\bar q_k\). Dividing by \(n=\sum_k n_k\) gives the pooled mean as \(\sum_k(n_k/n)\bar q_k\). An unweighted fold mean instead assigns coefficient \(1/K\) to every fold. This proves the weighting distinction without relying on a particular loss formula.

AUC is not a per-observation mean: it is a mean over positive-negative pairs. The pooled set of pairs contains both within-fold and cross-fold comparisons. Also, a simple mean of fold AUCs weights folds equally even when their numbers of positive-negative pairs differ. A pair-count-weighted average of fold AUCs weights the within-fold pairs equally, but still does not reproduce pooled AUC unless there are no cross-fold pairs or their contribution happens to match the within-fold result.

Choosing and Reporting a Summary

For accuracy or loss, use pooled evaluation when the intended summary is average performance per observation in the combined held-out sample. Use an unweighted fold mean when the intended summary gives each fold equal importance, for example when folds deliberately represent distinct sites and the target question concerns a typical site rather than a typical observation. Neither rule is universally preferable; the estimand and fold design determine the choice.

For AUC, a mean of fold AUCs has a direct interpretation as average within-fold ranking performance, subject to the chosen fold weighting. Pooled AUC instead describes ranking across the combined out-of-fold scores, including comparisons between models' score outputs. Such comparisons are most defensible when the score scales are meaningfully comparable across folds. If each fitted model can produce scores on a different scale, report fold AUCs or their clearly specified average rather than implying that pooled cross-fold comparisons are automatically meaningful.

A transparent report names the aggregation rule, gives fold sizes and relevant class counts, and states whether AUC is pooled or averaged across folds. If both summaries are useful, report both with their interpretations rather than selecting whichever looks better. In repeated cross-validation, the same observation may receive several held-out predictions; the analyst must additionally state how repeats were combined before calculating a metric. A single-prediction pooling identity cannot be applied without specifying that step.

Diagnostics and Common Pitfalls

• Assuming a simple fold average is pooled performance. This is false for unequal fold sizes. Check the fold sizes and compare the unweighted average with the size-weighted calculation.
• Interpreting pooled AUC as only within-fold ranking. It includes cross-fold pairs. Inspect fold-specific score distributions and determine whether scores from separate fitted models are comparable.
• Ignoring class counts. A fold without both outcome classes has no defined empirical AUC. Very small positive or negative counts also make a fold AUC based on few pairs and potentially unstable.
• Comparing summaries built from different predictions. Confirm that both calculations use the same held-out observations and the same out-of-fold predictions. Do not refit or preprocess using held-out outcomes to make one summary appear more favorable.
• Changing the target implicitly. Equal weighting of sites, folds, and individuals are distinct choices. State which units receive equal weight and why that matches the intended application.

These metrics remain estimates based on a particular validation design. They can be sensitive to the fold assignment, and the folds may not represent future cohorts or sites. Aggregation cannot repair leakage, an inappropriate split, or a mismatch between the validation population and intended use. For uncertainty, retain the validation design and its dependencies in view rather than treating individual fold results as automatically independent observations.