From Resampling Design to a Defensible Performance Estimate
A performance number is meaningful only if the validation design represents how predictions will be used. Randomly splitting related records can place near-duplicates in training and validation; shuffling future observations into a time-series training set can reveal information unavailable at prediction time. A carefully calculated error metric does not repair either design error.
An integrated analysis therefore asks three questions in order: what is the prediction unit, what information will be available at prediction time, and what loss or metric matches the decision? The resampling scheme must respect the answers. The examples below use hypothetical insurance and risk-modeling data; no interest-rate or mortality model is involved. They treat the observed records as empirical data and do not assume a parametric distribution for prediction errors.
For a loss function \(L(y_i,\hat y_i)\), where \(y_i\) is an observed outcome and \(\hat y_i\) is its prediction, the pooled validation loss over \(N\) validation predictions is
For squared-error loss, \(L(y_i,\hat y_i)=(y_i-\hat y_i)^2\); the pooled mean squared error is the average squared error, and the pooled root mean squared error is its square root. For unequal fold sizes, pooling each validation observation once generally differs from giving every fold equal weight. Which summary is appropriate depends on the target: average loss per prediction weights records equally, whereas an average of fold-level losses weights folds equally.
Worked Example: Grouped Validation and Unequal Fold Sizes
A hypothetical commercial insurer wants to predict claim severity for new employer groups. Each employer contributes one record to the analysis, and records from a given employer must not be split between training and validation. A grouped two-fold validation produces residuals of \(2\) and \(2\) dollars-thousands in the first validation fold, and \(1,1,2,2\) dollars-thousands in the second. Calculate pooled mean squared error and compare it with the unweighted average of fold mean squared errors.
Method. Hold out whole employer groups in each fold, then pool the squared errors because the target is the average squared error per held-out employer. The six employers are the evaluation units; the groups are kept separate by construction.
First fold. Its sum of squared errors is \(2^2+2^2=4+4=8\), so its mean squared error is \(8/2=4\) in squared-thousands of dollars.
Second fold. Its sum of squared errors is \(1^2+1^2+2^2+2^2=1+1+4+4=10\), so its mean squared error is \(10/4=2.5\).
Pooled result. The total squared error is \(8+10=18\) over \(2+4=6\) employers. Thus the pooled mean squared error is \(18/6=3\). The pooled root mean squared error is \(\sqrt{3}\approx1.732\) thousand dollars.
The unweighted average of fold mean squared errors is \((4+2.5)/2=3.25\), not \(3\). It assigns half the weight to a fold of two employers and half to a fold of four. The pooled calculation gives each employer weight \(1/6\). The common mistake is to report \(3.25\) as the per-employer mean squared error; the correction is to weight each fold by its validation size, giving \((2(4)+4(2.5))/6=3\). The grouped split also avoids the separate mistake of allowing records from one employer to appear on both sides of a split.
Check. The pooled value \(3\) lies between the fold values \(2.5\) and \(4\), and is closer to \(2.5\), the result from the larger fold. That is consistent with its larger weight.
Choosing the Design Before Calculating the Score
The resampling design should mimic the information boundary in deployment. With clustered records, such as multiple policies per employer, hold out whole groups if predictions concern new groups. With sequential data, train only on information available before the validation period and evaluate later observations. With rare outcomes, stratified folds can help ensure that each fold contains events, but stratification does not make dependent observations independent or turn an unrepresentative sample into a representative one.
State whether the target is a new record from a known group, a record from a new group, or a future observation. This determines what must be separated.
Keep groups intact when group membership creates dependence. For forecasting, keep training periods earlier than validation periods. Fit any preprocessing using training data only.
Specify the loss before examining results. Pool observation-level losses for an average per observation; use a fold average only when equal weighting of folds is the intended target.
If the same data are used to compare candidate models and report the winning score, the reported minimum is optimistic. Use an outer validation layer to evaluate the entire selection procedure.
The pooling rule follows directly from the definition of an average. If fold \(k\) contains \(n_k\) validation observations and has mean loss \(\widehat R_k\), its total loss is \(n_k\widehat R_k\). Summing those fold totals and dividing by the total number of validation observations gives
This is a weighted average of fold losses, with weights proportional to fold sizes. It equals the simple average \(K^{-1}\sum_k\widehat R_k\) when all folds have the same size, or when the particular fold losses happen to make the two averages equal. This justification concerns aggregation only; it does not establish that the resampling design estimates the intended deployment performance.
Worked Example: Time-Ordered Tuning and Forecast Evaluation
A hypothetical insurer forecasts quarterly claim counts. At each of four forecast origins, analysts use only earlier quarters for training. Within that historical training set, they compare two candidate forecasting procedures by rolling-origin validation and select the one with the smaller inner validation loss. The selected procedure then forecasts the next quarter. The four outer forecast errors, observed minus predicted, are \(2,-3,1,-4\) claims. Calculate the outer mean absolute error and root mean squared error, and explain what they estimate.
Method. Preserve time order at both stages: inner rolling-origin comparisons select a procedure using the available past, and each outer forecast evaluates that selection on the next period. The four outer observations are not used to choose the procedure that produced their respective forecasts.
The absolute errors are \(|2|=2\), \(|-3|=3\), \(|1|=1\), and \(|-4|=4\). Their sum is \(2+3+1+4=10\), so the outer mean absolute error is \(10/4=2.5\) claims per quarter.
The squared errors are \(2^2=4\), \((-3)^2=9\), \(1^2=1\), and \((-4)^2=16\). Their sum is \(4+9+1+16=30\), so the outer mean squared error is \(30/4=7.5\) claims squared. The outer root mean squared error is \(\sqrt{7.5}\approx2.7386\) claims.
These estimates describe the performance of the procedure that tunes among the two candidates using past data and then forecasts the next quarter. They do not describe a model fitted using future observations. A common trap is random shuffling: it could let a later quarter contribute to training for an earlier-quarter forecast, which violates the stated information boundary. The correction is time-ordered validation at both the tuning and outer evaluation stages.
Check. The RMSE exceeds the MAE here, as expected when squaring gives relatively more influence to larger errors such as \(4\). Both calculations use the same four forecasts and the same claim-count units before squaring.
Rare Outcomes and the Choice of Classification Metric
In an imbalanced classification problem, accuracy can look high even when the classifier misses many events. For a binary outcome, define a positive case as an event of interest. A true positive is an event predicted positive; a false negative is an event predicted negative; a false positive is a non-event predicted positive; and a true negative is a non-event predicted negative. Sensitivity is the fraction of actual positives identified, while specificity is the fraction of actual negatives identified. Balanced accuracy gives these two class-specific rates equal weight.
Stratification can distribute positive cases across folds, making fold-level metrics more calculable and less erratic when the event is rare. It does not justify changing the validation prevalence unless that change matches the intended use. In particular, oversampling events in training can be a modeling choice, but the held-out evaluation should represent the target population if the goal is performance there.
Worked Example: Reading Metrics Under Class Imbalance
A hypothetical fraud-screening model is evaluated on 100 independent applications: 10 are fraudulent and 90 are not. The pooled predictions produce 6 true positives, 4 false negatives, 9 false positives, and 81 true negatives. Calculate accuracy, sensitivity, specificity, precision, and balanced accuracy.
Method. Use the full held-out confusion counts for each metric. The data are independent application-level records for this example, and the event prevalence in the evaluation sample is \(10/100=0.10\).
Accuracy is the fraction classified correctly: \((6+81)/100=87/100=0.87\), or \(87\%\).
Sensitivity is the fraction of fraudulent applications identified: \(6/(6+4)=6/10=0.60\), or \(60\%\). Specificity is the fraction of non-fraudulent applications correctly rejected: \(81/(81+9)=81/90=0.90\), or \(90\%\).
Precision is the fraction of applications flagged as fraudulent that are actually fraudulent: \(6/(6+9)=6/15=0.40\), or \(40\%\). Balanced accuracy is \((0.60+0.90)/2=0.75\), or \(75\%\).
The common trap is to rely on the \(87\%\) accuracy alone and conclude that event detection is strong. In fact, the model misses \(4\) of the \(10\) fraud cases, so sensitivity is only \(60\%\). The correction is to report metrics that expose performance in each class and choose among them according to the decision costs. Precision also matters here: fewer than half of the flagged applications are fraudulent.
Check. The confusion counts reconcile: \(6+4=10\) actual positives, \(9+81=90\) actual negatives, and \(6+4+9+81=100\) total applications.
Selection, Leakage, and What the Score Evaluates
Every operation that learns from data belongs inside the resampling design. This includes imputation, scaling, feature screening, threshold selection, and hyperparameter tuning. If a preprocessing step uses the validation observations, information crosses the boundary even if the final model-fitting command excludes those observations. Likewise, if many candidate models are compared on the same validation folds and only the best score is reported, the winning score benefits from selection noise.
Worked Example: Nested Validation After Candidate Selection
A hypothetical underwriting team has 240 independent applications and compares 30 model configurations. It uses stratified five-fold outer validation, with 48 applications in each outer fold. Within each outer training set, it uses inner validation to choose one of the 30 configurations; it then evaluates that choice on the untouched outer fold. The outer-fold misclassification counts are \(12,14,13,13,12\). Calculate the pooled outer error rate and explain how to interpret an inner score of \(0.20\) reported for a selected configuration.
Method. The target is the error rate of the whole selection procedure, not the score of a configuration chosen with advance knowledge. Perform selection within each outer training set and pool the outer-fold errors.
The total number of outer errors is \(12+14+13+13+12=64\). The total number evaluated is \(5(48)=240\). Therefore the pooled outer error rate is \(64/240=0.2667\), or \(26.67\%\) when expressed as a percentage.
The inner score \(0.20\) is used to select a configuration; it is not a fresh evaluation of the selected procedure. Because the choice is made to favor low inner loss among 30 candidates, the reported minimum can be optimistic. The outer estimate of \(0.2667\) evaluates the repeated process of selecting within training data and predicting held-out applications. The difference \(0.2667-0.20=0.0667\) is a numerical comparison of these two reported values, not a universal correction to add to an inner score.
The trap is to report \(20\%\) as the final error rate simply because it is the lowest inner score. The correction is to report the untouched outer estimate for the selection procedure, while retaining inner validation for choosing the configuration. If an operation such as scaling or feature screening also uses data, it must be refitted inside the relevant training split, not once on all 240 applications.
Check. Each outer fold contributes exactly 48 applications, so pooling gives \(240\) evaluation cases. The calculated error proportion \(64/240\) is between \(0\) and \(1\), as a rate must be.
Practical Interpretation and Common Traps
A resampling estimate is not a context-free property of a fitted model. It is evidence about a specified prediction procedure, population, information boundary, and metric. A useful report states these elements explicitly: the unit separated across folds; whether time or groups constrain splitting; what preprocessing and tuning occurred inside training data; the loss or classification metric; and whether fold results were pooled or averaged equally.
- Dependence: Randomly splitting records can overstate performance when related records occur in both training and validation. Group the records at the level that must be new at deployment.
- Temporal leakage: Future observations must not train a forecast for an earlier time. Use a time-ordered design that matches the forecast horizon.
- Preprocessing leakage: A transformation estimated using all observations has already used validation information. Estimate it from each training portion and apply it to that portion's validation data.
- Imbalance: High accuracy does not establish good event detection. Examine class-specific metrics and preserve the target evaluation prevalence unless a different target is intended.
- Tuning optimism: The best score among many candidates is selected partly because of favorable noise. Use inner resampling for selection and an outer layer for evaluation when estimating the performance of the selection process.
- Aggregation: An equal fold average and a pooled observation-level estimate answer different questions when fold sizes differ. State which one is reported.
Check Your Understanding
For each hypothetical case, identify the key design or calculation and check your result against the answer.
- A grouped validation has two folds. Fold A has 3 held-out units and total squared error 12; fold B has 5 held-out units and total squared error 10. What is the pooled mean squared error per unit? (Answer: \(22/8=2.75\).)
- A forecast study has three outer errors \(-2, 0, 4\). What are the outer mean absolute error and root mean squared error? (Answer: MAE \(=2\); RMSE \(=\sqrt{20/3}\approx2.582\).)
- A classifier has 8 true positives, 2 false negatives, 12 false positives, and 78 true negatives. Find sensitivity, precision, and balanced accuracy. (Answer: sensitivity \(=0.8000\), precision \(=0.4000\), balanced accuracy \(=(0.8000+0.8667)/2=0.8333\).)
- A data set contains repeated records from each clinic, but the intended prediction is for patients at clinics never represented in training. Should records or clinics be held out, and what is the main risk of a record-level random split? (Answer: hold out whole clinics; a record-level split can let clinic-specific information appear in both training and validation, overstating performance for new clinics.)
- Analysts compare 40 candidate models using the same validation folds and report the lowest fold-averaged loss as the final estimate. What is the central concern, and what design change addresses it? (Answer: selection optimism because the winning score was used for both selection and evaluation; use inner resampling for selection and untouched outer validation to evaluate the selection procedure.)