Why Tuning Needs an Organized Resampling Plan
A model’s hyperparameters are settings chosen before or during fitting, such as a ridge penalty, a tree’s maximum depth, or the number of features considered at each split. Tuning means comparing candidate settings and selecting one according to a chosen performance measure. Because the same data are used to compare candidates, the resampling plan affects whether that comparison is fair and what conclusions the scores support.
A sound workflow separates three activities: fitting a candidate on a training portion, selecting hyperparameters using validation results, and assessing the performance of the selection procedure on data not used to make that selection. These are related, but they are not interchangeable. In particular, the best score observed while searching many settings is not automatically an unbiased estimate of future performance.
For example, if five-fold cross-validation is used, each candidate is fitted five times. On each repetition, it is trained on four folds and scored on the remaining fold. The fold membership should be fixed across candidates: candidate A’s first fit and candidate B’s first fit should be trained and validated on the same observations. This makes the comparison less sensitive to accidental differences in which records were held out.
Separate Fitting, Selection, and Assessment
The tuning resamples estimate how candidate configurations perform under the chosen metric. The configuration with the best average validation result is selected; for a loss metric, “best” usually means smallest, whereas for a score such as accuracy it usually means largest. The selected configuration can then be refitted using the available development data.
However, reporting the best tuning score as if it were an independent performance assessment is generally optimistic. The selection process favors candidates that performed well partly because of favorable variation in their validation results. When a separate estimate of the complete tuning procedure is needed, an outer resampling loop can assess it. In nested cross-validation, each outer training portion has its own inner cross-validation for tuning. The outer held-out portion is used only after the inner process has selected a configuration.
List the hyperparameter settings to compare and define in advance whether larger or smaller metric values are preferred.
Create fold assignments appropriate to the data structure, then apply the same assignments to every candidate.
Fit preprocessing and the candidate model using only that resample’s training portion; calculate its score on the held-out portion.
Aggregate the validation results according to the declared metric and choose a configuration using a stated rule.
Use an outer resampling loop or a protected assessment sample to evaluate the selection procedure without using that assessment result to choose the configuration.
A fold assignment must also respect the data. If the same policyholder contributes multiple records, records from that person should not be split across training and validation when the target is performance on new policyholders. For time-ordered prediction, the assignment must preserve the relevant time direction. These constraints determine the resampling design; comparability means using the same valid design for every candidate, not ignoring the structure of the observations.
Worked Example: Selecting a Ridge Penalty on Shared Folds
A hypothetical insurer is tuning a ridge regression model to predict claim severity. The tuning metric is validation RMSE in thousands of dollars, so smaller values are preferred. Three penalty values are evaluated on the same four folds. The fold RMSE values are:
| Penalty | Fold 1 | Fold 2 | Fold 3 | Fold 4 |
|---|---|---|---|---|
| 0.01 | 8 | 10 | 12 | 10 |
| 0.10 | 9 | 9 | 10 | 8 |
| 1.00 | 10 | 8 | 9 | 10 |
Method. Calculate each candidate’s mean fold RMSE and select the penalty with the smallest mean. The folds have equal validation sizes, so the unweighted mean of their fold RMSE values is suitable for this example.
For penalty 0.01, the mean is \((8+10+12+10)/4=40/4=10\) thousand dollars.
For penalty 0.10, the mean is \((9+9+10+8)/4=36/4=9\) thousand dollars.
For penalty 1.00, the mean is \((10+8+9+10)/4=37/4=9.25\) thousand dollars. The selected penalty is therefore 0.10, with a mean tuning RMSE of 9 thousand dollars.
The next fitting step is to refit the ridge model with penalty 0.10 using all available development data. The value 9 is a tuning result, not a separate unbiased estimate of the final model’s performance, because the same fold results were used to select the penalty. The shared assignments make the candidate comparison more controlled; they do not remove the optimism caused by selecting the best result.
Why Shared Folds Improve the Comparison
Let \(L_{jk}\) be the validation loss for candidate \(j\) on fold \(k\). To compare candidates A and B, the relevant fold-level quantity is the paired difference \(D_k=L_{Ak}-L_{Bk}\). Both losses in a difference come from the same held-out observations. If a fold happens to be unusually difficult, both candidates may have higher losses on it; pairing helps control for that shared difficulty.
This identity follows by expanding the variance of a difference: for random variables \(U\) and \(V\), \(\operatorname{Var}(U-V)=E[((U-E[U])-(V-E[V]))^2]\). Expanding the square gives the two variances minus twice the covariance. When the candidates’ losses tend to move together across the same folds, the covariance is positive, reducing the variance of their difference compared with a comparison on unrelated folds. This is the justification for paired comparisons on shared folds. Positive covariance is not guaranteed, so shared assignments do not promise a particular numerical reduction.
The fold scores are also not independent replicates in the ordinary sense: their training sets overlap. Treating the number of folds as if it were a large independent sample can therefore overstate the precision of a comparison. The practical purpose of shared folds is a fair, controlled comparison, not a claim that fold-to-fold variation alone provides a complete uncertainty analysis.
Worked Example: Nested Cross-Validation Separates Tuning From Assessment
A hypothetical model team is comparing tree depths 2, 4, and 6 using RMSE. It uses three outer folds. Within each outer training portion, it conducts three-fold inner cross-validation with shared assignments for all three depths. The inner fold RMSE values and the loss on the corresponding outer held-out fold are shown below. All outer held-out folds contain 30 observations.
| Outer fold | Depth 2 inner RMSEs | Depth 4 inner RMSEs | Depth 6 inner RMSEs | Selected depth | Outer RMSE |
|---|---|---|---|---|---|
| 1 | 11, 12, 13 | 9, 10, 11 | 10, 11, 12 | 4 | 13 |
| 2 | 8, 9, 10 | 9, 10, 11 | 7, 8, 9 | 6 | 9 |
| 3 | 10, 11, 12 | 8, 9, 10 | 9, 10, 11 | 4 | 10 |
Method. In each outer fold, average the inner RMSEs, choose the depth with the smallest inner mean, fit that depth on the full outer training portion, and evaluate it once on the outer held-out fold.
For outer fold 1, the depth means are \((11+12+13)/3=12\), \((9+10+11)/3=10\), and \((10+11+12)/3=11\). Depth 4 is selected. Its outer RMSE is 13.
For outer fold 2, the means are \((8+9+10)/3=9\), \((9+10+11)/3=10\), and \((7+8+9)/3=8\). Depth 6 is selected. Its outer RMSE is 9.
For outer fold 3, the means are \((10+11+12)/3=11\), \((8+9+10)/3=9\), and \((9+10+11)/3=10\). Depth 4 is selected. Its outer RMSE is 10.
Because the outer folds are equal in size, their mean RMSE is \((13+9+10)/3=32/3=10.6667\), or 10.67 when rounded to two decimals. This estimates the performance of the tuning procedure on outer held-out data. The outer losses were not used to choose the depth within their respective outer folds, which is the key separation. The result does not say that depth 4 alone has RMSE 10.67; the selected depth varied across outer training samples.
Metric Aggregation and a Common Trap
When validation folds have unequal sizes, the aggregation rule matters. If the target is observation-level RMSE across all held-out predictions, calculate the total squared error divided by the total number of observations, then take the square root. Averaging fold RMSEs gives each fold equal weight instead, which can represent a different target. State the intended aggregation rather than assuming the two calculations agree.
Worked Example: Unequal Fold Sizes Can Change the Winner
Two hypothetical severity models are evaluated on the same four grouped folds, containing 10, 30, 30, and 30 observations. Their sums of squared errors (SSE) are:
| Fold size | Model A SSE | Model B SSE |
|---|---|---|
| 10 | 160 | 28 |
| 30 | 30 | 84 |
| 30 | 30 | 84 |
| 30 | 30 | 84 |
Method. The target is pooled observation-level RMSE, so combine squared errors before taking the square root. There are \(10+30+30+30=100\) observations.
For Model A, total SSE is \(160+30+30+30=250\). Its pooled RMSE is \(\sqrt{250/100}=\sqrt{2.5}\approx1.5811\).
For Model B, total SSE is \(28+84+84+84=280\). Its pooled RMSE is \(\sqrt{280/100}=\sqrt{2.8}\approx1.6733\). On the stated target, Model A is better.
Common mistake. Averaging the four fold RMSEs equally would give Model A \((\sqrt{160/10}+\sqrt{30/30}+\sqrt{30/30}+\sqrt{30/30})/4=(4+1+1+1)/4=1.75\). Model B has fold RMSE \(\sqrt{28/10}=\sqrt{84/30}=\sqrt{2.8}\approx1.6733\) in every fold, so its unweighted fold mean is about 1.6733. That calculation would incorrectly declare Model B the winner for the observation-level target. Equal fold weighting can be appropriate if the target is average performance across folds, but it is not the same as pooling all observations.
Implementation Checks Before Trusting a Tuning Result
An organized tuning workflow makes its choices auditable. Keep the candidate list, metric direction, fold assignments, and aggregation rule explicit. If preprocessing is needed, such as imputing missing values or scaling predictors, estimate its parameters using only the training portion of each resample, then apply those fitted transformations to that resample’s validation portion. Otherwise validation information can influence the fitted pipeline.
For group-based or time-based data, generate assignments that respect the relevant boundaries, then reuse those assignments for every candidate. If a candidate cannot be fitted on a particular training portion, record and handle that failure consistently rather than quietly changing the folds for that candidate. A different set of validation records can make one candidate appear better simply because its evaluation conditions were easier.
- Fair comparison: hold the resampling assignments fixed across candidate configurations.
- Valid fitting: fit preprocessing and the model using only the current training portion.
- Clear selection: define the metric, its preferred direction, and the aggregation rule.
- Honest assessment: do not present the best tuning score as an independent assessment of the selected model.
The resulting workflow is aligned with the SOA Exam PA (Predictive Analytics) syllabus because it requires candidates to distinguish model selection from performance assessment, apply resampling consistently, and interpret validation results in context. A small improvement in a tuning score should not be overstated: it is a comparison under a particular data design, metric, candidate set, and resampling scheme.
Check Your Understanding
For each problem, identify the relevant tuning or assessment principle and work through the calculations before checking the answer.
- A candidate has fold RMSEs of 6, 8, and 10, while another has 7, 7, and 9, on the same three equal-sized folds. Calculate each mean and select the better candidate. (Answer: The first mean is 8; the second is \(23/3\approx7.6667\), so select the second candidate.)
- In a nested design, the inner means for three settings are 5.2, 4.8, and 5.0. The selected setting has outer-fold RMSE 6.1. Which value is used for selection, and which is the assessment score for that outer fold? (Answer: Select the second setting using inner mean 4.8; the outer assessment score is 6.1.)
- Two candidates are evaluated on different random folds, and one has a smaller mean RMSE. Why is the difference harder to interpret than if both had used the same folds? (Answer: Candidate performance is confounded with differences in the held-out observations; shared folds provide a paired comparison under the same validation conditions.)
- Three validation folds have sizes 20, 20, and 60, with SSEs 80, 80, and 240 for one model. Calculate its pooled observation-level RMSE. (Answer: Total SSE \(=400\), total size \(=100\), so RMSE \(=\sqrt{400/100}=2\).)
- A tuning procedure selects a different tree depth in each of four outer folds. What does the mean outer-fold loss assess: one fixed depth or a procedure? (Answer: It assesses the tuning procedure, including its selection step, because each outer training portion selected its own depth.)