Why Plot a Validation Curve?
A boosting model has settings that control its flexibility and fitting process. A validation curve shows how a chosen performance measure changes as one such setting varies. Its purpose is not simply to name the setting with the smallest observed error. It is to show whether performance improves, deteriorates, or levels off over the tested range, while making clear which other settings were held constant.
For a fair comparison, fit each candidate model on the same training data and evaluate it using the same validation data, or use the same cross-validation folds for each candidate. This reduces the chance that an apparent improvement is really due to a different data split. The validation results guide model selection; a separate test set, if available, should not be used repeatedly to choose settings.
The horizontal axis contains the candidate settings that were actually evaluated. The plotted line is a visual guide between those points, not evidence that every intermediate value was tested. A curve is conditional: it describes the effect of one setting at the particular values chosen for all the others. That qualification is essential when interpreting the result.
Constructing and Reading a Curve
Before interpreting any curve, record the target hyperparameter, its tested values, the fixed settings, the data split or folds, and the metric. For example, a tree-depth curve is incomplete if the learning rate, number of boosting iterations, and other controls are not specified. If those controls change between candidate depths, the plotted differences cannot be attributed to depth alone.
Select one hyperparameter and a set of plausible candidate values. Keep the other model settings unchanged.
Fit each candidate using the same training data and calculate the same validation metric on the same observations or folds.
Identify the best observed point, any plateau, and whether performance worsens at one or both ends of the tested range.
Consider the size and stability of score differences, not just the location of the smallest or largest plotted value.
For a regression problem with \(n\) validation observations, let \(y_i\) be the observed response and \(\hat{y}_i\) the prediction for observation \(i\). The root mean squared error is
It is measured in the response’s units, and lower values indicate smaller squared prediction errors on average. When evaluating a classification model, use a metric suited to the problem, such as validation log loss or AUC, and interpret the direction correctly. A metric should be selected before comparing candidates, rather than chosen afterward because it makes a preferred model look best.
Worked Examples
Worked Example: Finding a Useful Tree-Depth Range
A hypothetical insurer models the dollar amount of a claim using gradient boosting. A validation set is held aside from fitting. The learning rate is fixed at 0.05, the number of boosting iterations at 300, and all other settings are unchanged. The following RMSE values are observed, in dollars.
| Maximum tree depth | Training RMSE | Validation RMSE |
|---|---|---|
| 1 | 12.2 | 12.5 |
| 2 | 8.7 | 10.1 |
| 3 | 7.0 | 9.2 |
| 4 | 6.3 | 9.0 |
| 5 | 5.9 | 9.1 |
Method. Compare validation RMSE across depths, since the objective is prediction on data not used to fit the trees. Moving from depth 1 to depth 2 reduces validation RMSE by \(12.5-10.1=2.4\) dollars, or \(2.4/12.5=0.192\), a 19.2% reduction. Moving from depth 2 to 3 reduces it by \(10.1-9.2=0.9\) dollars, or \(0.9/10.1\approx0.0891\), an 8.91% reduction. Moving from depth 3 to 4 reduces it by just \(9.2-9.0=0.2\) dollars, or \(0.2/9.2\approx0.0217\), a 2.17% reduction. Depth 5 increases validation RMSE by \(9.1-9.0=0.1\) dollar compared with depth 4.
Depth 4 has the lowest observed validation RMSE, but depths 3 through 5 are a plausible useful range based on these results: performance has flattened after depth 3, and the best value is not sharply separated from its neighbors. The scores alone do not establish that depth 4 is reliably better than depth 3. That would require considering variation across validation splits or other uncertainty in the comparison.
Check. The chosen setting has validation RMSE \(9.0\), lower than the other four validation values. The calculation uses dollars throughout; no unit conversion is needed. The common mistake is to choose depth 5 because it has the smallest training RMSE, \(5.9\). That would select using fit to training data, not the stated validation objective.
Worked Example: Averaging Validation Scores Across Folds
A hypothetical analyst compares row-subsampling fractions for a boosting model. Four cross-validation folds are used, with the same folds for every candidate. The learning rate, tree depth, iteration count, and other settings remain fixed. The fold RMSE values are:
| Subsampling fraction | Fold 1 | Fold 2 | Fold 3 | Fold 4 |
|---|---|---|---|---|
| 0.5 | 8.2 | 7.8 | 8.4 | 7.6 |
| 0.7 | 7.7 | 7.9 | 7.5 | 7.7 |
| 0.9 | 7.6 | 7.8 | 7.7 | 7.7 |
| 1.0 | 7.8 | 7.6 | 7.9 | 7.5 |
Method. For each candidate, calculate the arithmetic mean of the four fold RMSE values. At fraction 0.5, the mean is \((8.2+7.8+8.4+7.6)/4=32.0/4=8.0\). At fraction 0.7, it is \((7.7+7.9+7.5+7.7)/4=30.8/4=7.7\). At fraction 0.9, it is \((7.6+7.8+7.7+7.7)/4=30.8/4=7.7\). At fraction 1.0, it is \((7.8+7.6+7.9+7.5)/4=30.8/4=7.7\).
The mean validation RMSE is lower at 0.7, 0.9, and 1.0 than at 0.5. The curve therefore suggests that 0.5 is less attractive in this tested range, while 0.7 to 1.0 is a useful range for further consideration. There is no basis here for declaring one of the three tied values best by average score. The fold scores also remind us that a mean is a summary: it does not show every aspect of variation across folds.
Check. All candidates were scored on four folds, so each mean uses four values. The common mistake is to choose 0.9 because one fold has the smallest score, 7.6. The comparison calls for the average across folds, not the best single fold.
Worked Example: Detecting a Parameter Interaction
A hypothetical validation study considers tree depth and minimum leaf size, measured as the minimum number of observations in a terminal leaf. All other settings and the validation observations are fixed. The validation RMSE values are:
| Maximum depth | Minimum leaf size 5 | Minimum leaf size 20 |
|---|---|---|
| 2 | 10.0 | 10.1 |
| 4 | 10.2 | 7.8 |
Method. Start at depth 2 and minimum leaf size 5, where RMSE is 10.0. If depth alone is changed to 4, RMSE rises to 10.2, a worsening of \(10.2-10.0=0.2\). If minimum leaf size alone is changed to 20, RMSE rises to 10.1, a worsening of \(10.1-10.0=0.1\). Yet making both changes gives RMSE 7.8, an improvement of \(10.0-7.8=2.2\), or \(2.2/10.0=22\%\), relative to the starting value.
The effect of depth depends on minimum leaf size: at leaf size 5, increasing depth changes RMSE by \(+0.2\); at leaf size 20, it changes RMSE by \(7.8-10.1=-2.3\). The corresponding curves would have very different shapes. This is evidence of an interaction in the observed validation results: the effect of one setting is conditional on the other.
Check. Comparing only changes from the starting combination would suggest keeping both starting values, because each individual change worsens the score. That conclusion misses the better combined setting. The four scores are illustrative validation results, not a claim that these settings will have the same effects in another data set.
Why One-at-a-Time Curves Can Mislead
A curve isolates one setting only at the particular values chosen for the others. If the effect of a hyperparameter changes with another hyperparameter, a curve at one fixed setting may not represent its effect elsewhere. The worked interaction illustrates this directly: the depth curve at minimum leaf size 5 suggests that greater depth is slightly worse, while the curve at leaf size 20 suggests that greater depth is substantially better.
One-at-a-time comparisons are still valuable. They can identify an obviously poor region, show that a metric has flattened, and make results easier to explain. They are not a guarantee that the best joint combination has been found. A practical interpretation should state the fixed settings and avoid presenting a conditional curve as a universal effect.
The direction of improvement also depends on the metric. For RMSE or log loss, smaller is better; for AUC, larger is better. If a comparison includes a metric whose direction is not stated, the conclusion can be reversed. And if candidate differences are very small, a single validation split may not support a confident ranking. A broad, stable region is often more informative than a tiny apparent advantage at one point.
Common Mistakes and Exam Traps
- Choosing by training performance. Training error often continues to fall as a model becomes more flexible. Use the specified validation metric to compare candidate settings.
- Changing multiple settings without saying so. If more than one value changes, the result does not isolate the effect of the named hyperparameter.
- Calling the smallest score a proven winner. The best observed candidate is only best among those tested, under the stated split and fixed settings. Small differences may not be meaningful.
- Ignoring metric direction. Select the minimum for an error measure such as RMSE, but the maximum for a measure such as AUC.
- Reusing the test set for tuning. Repeated selection based on test performance turns that set into part of model development. Keep model selection on training and validation data.
- Assuming the curve applies at every setting of other parameters. A one-parameter curve is conditional; interactions can change its shape and the preferred range.
Check Your Understanding
For each problem, identify the relevant comparison and check the arithmetic before interpreting the result.
- A depth curve reports validation MAE values of 5.2, 4.6, 4.4, and 4.5 for depths 1 through 4. All other settings are fixed. Which depth has the lowest observed MAE, and what range appears worth considering? (Answer: Depth 3 has the lowest value, 4.4; depths 2–4 are a reasonable range to consider because the scores are close after depth 2.)
- For one candidate, validation residuals are \(1,-2,2,-1\). Calculate its RMSE. (Answer: \(\sqrt{(1+4+4+1)/4}=\sqrt{2.5}\approx1.58\).)
- Three-fold validation RMSE values for two settings are 6, 8, 7 and 7, 7, 7. Calculate each mean and state which has the lower mean. (Answer: The means are \(21/3=7\) and \(21/3=7\); they tie on mean RMSE.)
- AUC values for candidate settings are 0.71, 0.76, and 0.74, with every other setting fixed. Which value is best under AUC, and what is the best observed AUC? (Answer: Select the setting with AUC 0.76; higher AUC is better.)
- At leaf size 10, depths 2 and 4 have validation RMSE 9.0 and 9.2. At leaf size 30, those depths have RMSE 9.1 and 7.9. What does this suggest about the depth effect? (Answer: It depends on leaf size: increasing depth worsens RMSE by 0.2 at leaf size 10 but improves it by 1.2 at leaf size 30, suggesting an interaction.)