Reading Boosting Error Patterns
Boosting adds learners sequentially, so a model’s fit can change as the number of boosting rounds grows. The practical question is not simply whether a later model fits the training data better. It is whether its added complexity also improves performance on observations that were not used to fit it. Training and validation errors help answer that question, provided they are calculated on the same response scale and with a clearly stated metric.
A low training error alone does not establish that a model predicts well on new data. Conversely, a larger training error is not automatically evidence of underfitting: a model that generalizes well may reasonably leave some training error. Diagnose a pattern across model complexity, rather than assigning a label from one number.
For example, with squared-error loss and observed responses \(y_i\), predictions \(\hat f_M(x_i)\), and \(n\) observations, mean squared error is \(n^{-1}\sum_{i=1}^{n}(y_i-\hat f_M(x_i))^2\). Apply this formula separately to the training and validation samples. If the response is measured in dollars, the mean squared error is in squared dollars; the root mean squared error is in dollars. The metric must be stated before comparing values.
Three Diagnostic Patterns
Underfitting is a plausible diagnosis when both training and validation errors remain relatively high and similar, and reasonable increases in model complexity improve both. The model may be too restricted, or it may not have undergone enough boosting rounds to capture structure. Error curves alone do not identify the cause: feature limitations, the chosen loss, and the data itself can also constrain performance.
Useful added complexity is indicated when adding boosting rounds or changing a configuration improves validation error as well as training error. A falling training error is expected in many boosting procedures, but the important evidence for useful complexity is a validation improvement measured consistently on the same validation observations.
Overfitting is a concern when training error continues to decrease while validation error rises or stops improving over a sustained range of added complexity. This pattern suggests the model is fitting training-specific detail that does not transfer well. A small one-round rise is not conclusive: validation error varies with the validation sample, and the pattern should be interpreted in context.
| Pattern across complexity | Possible interpretation | What to check |
|---|---|---|
| Training and validation errors are both high; both improve as complexity increases | Underfitting at the simpler settings | Whether added complexity improves validation error, not just training fit |
| Both errors decrease | Added complexity is useful over the range examined | Whether validation improvement is consistent and measured on the same scale |
| Training error decreases while validation error rises for several settings | Overfitting is plausible | Whether the increase is sustained and not just small validation-sample variation |
| Errors move irregularly or disagree across metrics | No simple diagnosis from one curve | The metric, threshold, split, and size of each change |
Why Training Loss Often Falls
There is a simple reason training loss often decreases during boosting, but the conclusion has conditions. Suppose the current fitted function is \(f\), and the next stage considers an update \(f+\alpha h\), where \(h\) is a proposed learner and \(\alpha\) is a step size. If the training procedure minimizes the empirical loss over \(\alpha\) and allows \(\alpha=0\), then the best permitted update cannot have greater training loss than taking no update.
The inequality follows because \(\alpha=0\) is one of the choices in the minimization; it gives exactly the current training loss. This justifies a nonincrease when the update is truly minimized over a set containing zero. It does not guarantee that every implementation has a strictly decreasing loss at every round. Approximate updates, subsampling, constraints, or a prescribed step that is not chosen by this minimization can alter the pattern. Most importantly, the inequality says nothing about validation loss: the validation observations were not the objective of that minimization.
Worked Examples
Worked Example: Locating the Onset of Overfitting in Regression
A hypothetical insurer models claim severity in thousands of dollars using squared-error loss. Assume the response is evaluated empirically, with no probability distribution imposed for this comparison. On the same training and validation samples, the residuals at three boosting rounds are as follows. At round 0, the four training residuals are \(4,-4,2,-2\), and the four validation residuals are \(3,-3,2,-2\). At round 10, they are \(2,-2,1,-1\) and \(1,-2,2,-2\). At round 40, they are \(1,-1,0.5,-0.5\) and \(0.5,-2.5,3,-3\). Compute the training and validation mean squared errors and interpret them.
Method. For each sample and round, square each residual and divide the sum by four. The squared-error loss is being compared on the same response scale and sample split at every round.
At round 0, the training sum of squared residuals is \(4^2+(-4)^2+2^2+(-2)^2=16+16+4+4=40\). The training MSE is \(40/4=10\). The validation sum is \(3^2+(-3)^2+2^2+(-2)^2=9+9+4+4=26\), so validation MSE is \(26/4=6.5\).
At round 10, the training sum is \(2^2+(-2)^2+1^2+(-1)^2=4+4+1+1=10\), giving MSE \(10/4=2.5\). The validation sum is \(1^2+(-2)^2+2^2+(-2)^2=1+4+4+4=13\), giving MSE \(13/4=3.25\).
At round 40, the training sum is \(1^2+(-1)^2+0.5^2+(-0.5)^2=1+1+0.25+0.25=2.5\), giving MSE \(2.5/4=0.625\). The validation sum is \(0.5^2+(-2.5)^2+3^2+(-3)^2=0.25+6.25+9+9=24.5\), giving MSE \(24.5/4=6.125\).
Thus, the training MSE falls from \(10\) to \(2.5\) to \(0.625\), while validation MSE falls from \(6.5\) to \(3.25\) and then rises to \(6.125\). Round 10 has the lowest validation MSE among these three settings, so its added complexity appears useful. The round-40 pattern is consistent with overfitting relative to this validation sample. The common mistake is to select round 40 because it has the lowest training MSE; that ignores the deterioration in held-out performance. The computed MSE values are in squared thousands of dollars.
Worked Example: Probability Loss and Classification Error Disagree
A hypothetical fraud-screening model predicts a binary outcome \(Y\), treated as Bernoulli with predicted probability \(p\) for \(Y=1\). Four training labels are \(1,1,0,0\). At rounds 1, 2, and 3, their predicted probabilities are respectively \((0.55,0.55,0.45,0.45)\), \((0.80,0.70,0.30,0.20)\), and \((0.95,0.40,0.10,0.05)\). For validation, labels are \(1,0,1,0\), and the corresponding probabilities are \((0.55,0.45,0.55,0.45)\), \((0.70,0.30,0.60,0.40)\), and \((0.80,0.20,0.45,0.55)\). Compute log loss and classification error at each round. Use threshold \(0.5\), with probability \(0.5\) classified as 1.
Method. For a binary label \(y\), log loss is \(-[y\ln(p)+(1-y)\ln(1-p)]\); average these four losses. Classification error is the fraction of thresholded predictions that disagree with the labels. Report log loss to four decimals.
For the training sample at round 1, each observation has the probability \(0.55\) assigned to its correct class, so each loss is \(-\ln(0.55)=0.5978\). The mean log loss is \(0.5978\), and all four predictions are correct, so classification error is \(0/4=0\).
At round 2, the four correct-class probabilities are \(0.80,0.70,0.70,0.80\). Their mean log loss is \(-[2\ln(0.80)+2\ln(0.70)]/4=-\ln(0.56)/2=0.2899\). All predictions are still correct, so error is \(0/4=0\).
At round 3, the correct-class probabilities are \(0.95,0.40,0.90,0.95\). The mean loss is \(-[\ln(0.95)+\ln(0.40)+\ln(0.90)+\ln(0.95)]/4=-\ln(0.3249)/4=0.2811\). The second positive observation is classified as 0 because \(0.40<0.5\), so classification error is \(1/4=0.25\).
For validation, round 1 again has four correct-class probabilities of \(0.55\), so log loss is \(0.5978\) and error is \(0/4=0\). At round 2 the correct-class probabilities are \(0.70,0.70,0.60,0.60\), giving mean loss \(-\ln(0.70\cdot0.60)/2=-\ln(0.42)/2=0.4338\); all four are correctly classified, so error is 0. At round 3 they are \(0.80,0.80,0.45,0.45\), giving mean loss \(-\ln(0.80\cdot0.80\cdot0.45\cdot0.45)/4=-\ln(0.1296)/4=0.5108\). The last two predictions are wrong at threshold \(0.5\), so error is \(2/4=0.50\).
Training log loss decreases at each round, but training classification error rises at round 3. Validation log loss and classification error both worsen from round 2 to round 3. The evidence flags deterioration for the later setting, but the measures are not interchangeable: probability loss accounts for confidence, while classification error only counts threshold crossings. A common mistake is to say that round 3 improves every training metric because its log loss is lower. The calculations show otherwise.
Worked Example: Choosing Among Boosting Configurations
A hypothetical model estimates the annual number of service requests per policy using squared-error loss. For a fixed split and the same response units, a team reports these RMSE values: a depth-1 model with 100 rounds has training RMSE \(9.0\) and validation RMSE \(10.2\); a depth-2 model with 100 rounds has \(5.8\) and \(6.4\); a depth-5 model with 500 rounds has \(2.0\) and \(8.7\). A baseline prediction has validation RMSE \(12.0\). Compare the configurations and select the best of the three by validation RMSE.
Method. For each configuration, subtract training RMSE from validation RMSE to describe the gap, then compare validation RMSE with the baseline. These differences are descriptive; RMSE gaps are not formal tests of statistical significance.
For depth 1, the gap is \(10.2-9.0=1.2\). Its validation improvement over baseline is \(12.0-10.2=1.8\), or \(1.8/12.0=0.15=15\%\).
For depth 2, the gap is \(6.4-5.8=0.6\). Its baseline improvement is \(12.0-6.4=5.6\), or \(5.6/12.0=0.4667\), approximately \(46.7\%\).
For depth 5, the gap is \(8.7-2.0=6.7\). Its baseline improvement is \(12.0-8.7=3.3\), or \(3.3/12.0=0.275=27.5\%\). It has the smallest training RMSE but not the smallest validation RMSE. The depth-2 configuration is best among the three by the stated selection rule because \(6.4<8.7<10.2\). The shallow model may be too restricted relative to the alternatives, while the deep model’s much larger gap is consistent with overfitting. The common mistake is to choose the configuration with the lowest training RMSE; the stated objective is lowest validation RMSE, which selects depth 2.
Common Diagnostic Traps
- Comparing different metrics. A decrease in log loss and an increase in classification error can both be correct. State the metric and threshold before interpreting a curve.
- Treating a gap as a diagnosis by itself. A positive validation-minus-training gap is common. Its size, trend, and validation performance relative to alternatives matter.
- Reacting to one small fluctuation. Validation error is calculated from a finite sample. A small rise may be noise rather than a clear transition to overfitting.
- Comparing incompatible results. A conclusion is difficult to interpret if configurations use different validation observations, response scales, or loss functions.
- Choosing solely on training fit. Training error measures fit to the data used for fitting, not performance on new observations.
These diagnostics are aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus. In applied work, the validation set may be used repeatedly to compare settings, so the best observed validation result is not a guarantee of future performance. Keep conclusions proportional to the evidence: error patterns diagnose warning signs and useful changes, but do not by themselves prove why a model behaves as it does.
Check Your Understanding
For each problem, use the stated metric and explain the diagnostic pattern before checking the answer.
- A regression booster has training and validation MSEs of \(18,20\) at 20 rounds, \(12,14\) at 60 rounds, and \(8,19\) at 120 rounds. Which range shows useful added complexity, and what does the last setting suggest? (Answer: Complexity through 60 rounds is useful because both errors fall; the 120-round result is consistent with overfitting because training MSE falls while validation MSE rises.)
- Two classification models have training and validation log losses \(0.42,0.48\) and \(0.30,0.44\), respectively. Which has lower validation loss, and how does its validation-minus-training gap compare? (Answer: The first has lower validation loss, \(0.48<0.44\) is false; the second is lower at \(0.44\). Its gap is \(0.44-0.30=0.14\), versus \(0.48-0.42=0.06\) for the first, so it has the lower validation loss but the larger gap.)
- A binary classifier predicts probabilities \(0.49,0.51,0.20,0.80\) for labels \(0,1,0,1\), using threshold \(0.5\). What is its classification error? (Answer: \(0/4=0\); every thresholded prediction matches its label.)
- For a regression model, training RMSE decreases from \(5.0\) to \(3.0\), while validation RMSE decreases from \(6.0\) to \(4.2\). Calculate the two gaps and interpret the change. (Answer: The gaps are \(6.0-5.0=1.0\) and \(4.2-3.0=1.2\); validation performance improves, though the gap widens slightly, so the added complexity is useful over this comparison.)
- Three configurations have validation errors \(7.1,6.8,7.0\), measured on the same finite validation sample. What is the smallest observed error, and what caution is appropriate? (Answer: \(6.8\) is the smallest observed error; the differences are small, so do not treat the middle configuration as decisively better without considering validation-sample variability and other evidence.)