Why Ensemble Tuning Needs Care
A tree ensemble can have many settings that affect predictive performance. For a random forest, these include the number of candidate predictors considered at each split, the minimum leaf size, and the number of trees. For boosting, the number of iterations, tree depth, and learning rate determine how the fitted sequence develops. A model that is too simple may miss useful structure; a model that is too complex may adapt to noise or to quirks of the evaluation data.
The tuning task is to compare candidate settings using data that are not used to fit each candidate model. Random forests provide a convenient internal estimate called out-of-bag error. Cross-validation provides another way to compare settings and is broadly applicable, including to boosting. In either case, a separate test set should remain untouched until the modeling decisions are complete.
Out-of-Bag Error in a Random Forest
A bootstrap sample of size \(n\) is drawn with replacement from \(n\) training observations. For any particular observation, the probability of not being selected on one draw is \(1-1/n\). Its probability of being absent from all \(n\) draws is therefore \((1-1/n)^n\), which approaches \(e^{-1}\), about 0.368, as \(n\) grows. Thus, a typical tree is fit using about 63.2% of the distinct training observations, leaving about 36.8% out of that tree's bootstrap sample.
For a given training observation, its OOB prediction is formed from the trees for which it was absent from the bootstrap sample. For a regression forest, this is ordinarily the average of those trees' predictions. For a classification forest, it can be formed from the OOB vote proportions, with a chosen rule converting votes into a class prediction. An OOB error then compares the observed outcomes with the corresponding OOB predictions using a specified loss, such as mean squared error for regression or misclassification rate for classification.
OOB error is useful because it does not require setting aside a separate validation fold for every tree. It is an internal estimate, however, not an independent test-set result. The estimate can be noisy if there are few trees or few observations, and repeatedly choosing settings to minimize the same OOB error can make that minimum optimistic. Check that each observation has enough OOB trees contributing to its prediction.
Worked Example: Comparing Two Forest Sizes by OOB MSE
A hypothetical insurer models annual claim cost in thousands of dollars. The training outcomes for four observations are \(10, 12, 14,\) and \(16\). For each observation, the table gives its OOB prediction from a forest with 20 trees and from a forest with 80 trees. Assume both predictions for each observation were formed only from trees that left that observation out. Compare the forests by OOB MSE.
| Observation | Outcome | 20-tree OOB prediction | 80-tree OOB prediction |
|---|---|---|---|
| 1 | 10 | 11 | 10.5 |
| 2 | 12 | 11 | 12.5 |
| 3 | 14 | 15 | 13 |
| 4 | 16 | 15 | 17 |
Method. Calculate each squared prediction error, average the four errors, and select the forest with the lower OOB MSE.
20 trees. The errors are \(10-11=-1\), \(12-11=1\), \(14-15=-1\), and \(16-15=1\). Their squares are \(1,1,1,1\), so \[ \operatorname{OOB\ MSE}_{20}=(1+1+1+1)/4=1. \]
80 trees. The errors are \(10-10.5=-0.5\), \(12-12.5=-0.5\), \(14-13=1\), and \(16-17=-1\). Their squares are \(0.25,0.25,1,1\), so \[ \operatorname{OOB\ MSE}_{80}=(0.25+0.25+1+1)/4=2.5/4=0.625. \]
The 80-tree forest has the lower OOB MSE, \(0.625\) rather than \(1\), in squared thousands of dollars. This comparison supports choosing 80 trees among these two candidates, subject to adequate OOB coverage and the broader tuning plan. The common mistake is to calculate error using each observation's prediction from all trees, including trees that used the observation to fit. That is training error, not OOB error; the correction is to use only the trees that excluded that observation.
Cross-Validation for Ensemble Complexity
In \(K\)-fold cross-validation, divide the available training data into \(K\) folds. For each fold \(k\), fit the candidate model on the other \(K-1\) folds and calculate its loss on fold \(k\). The average fold loss estimates how the candidate may predict observations not used to fit it. To compare settings fairly, use the same fold assignments for each setting whenever practical.
For boosting, a central complexity choice is the number of iterations \(B\). Each additional boosting iteration adds another tree to the sequence. A learning rate also affects the path: a smaller rate often requires more iterations, so these settings may need to be considered together. Cross-validation can compare combinations of iterations, learning rates, and tree sizes. Some stochastic boosting implementations offer an OOB estimate based on observations omitted from a subsample at each step, but ordinary deterministic boosting does not automatically provide the same bootstrap OOB predictions as a random forest. Cross-validation is a generally applicable comparison method.
Here \(\lambda\) denotes a candidate setting and \(L_k(\lambda)\) is its loss on fold \(k\). The loss must match the prediction task and decision purpose. For a squared-error regression objective, a fold's mean squared error is a natural choice. The fold-level values matter as well as their average: large variation between folds warns that the estimated ranking may be unstable.
Worked Example: Selecting Boosting Iterations From Four Folds
A hypothetical model predicts quarterly repair costs. The training data are divided into four folds. For a fixed learning rate and fixed tree depth, the fold mean squared errors, in squared thousands of dollars, are as follows. Select the iteration count with the smallest mean CV error, and then apply the one-standard-error rule.
| Boosting iterations | Fold 1 | Fold 2 | Fold 3 | Fold 4 |
|---|---|---|---|---|
| 50 | 0.90 | 0.82 | 0.94 | 0.86 |
| 100 | 0.78 | 0.84 | 0.80 | 0.82 |
| 200 | 0.80 | 0.79 | 0.83 | 0.78 |
Step 1: Calculate the mean fold loss. At 50 iterations, the mean is \((0.90+0.82+0.94+0.86)/4=3.52/4=0.88\). At 100 iterations, it is \((0.78+0.84+0.80+0.82)/4=3.24/4=0.81\). At 200 iterations, it is \((0.80+0.79+0.83+0.78)/4=3.20/4=0.80\). The minimum mean is \(0.80\), at 200 iterations.
Step 2: Calculate the standard error at the minimum. The 200-iteration fold losses have mean \(0.80\) and deviations \(0,-0.01,0.03,-0.02\). The sum of squared deviations is \(0+0.0001+0.0009+0.0004=0.0014\). The sample variance across folds is \(0.0014/(4-1)=0.0004667\); its square root is approximately \(0.0216\). The standard error of the mean fold loss is \(0.0216/\sqrt{4}=0.0108\).
Step 3: Apply the rule. The one-standard-error threshold is \(0.80+0.0108=0.8108\). The 100-iteration mean, \(0.81\), is within this threshold; the 50-iteration mean, \(0.88\), is not. If fewer iterations are the simpler choice, select 100 iterations under this rule. The rule does not claim that 100 is the unique best model; it favors a simpler candidate whose estimated error is close to the minimum. The common mistake is to select 50 because its mean looks close at a glance, without checking the threshold: \(0.88\) exceeds \(0.8108\).
Reading Validation Curves and Choosing the Procedure
A validation curve plots an error estimate against a tuning parameter, or displays a table of the same comparisons. Read it as evidence about a set of candidate procedures, not as a guarantee that the lowest observed value is the true lowest generalization error. A curve that falls and then levels off suggests that additional complexity may provide little practical gain. A curve that rises at larger values may indicate that added complexity is harming validation performance, although fold variation and sample size also affect its shape.
Interpret the tuning parameter according to the method. In a random forest, increasing the number of trees commonly reduces Monte Carlo variability in the forest prediction; it is not the same kind of complexity change as allowing deeper trees or smaller leaves. Once OOB error has stabilized, adding many trees may increase computation without materially improving prediction. By contrast, for boosting, increasing the number of iterations can materially change the fitted function, so validation error may eventually worsen. A low validation error at one isolated setting should be treated cautiously if neighboring settings perform quite differently.
For a simple comparison among a modest set of settings, it is reasonable to select using CV or OOB error and then evaluate the resulting choice once on the test set. If the search is extensive, the selected CV minimum may be optimistic because many candidates were compared. A later analysis of nested cross-validation addresses that more demanding setting; the essential safeguard here is not to treat the test set as a second validation set.
Separate training, validation-by-resampling, and test roles before comparing candidate settings. Keep all preprocessing and feature decisions inside the training portion of each fold.
Use OOB predictions for a random forest when suitable, or cross-validation for either forest or boosting settings. Record the fold losses or OOB errors, not only the winning value.
Use the preselected comparison rule, such as minimum CV loss or the one-standard-error rule. Refit the chosen procedure using the full training data.
Calculate the prespecified final performance measure. Do not adjust settings after seeing this result; doing so makes the test set part of model selection.
Keep the Test Set Untouched
The test set answers a different question from OOB error or cross-validation: how well does the final selected procedure perform on data that played no part in fitting or selection? Once test outcomes have influenced a choice—for example, which forest setting to report—the test set is no longer untouched. Its error can still describe those observed test cases, but it no longer serves as an unbiased final evaluation of the selection process.
The splitting strategy must also respect the data structure. Random folds can be inappropriate when multiple records belong to one policyholder, provider, or other group, because records from the same group may appear in both fitting and validation folds. Grouped folds should keep such units together. For time-ordered observations, ordinary random folds can allow future information to help predict the past; blocked or rolling-origin validation is often more appropriate. The aim is always to make validation resemble the intended prediction task without leaking information.
Worked Example: Reporting a Final Test-Set Estimate
A hypothetical severity model is selected using cross-validation on the training data. The final model is then fit on all training observations and used once to predict four untouched test outcomes, measured in thousands of dollars. The actual outcomes are \(20,30,40,50\), and the predictions are \(22,27,43,48\). Calculate the test MSE and RMSE.
Method. Calculate the four test residuals, square them, average the squared residuals for MSE, and take the square root for RMSE.
The residuals, actual minus predicted, are \(20-22=-2\), \(30-27=3\), \(40-43=-3\), and \(50-48=2\). Their squares are \(4,9,9,4\). Therefore, \[ \operatorname{Test\ MSE}=(4+9+9+4)/4=26/4=6.5. \] The test RMSE is \(\sqrt{6.5}\approx 2.55\) thousand dollars, rounded to two decimal places. This is the final test estimate for the selected procedure; it is not a reason to return to CV and change the model. The common mistake is to compare many candidate settings on this same test set, select the smallest test error, and still report that minimum as an untouched estimate. The correction is to make the selection with OOB or CV results and reserve the test set for one final evaluation.
Common Traps
- Calling training error OOB error. OOB predictions must exclude every tree whose bootstrap sample included the observation being predicted.
- Assuming more trees always means more overfitting. Forest size often primarily stabilizes predictions; other settings such as leaf size or depth change the individual trees' flexibility more directly.
- Ignoring variability in the validation curve. Small differences in mean loss may be unconvincing when fold losses vary substantially. Report the comparison method and consider a simpler competitive setting.
- Tuning on the test set. Even a single model choice based on test outcomes uses the test data for selection. Make that choice using training-only OOB or CV evidence.
- Using random folds despite dependence or groups. Fold design must prevent related observations or future information from crossing into validation in a way that would not occur at prediction time.
Check Your Understanding
Use the stated selection rule in each problem; answers are given for checking.
- A forest has OOB residuals \(1,-2,1\) for three training observations. Calculate its OOB MSE. (Answer: \( (1+4+1)/3=2 \))
- A random forest uses bootstrap samples of size \(n=10\). Calculate the probability that a specified observation is OOB for one tree, and the expected number of OOB trees among 200 trees. (Answer: \(0.9^{10}\approx0.3487\); about \(200(0.3487)=69.74\), or approximately 70 trees.)
- Two boosting settings have four-fold MSE values \(1.2, 1.0, 1.1, 0.9\) and \(0.95, 1.05, 1.00, 1.00\), respectively. Calculate each mean and identify the setting with the lower mean. (Answer: \(1.05\) and \(1.00\), respectively; the second setting has the lower mean.)
- CV selects a forest setting with mean loss \(4.8\). The team then compares three settings on the test set and reports the smallest test MSE, \(4.2\). Has the test set remained an untouched final evaluation? State the correction. (Answer: No; test outcomes were used to select among settings. Select using training-only OOB or CV evidence, then evaluate the chosen setting once on the test set.)
- For a boosting candidate with four fold losses \(0.70,0.80,0.90,1.00\), calculate the mean loss and the standard error across folds, using the sample standard deviation divided by \(\sqrt{4}\). (Answer: Mean \(=0.85\); sample variance \(=0.0167\), sample standard deviation \(\approx0.1291\), and standard error \(\approx0.0645\).)