Why Distinguish Boosting From Random Forests?
Boosted trees and random forests both combine many decision trees, but they build and combine those trees in different ways. A random forest typically fits trees independently, then averages their predictions or takes a vote. Boosting fits learners sequentially: each new learner is trained to improve the current ensemble according to a chosen loss function. Those differences affect what is sampled, which tuning parameters matter, and how predictions should be interpreted.
In an actuarial prediction task, either method might be considered for outcomes such as claim cost, lapse, or whether a claim exceeds a threshold. The appropriate comparison depends on the prediction objective, the validation design, and the cost of prediction errors. Neither method is automatically superior.
How the Learners Are Constructed
A standard random forest uses two sources of randomness. Each tree is fitted to a bootstrap sample: a sample of the training rows drawn with replacement, usually with the same number of draws as there are original rows. At a candidate split, the algorithm considers a random subset of the available predictors rather than all predictors. A different sample and different candidate predictors can produce trees with different structures.
Random forests are usually designed to reduce the variation of an individual tree by averaging many trees. The trees need not be identical, but their predictions should contain useful signal. If every tree makes nearly the same errors, averaging cannot remove much of that shared error. A common implementation grows trees relatively deeply and controls their similarity through the sampling and splitting rules, though exact settings depend on the problem and software.
Boosting instead maintains a sequence of ensemble predictions. At each stage, a new tree is fitted to information about the current errors. For squared-error regression, this information is commonly the residuals. More generally, gradient boosting fits a learner to the negative gradient of the loss with respect to the current predictions. The new tree is then added with a learning-rate multiplier. With shrinkage, each tree makes a relatively small adjustment, and many stages may be needed.
Some boosting methods use all training observations at each stage; stochastic gradient boosting uses a random subset of rows for each stage. This row subsampling can add randomness and help control overfitting. It is not the same operation as a random forest's feature subsampling at each split. Boosting can also use feature subsampling, but that is a separate design choice.
| Feature | Random forest | Boosted trees |
|---|---|---|
| How trees are fitted | Usually independently, conditional on the training data and random draws | Sequentially; later learners respond to the current ensemble's errors or loss gradient |
| Common row sampling | Bootstrap sample for each tree | All rows, or a subsample at each boosting stage |
| Feature sampling | Random subset considered at each split is a defining common feature | May be used, but is a separate optional choice |
| Prediction combination | Average for regression; vote or probability average for classification | Add learner contributions to form a score or prediction |
| Common tuning emphasis | Number of predictors tried per split, tree size, and number of trees | Learning rate, number of stages, tree complexity, and row-subsampling fraction |
Prediction Aggregation and a Variance Justification
Suppose a regression forest contains \(B\) trees, and let \(T_b(x)\) be tree \(b\)'s prediction at predictor value \(x\). Its forest prediction is commonly
A regression boosting model, by contrast, typically starts with an initial prediction and adds shrunken tree contributions:
where \(h_m(x)\) is the contribution from stage \(m\), \(M\) is the number of stages, and \(0<\eta\leq 1\) is the learning rate. This expression describes a common additive form; the exact learner-fitting step depends on the loss and boosting algorithm. In classification, the sum may be a score on a link or margin scale, not a probability. A probability requires a suitable link transformation or calibration method.
The variance reduction from averaging can be seen under a specific simplifying assumption: at a fixed \(x\), suppose the tree prediction errors have common variance \(\sigma^2\), and every distinct pair has correlation \(\rho\). Then
The first equality follows by expanding the variance of a sum: there are \(B\) variance terms, each \(\sigma^2\), and \(B(B-1)\) ordered covariance terms, each \(\rho\sigma^2\). Dividing by \(B^2\) gives the variance of the average. If the trees are uncorrelated, \(\rho=0\), the variance is \(\sigma^2/B\). If \(B\) grows, the expression approaches \(\rho\sigma^2\), so a large forest cannot eliminate the part of the error caused by correlation. This result assumes equal variances and equal pairwise correlations; it is an explanatory model, not a guarantee that a real forest has those exact properties.
Worked Example: Comparing Regression Aggregation
A hypothetical insurer predicts the annual claim cost for a policy using two separately fitted ensembles. At a particular rating profile, three random-forest trees predict \$48, \$54, and \$60. A squared-error boosting model begins with \(F_0(x)=\$50\). At three successive stages, the fitted trees predict residual adjustments of \$10, \$6, and \$3 at this profile. The learning rate is \(\eta=0.20\). Find both predictions and identify the aggregation difference.
Random forest. Average the three tree predictions:
The forest prediction is \$54. Each tree contributes equally to this average.
Boosting. Apply each shrunken adjustment in sequence. The first stage contributes \(0.20(10)=2\), so \(F_1(x)=50+2=52\). The second contributes \(0.20(6)=1.2\), so \(F_2(x)=52+1.2=53.2\). The third contributes \(0.20(3)=0.6\), so \(F_3(x)=53.2+0.6=53.8\). Equivalently,
The boosted prediction is \$53.80. The forest averages tree predictions; boosting adds stagewise adjustments to its starting prediction. These are predictions from two fitted procedures, not a claim that the supplied trees came from the same training process. The common mistake is to average the three boosting adjustments as if they were independent forest predictions. That would discard the initial value and the sequential additive construction.
Classification: Voting Versus Weighted Scores
For classification, a random forest often assigns the class receiving the most tree votes. Some implementations instead average estimated class probabilities and classify using a chosen threshold. Boosting methods can combine learners differently. For example, AdaBoost for a two-class problem encodes the classes as \(-1\) and \(+1\), and combines weak-learner predictions using weights related to their training errors. Other boosting algorithms construct an additive score by optimizing a different classification loss. Thus, “boosting combines weighted votes” is true for a particular setup such as AdaBoost, not for every boosting method.
Worked Example: Forest Vote and AdaBoost Score
A hypothetical claims team classifies policies as high risk \((+1)\) or standard risk \((-1)\). At one policy profile, five random-forest trees vote \(+1,-1,+1,+1,-1\). Separately, two AdaBoost learners have weighted classification errors \(0.20\) and \(0.30\), and predict \(+1\) and \(-1\), respectively, for this profile. Calculate the forest vote proportion for high risk and the AdaBoost score and class.
Forest vote. Three of the five trees vote high risk:
Thus, the majority-vote classification is high risk. The value \(0.60\) is a vote proportion; it is not automatically a calibrated probability of high risk.
AdaBoost learner weights. For a binary AdaBoost step with weighted error \(e_m<0.5\), the learner weight is \(\alpha_m=\tfrac12\log((1-e_m)/e_m)\). Therefore,
The score for this profile is
The score is positive, so the AdaBoost classification is high risk. The learner with lower error receives more weight. The result is a score and class, not a probability; converting it to a probability depends on the model formulation and any calibration procedure. Do not confuse the forest's unweighted vote proportion with AdaBoost's error-weighted score.
Tuning and a Fair Applied Comparison
The methods have different tuning parameters, so a fair comparison is based on predictive performance under the same validation design, outcome definition, and loss metric—not on matching parameter values. A random forest's number of predictors considered at a split is often called \(m_{\mathrm{try}}\). A small \(m_{\mathrm{try}}\) can make trees less alike, but may weaken individual splits; a larger value can improve individual trees while increasing similarity. Tree depth or minimum leaf size also changes the flexibility of the forest. Adding more trees often stabilizes the average, but does not by itself guarantee better predictions on new data.
For boosting, the learning rate controls the size of each update, while the number of stages controls how many updates are made. These parameters interact: a lower learning rate often requires more stages. Tree depth controls the complexity of each learner; shallow trees capture limited interactions, while deeper trees can represent more complex structure. Row subsampling is another possible regularization choice. The best combination depends on the data and prediction task, so tuning should use validation or cross-validation rather than training performance alone.
Worked Example: Selecting an Ensemble by Cross-Validated Error
A hypothetical actuary compares squared-error predictions of annual benefit amounts using three-fold cross-validation. The same folds and loss definition are used for each candidate. The fold mean squared errors, in thousands of dollars squared, are:
| Candidate | Fold 1 | Fold 2 | Fold 3 |
|---|---|---|---|
| Random forest A: \(m_{\mathrm{try}}=2\), minimum leaf size 5 | 19 | 22 | 20 |
| Random forest B: \(m_{\mathrm{try}}=4\), minimum leaf size 5 | 21 | 18 | 23 |
| Boosting A: learning rate 0.05, 300 stages, depth 2 | 17 | 20 | 19 |
| Boosting B: learning rate 0.10, 150 stages, depth 2 | 16 | 24 | 21 |
Step 1: Calculate each mean. Random forest A has \((19+22+20)/3=61/3=20.333\). Random forest B has \((21+18+23)/3=62/3=20.667\). Boosting A has \((17+20+19)/3=56/3=18.667\). Boosting B has \((16+24+21)/3=61/3=20.333\).
Step 2: Choose by the stated criterion. The smallest mean cross-validated error is \(18.667\), so boosting A is the leading candidate under this comparison. The result does not prove boosting is generally better: it identifies the candidate with the lowest observed average error among these four settings on these folds.
Step 3: Keep the comparison fair. The same outcome, folds, and squared-error metric were used. A final assessment should use data not used to select the candidate, or an appropriate nested validation design when an unbiased estimate after extensive tuning is required. Selecting by training error instead would favor flexibility and could overstate performance on new policies.
Strengths, Limitations, and Common Traps
Random forests are often useful when a stable nonlinear prediction is needed and many predictors or interactions may matter. Their averaging can reduce the variability of individual trees, and the trees can be fitted in parallel. Forests may be less sensitive than a stagewise method to some individual tuning choices, but their predictions can still be poor if the features do not contain the needed information, the response is poorly represented, or the validation scheme does not reflect the intended use.
Boosting can build a strong predictor from relatively simple learners by repeatedly improving the ensemble. It can perform well when careful tuning captures useful structure, but the sequence means later learners depend on earlier fits. Too many stages, overly complex trees, or a learning rate that is too large can lead to weak validation performance. The chosen loss also matters: a loss aligned with the target prediction is important, and a classification score should not be presented as a calibrated probability without justification.
A common exam trap is to treat all randomness as the same. In a random forest, bootstrap row sampling and random predictor selection at splits are distinct mechanisms. In stochastic boosting, a row subset is used to fit a stagewise correction; that does not make the procedure a random forest. Another trap is to use the number of trees as if it had the same role in both methods: forest trees are aggregated, while boosting stages accumulate adjustments. Finally, a lower training error is not sufficient evidence for choosing either approach; compare candidates on held-out data using the relevant metric.
Check Your Understanding
Work each problem before checking the answer shown at the end of its prompt.
- A regression forest gives predictions of 12, 15, 18, and 15 for one observation. What is its averaged prediction? (Answer: 15)
- In binary AdaBoost, a learner has weighted error \(e=0.10\). Another learner with error \(e=0.40\) predicts the opposite class for the same case. Using \(\alpha=\tfrac12\log((1-e)/e)\), find each weight to four decimals, the combined score, and the predicted class if the first learner predicts \(+1\). (Answer: weights \(1.0986\) and \(0.2027\); score \(0.8959\); class \(+1\))
- Two candidates have three-fold validation errors \(0.14, 0.18, 0.16\) and \(0.17, 0.15, 0.19\). Which has the lower mean error, and what are the means? (Answer: the first candidate; means \(0.1600\) and \(0.1700\))
- Assume equal tree-error variance \(\sigma^2=25\) and common pairwise correlation \(\rho=0.20\). Using \(\operatorname{Var}(\bar T)=\sigma^2[\rho+(1-\rho)/B]\), find the variance of the average of \(B=4\) tree predictions. (Answer: \(10\))
- A team wants a nonlinear classifier and proposes using random feature subsets at splits, bootstrap samples, and sequentially fitting each new tree to current errors. Which description is inconsistent, and why? (Answer: the construction mixes the defining ideas of both methods; bootstrap trees with random feature subsets are characteristic of a random forest, while sequential error correction is boosting. The team should specify a coherent method or explicitly describe a hybrid.)