Explain Why Bagging And Random Forests Stabilize Trees

Why Combine Trees?

A classification tree makes a prediction by dividing predictor space into regions and assigning a class to each region. Small changes in the training data can change which split is chosen near the top of the tree. That can alter many later splits and change the predicted class for a particular case. A single tree can therefore be useful for describing a pattern yet unstable as a predictor.

Bagging and random forests address this instability by fitting many trees and combining their predictions. The central idea is not that every tree will be correct. Rather, if their errors are not too strongly aligned, their combined prediction can be more stable than the prediction of one tree.

Definition: Bagging (bootstrap aggregation) fits a model to each of many bootstrap samples of the training data, then aggregates the fitted models' predictions. A bootstrap sample of size \(n\) is formed by drawing \(n\) observations independently with replacement from the original \(n\) training observations. For classification, aggregation is commonly by majority vote or by averaging estimated class probabilities and choosing the class with the largest average.

A random forest is an ensemble of trees that uses bootstrap samples and adds random feature selection. When a split is considered, the tree searches for a good split using only a randomly selected subset of the available predictors, rather than all predictors. The subset is typically drawn anew at each split. This makes random forests different from bagging: both resample observations, but random forests also restrict the features considered at a split.

Tree implementations can differ in details, including how class probabilities are estimated and how ties are resolved. Unless a problem specifies otherwise, assume a class is chosen by the largest aggregated probability or, for hard class predictions, by majority vote. State the convention when a tie is possible.

Bootstrap Aggregation in a Classification Example

Each bootstrap sample contains repeated observations and omits some original observations. Because the training data differ from sample to sample, a fitted tree may select different splits and produce different predictions. Bagging combines those predictions rather than relying on one particular resample.

Worked Example: Stabilizing a Local Tree Prediction

A hypothetical insurer trains nine classification trees from nine bootstrap samples of the same training set. At a particular applicant, the trees predict the following classes, where \(P\) means “higher review priority” and \(N\) means “standard review”: \(N, P, P, P, N, P, P, N, P\). A separately fitted tree using the original training set predicts \(N\). Find the bagged hard-vote prediction and its vote proportion for \(P\). Explain what can and cannot be concluded.

Method. Count the votes for each class and select the class with more votes.

There are six \(P\) predictions (trees 2, 3, 4, 6, 7, and 9) and three \(N\) predictions (trees 1, 5, and 8). Thus the vote proportions are

\widehat p_P=\frac{6}{9}=0.6667,\qquad \widehat p_N=\frac{3}{9}=0.3333.

The bagged prediction is \(P\), since \(6>3\). The single tree's prediction was \(N\), so the ensemble has changed the decision in this case. The vote proportion for \(P\) is \(0.6667\), but it should not automatically be treated as a calibrated probability that the applicant truly belongs to class \(P\). It is the fraction of these nine tree predictions in that class.

Check. The vote proportions sum to \(0.6667+0.3333=1.0000\), and \(P\) has a strict majority. The common mistake is to report the single tree's \(N\) prediction as the bagged result, or to describe the \(P\) vote fraction as verified predictive accuracy. Neither follows from the information given.

Why Correlation Limits the Benefit

Averaging can reduce variation, but the amount of reduction depends on how similar the trees' errors are. The following variance result is for numeric predictions, such as a predicted probability or a regression output. It is not, by itself, a formula for the misclassification rate of hard class votes.

Result: Suppose \(T_1,\ldots,T_B\) are tree predictions with common variance \(\sigma_T^2\), and every distinct pair has correlation \(\rho\). The average prediction \(\bar T=B^{-1}\sum_{b=1}^{B}T_b\) has variance \[ \operatorname{Var}(\bar T)=\sigma_T^2\left(\rho+\frac{1-\rho}{B}\right). \] This equal-variance, equal-correlation formula requires the stated covariance structure. It is a useful explanation of ensemble stabilization, not a guarantee that a fitted forest will have exactly this variance.

To justify the result, write the variance of a sum as the sum of individual variances plus twice the sum of pairwise covariances. There are \(B\) individual variances, each \(\sigma_T^2\), and \(B(B-1)/2\) distinct pairs, each with covariance \(\rho\sigma_T^2\). Dividing the variance of the sum by \(B^2\) gives the variance of the average:

\begin{aligned} \operatorname{Var}(\bar T) &=\frac{1}{B^2}\left[B\sigma_T^2+ 2\left(\frac{B(B-1)}{2}\right)\rho\sigma_T^2\right]\\ &=\frac{\sigma_T^2}{B^2}\left[B+B(B-1)\rho\right]\\ &=\sigma_T^2\left[\frac{1}{B}+\frac{B-1}{B}\rho\right]\\ &=\sigma_T^2\left[\rho+\frac{1-\rho}{B}\right]. \end{aligned}

If \(\rho=0\), the variance is \(\sigma_T^2/B\), the familiar reduction for uncorrelated predictions. If \(\rho>0\), increasing \(B\) reduces the term involving \(1/B\), but the variance approaches \(\rho\sigma_T^2\), not zero. Trees that make similar errors leave a shared component that averaging cannot remove.

Worked Example: Comparing Correlation Levels

A hypothetical ensemble produces numeric estimates of an applicant's probability of a late payment. Suppose each tree's prediction has variance \(\sigma_T^2=0.25\), and use the equal-correlation approximation. Compare the variance of the average of \(B=25\) trees when \(\rho=0.20\) with the variance when \(\rho=0.05\). Also compare each with 25 uncorrelated trees.

Method. Substitute each value of \(B\), \(\rho\), and \(\sigma_T^2\) into the variance formula.

With \(\rho=0.20\),

\operatorname{Var}(\bar T) =0.25\left(0.20+\frac{1-0.20}{25}\right) =0.25(0.20+0.032) =0.25(0.232) =0.058.

With \(\rho=0.05\),

\operatorname{Var}(\bar T) =0.25\left(0.05+\frac{1-0.05}{25}\right) =0.25(0.05+0.038) =0.25(0.088) =0.022.

If the 25 predictions were uncorrelated, \(\rho=0\), so

\operatorname{Var}(\bar T)=0.25\left(\frac{1}{25}\right)=0.010.

Under these assumptions, reducing correlation from \(0.20\) to \(0.05\) lowers the ensemble variance from \(0.058\) to \(0.022\). It does not reduce it to the uncorrelated value \(0.010\). As a check, the limiting variance as \(B\) grows is \(0.25\rho\): this is \(0.050\) at \(\rho=0.20\) and \(0.0125\) at \(\rho=0.05\). Both computed variances exceed their respective limits, as expected for a finite ensemble.

Common mistake. Treating all 25 trees as independent would give \(0.010\) in both cases. That calculation ignores the pairwise covariance and understates variance whenever the trees are positively correlated.

How Random Forests Reduce Tree Similarity

In ordinary bagging, each tree can consider all predictors at every split. If one predictor is especially strong, many trees may choose it near the root and make similar downstream decisions. Their predictions can then be highly correlated. A random forest restricts the candidate predictors at each split to a randomly selected subset. A strong predictor is not available at every split in every tree, so other predictors sometimes determine the structure. The resulting trees may be less alike.

This is a tradeoff, not a free improvement. Restricting the candidate features can make an individual tree less accurate or increase its bias. It can also reduce correlation among trees enough to improve the stability of the aggregate. The effect depends on the data and settings; random feature selection does not guarantee that correlation falls or that prediction accuracy improves.

Worked Example: Bagging Versus Feature-Subsampled Trees

A hypothetical claim-review model has four predictors: claim amount \(A\), policy tenure \(T\), repair category \(R\), and prior claim count \(C\). For one applicant, five bagged trees that consider all four predictors have terminal-node fractions for the positive class of \(0.80, 0.20, 0.70, 0.30,\) and \(0.40\). Five random-forest trees use two randomly selected candidate predictors at each split; their terminal-node positive fractions are \(0.70, 0.60, 0.80, 0.90,\) and \(0.20\). Average the probabilities and classify using a \(0.50\) cutoff. The feature subsets at the root of the five random-forest trees are \(\{A,T\}, \{T,R\}, \{A,C\}, \{R,C\}, \{A,T\}\), respectively. What are the two ensemble decisions, and what does the comparison establish?

Method. For each ensemble, add the five terminal-node fractions, divide by five, and compare the result with \(0.50\).

For bagging, the sum is \(0.80+0.20+0.70+0.30+0.40=2.40\). The average is

\bar p_{\mathrm{bag}}=\frac{2.40}{5}=0.48.

Since \(0.48<0.50\), the bagged ensemble predicts the negative class.

For the random forest, the sum is \(0.70+0.60+0.80+0.90+0.20=3.20\). The average is

\bar p_{\mathrm{RF}}=\frac{3.20}{5}=0.64.

Since \(0.64>0.50\), the random-forest ensemble predicts the positive class. Its trees were allowed to consider different feature subsets at the root, as listed; each tree also uses randomized candidate subsets at subsequent splits under the stated setup.

The decisions differ for this applicant, but this single constructed comparison does not show that random forests are generally more accurate. Different bootstrap samples and feature subsets can change a prediction in either direction. The common mistake is to attribute the difference solely to the feature subsets: the ensembles may also differ because their fitted trees and terminal-node fractions differ. Predictive performance must be assessed on data not used to fit the models.

What Stabilization Does—and Does Not—Mean

The main benefit of bagging is reduced sensitivity to the particular training sample. If one observation's presence or absence changes a single tree's split, it may not change the aggregate decision when many other trees vote differently. Random forests add a mechanism for reducing similarity among trees when they would otherwise rely on the same predictors. These ideas are especially relevant for deep trees, whose structure can vary substantially with the training data.

Stability is not the same as correctness. An ensemble can produce a consistent prediction and still be systematically wrong, for example if the training data omit an important predictor or the relationship changes in the population where predictions are used. Nor does the variance formula establish that a forest has lower bias than one tree. It isolates how the variance of an average responds to the number of trees and their correlation, under specific assumptions.

A practical interpretation should keep three questions separate:

Aggregation: Were class votes or class probabilities combined, and what rule produced the final class?
Diversity: Did bootstrap sampling and, for a random forest, feature subsampling create meaningfully different trees?
Predictive evidence: Was performance evaluated on data kept separate from model fitting and tuning?

A larger number of trees can make the ensemble average less sensitive to the randomness of the fitting procedure, particularly when the trees are not perfectly correlated. It cannot erase shared errors. Random feature subsets aim to reduce those shared patterns, but they may also weaken individual trees. The central exam-level distinction is therefore between more trees, which reduces finite-ensemble variation, and less correlated trees, which lowers the variance floor in the equal-correlation approximation.

Key takeaway: Bagging aggregates trees fitted to bootstrap samples. Random forests also sample candidate predictors at splits to encourage tree diversity. Averaging stabilizes predictions to the extent that tree errors are not too correlated; adding trees alone cannot remove a shared error component.

Check Your Understanding

For each problem, identify the aggregation or variance calculation required before interpreting the result.

A hypothetical fraud-screening ensemble has seven tree predictions for one record: \(F, F, N, F, N, F, N\). Find the hard-vote class and the vote proportion for \(F\). (Answer: \(F\), with \(4/7=0.5714\) of the votes.)
Each tree's numeric prediction has variance \(0.16\). For \(B=16\) trees with pairwise correlation \(\rho=0.25\), calculate the variance of the average. (Answer: \(0.16[0.25+0.75/16]=0.0475\).)
For a random forest with \(B=10\), each tree prediction has variance \(0.20\), and pairwise correlation is approximated as \(0.10\). Calculate the ensemble variance and its limiting value as \(B\) increases without bound under the same correlation assumption. (Answer: variance \(=0.20[0.10+0.90/10]=0.038\); limiting value \(=0.20(0.10)=0.020\).)
A bagged classifier averages positive-class probabilities \(0.35, 0.55, 0.45,\) and \(0.75\), and uses a \(0.50\) classification cutoff. Find the average and class decision. If a random forest gives a different result, can that alone establish it has better predictive performance? (Answer: average \(=(0.35+0.55+0.45+0.75)/4=0.525\), so the decision is positive; no, performance must be evaluated on appropriate held-out data.)

What You'll Learn