Bagging Classification Trees

Why Bag Classification Trees?

A classification tree can change noticeably when the training data change: a few different observations can lead to different splits, terminal nodes, and predicted classes. Bagging—short for bootstrap aggregating—uses multiple versions of the training data to fit multiple trees, then combines their predictions. For a hypothetical insurer classifying submitted claims as “refer” or “do not refer,” the combined prediction can be less dependent on the particular training sample than the prediction from one tree alone.

The central calculation is straightforward, but there are two distinct ways to combine classification trees. One combines their predicted classes as votes. The other combines their estimated probabilities for each class. These rules need not select the same class, so an answer should state which aggregation rule is being used.

Definition: Given \(n\) training observations, a bootstrap sample is a sample of size \(n\) drawn with replacement from those observations. A bagged classification classifier fits a classification tree to each of \(B\) bootstrap samples and combines the trees’ predictions for a new predictor value \(x\). An observation can appear more than once in a bootstrap sample, and some original observations can be absent.

From Bootstrap Samples to Tree Predictions

For each bootstrap sample, fit a classification tree using the chosen tree-growing procedure. The same predictor variables and fitting rules are ordinarily available to each tree; the bootstrap samples are what make the fitted trees differ. Bagging, by itself, does not randomly restrict the predictors considered at each split. That additional feature-subset procedure belongs to random forests.

Suppose the response has \(K\) possible classes, labelled \(1,\ldots,K\). For tree \(b\), let \(\hat p_{bk}(x)\) be the estimated probability of class \(k\) for an observation with predictors \(x\). A common convention is to use the class fraction in the terminal node reached by \(x\), counting repeated bootstrap observations with their multiplicities. The tree’s predicted class is a class with the largest estimated probability:

\hat c_b(x)=\arg\max_{k\in\{1,\ldots,K\}}\hat p_{bk}(x).

If two or more classes tie for the largest probability, the tree needs a tie rule, such as choosing the first class in a stated ordering. The same principle applies when the ensemble’s aggregated quantities tie. For hand calculations, use the tie rule specified in the problem; if none is supplied, report the tie rather than silently choosing a class.

Two Ways to Aggregate Classification Trees

Under class-vote aggregation, each tree contributes one vote: its predicted class. The vote proportion for class \(k\) is the number of trees predicting \(k\), divided by \(B\). The ensemble predicts the class with the most votes, subject to a tie rule.

V_k(x)=\frac{1}{B}\sum_{b=1}^{B} I\{\hat c_b(x)=k\}.

Here \(I\{\hat c_b(x)=k\}\) equals 1 when tree \(b\) predicts class \(k\), and 0 otherwise. Under class-probability aggregation, average the estimated probability for each class across the trees:

\bar p_k(x)=\frac{1}{B}\sum_{b=1}^{B}\hat p_{bk}(x).

The probability-aggregation classifier predicts a class with the largest \(\bar p_k(x)\). Both rules give every tree equal weight in the calculations shown here, but they summarize different information. A vote uses only each tree’s most likely class; an averaged probability uses the tree’s full estimated class-probability vector.

Key distinction: The vote proportion \(V_k(x)\) is not generally the averaged probability \(\bar p_k(x)\). In a vote, a tree’s contribution is determined by its predicted class, whether its leading probability is 0.51 or 0.99. In probability aggregation, the size of the estimated probability matters.

Worked Examples

Worked Example: Calculating a Two-Class Vote

A hypothetical insurer fits six bootstrap classification trees to label claims “refer” (R) or “do not refer” (N). For one new claim, the six trees predict, in order, R, R, N, R, N, R. Using class votes, calculate the vote proportions and the ensemble prediction.

Method. Count the predicted classes first, then divide each count by the total number of trees. This avoids treating a vote count as a probability estimate from a single tree.

There are \(B=6\) trees. Four predict R and two predict N, so:

V_{\mathrm{R}}(x)=\frac{4}{6}=0.6667,\qquad V_{\mathrm{N}}(x)=\frac{2}{6}=0.3333.

The vote proportions sum to \(0.6667+0.3333=1.0000\), as they should. R has the larger vote proportion, so the bagged classifier predicts refer. There is no tie. The common mistake this calculation avoids is using the number of trees, six, as the vote proportion rather than dividing each class’s count by six.

Worked Example: Averaging Tree Probabilities Versus Pooling Counts

Three bootstrap-fitted trees estimate the probability that a hypothetical equipment claim requires specialist review. Their terminal nodes contain, respectively, 3 review claims out of 4 observations, 2 out of 8, and 4 out of 5. Calculate the equal-weight average of the three tree probabilities. Then calculate the proportion obtained by pooling all three sets of counts, and explain why the results differ.

Method. Convert each tree’s count to a probability, then average the three probabilities with equal weight. For comparison, pooling counts weights each tree’s proportion according to its terminal-node size.

The tree probabilities are:

\hat p_{11}=\frac{3}{4}=0.7500,\qquad \hat p_{21}=\frac{2}{8}=0.2500,\qquad \hat p_{31}=\frac{4}{5}=0.8000.

Their equal-weight average is:

\bar p_1(x)=\frac{0.7500+0.2500+0.8000}{3} =\frac{1.8000}{3}=0.6000.

For the pooled proportion, add the review counts and divide by the total number of observations across the three nodes:

\frac{3+2+4}{4+8+5}=\frac{9}{17}=0.5294.

The values differ because the equal-weight average gives each tree one-third of the weight, while pooling gives the node containing 8 observations twice the weight of the node containing 4. Thus, \(0.6000\) is the equal-weight bagged probability estimate under the stated rule; \(0.5294\) is the pooled count proportion. Pooling these counts is not a substitute for averaging the tree probabilities. The check is that the pooled counts total 9 review observations out of 17, and \(9/17\) is approximately \(0.5294\), not \(0.7500\).

Worked Example: Vote Winner and Probability Winner Can Differ

Five trees classify a hypothetical policy application into one of three categories: A, B, or C. Their estimated probability vectors, in the order A, B, C, are \((0.40,0.35,0.25)\), \((0.40,0.35,0.25)\), \((0.10,0.55,0.35)\), \((0.10,0.55,0.35)\), and \((0.05,0.45,0.50)\). For a tie in class votes, use the rule “choose A before B before C.” Find the vote prediction and the probability-aggregation prediction.

Method. First find each tree’s largest component to obtain its class vote. Then average each probability component separately. Do not infer the probability-aggregation result from the vote counts.

The first two trees predict A, the third and fourth predict B, and the fifth predicts C. The vote counts are therefore A: 2, B: 2, and C: 1. Dividing by five gives:

V_{\mathrm{A}}(x)=\frac{2}{5}=0.4000,\qquad V_{\mathrm{B}}(x)=\frac{2}{5}=0.4000,\qquad V_{\mathrm{C}}(x)=\frac{1}{5}=0.2000.

A and B tie for the most votes, so the stated tie rule selects A.

Now average the A probabilities:

\bar p_{\mathrm{A}}(x)=\frac{0.40+0.40+0.10+0.10+0.05}{5} =\frac{1.05}{5}=0.2100.

For B:

\bar p_{\mathrm{B}}(x)=\frac{0.35+0.35+0.55+0.55+0.45}{5} =\frac{2.25}{5}=0.4500.

For C:

\bar p_{\mathrm{C}}(x)=\frac{0.25+0.25+0.35+0.35+0.50}{5} =\frac{1.70}{5}=0.3400.

The averaged probabilities sum to \(0.2100+0.4500+0.3400=1.0000\). B has the largest averaged probability, so probability aggregation predicts B. The two ensemble rules select different classes: the vote rule selects A by the stated tie rule, while the probability rule selects B. The common mistake is to assume that averaging the probability vectors must produce the same winner as counting each tree’s leading class.

Why Averaging Probabilities Works as a Calculation

Each tree supplies a probability vector: its \(K\) components are nonnegative and sum to 1. Taking the componentwise average preserves those properties. In particular, for each class \(k\), \(\bar p_k(x)\geq 0\), and:

\sum_{k=1}^{K}\bar p_k(x) =\sum_{k=1}^{K}\frac{1}{B}\sum_{b=1}^{B}\hat p_{bk}(x) =\frac{1}{B}\sum_{b=1}^{B}\sum_{k=1}^{K}\hat p_{bk}(x) =\frac{1}{B}\sum_{b=1}^{B}1=1.

The rearrangement is valid because both sums are finite. This shows that the averaged components form a probability vector. It does not show that they are perfectly calibrated probabilities for the population: the quality of the estimates still depends on the data, the tree-fitting procedure, and the setting in which predictions are used.

Common Mistakes and Practical Checks

Confusing a vote fraction with an averaged probability. A vote fraction counts tree-level class predictions. An averaged probability averages the numerical class probabilities supplied by the trees.
Pooling counts when equal tree weights are required. As in the second worked example, pooling gives more weight to larger terminal nodes. It does not calculate the equal-weight average across trees.
Leaving the aggregation rule unstated. Say whether the prediction uses class votes or averaged probabilities, especially when the two rules can give different answers.
Ignoring ties. Check for tied vote counts or tied averaged probabilities, then apply the specified rule or report the tie.
Forgetting to check totals. Vote proportions should sum to 1, and each averaged probability vector should also sum to 1, apart from rounding.

For calculations aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus, read the requested output carefully: a question about the most votes calls for class-vote aggregation, while a question about mean class probability calls for probability aggregation. Use the tree-level information appropriate to that rule.

Check Your Understanding

For each problem, identify the requested aggregation rule before calculating.

Four trees classify a hypothetical claim as “manual review” or “automatic processing.” Their predictions are manual, automatic, manual, manual. What are the vote proportions and the vote prediction? (Answer: Manual review \(3/4=0.7500\); automatic processing \(1/4=0.2500\); predict manual review.)
Three trees estimate the probability of a hypothetical policy being referred as \(0.80\), \(0.45\), and \(0.70\). What is the equal-weight averaged probability of referral? (Answer: \((0.80+0.45+0.70)/3=1.95/3=0.6500\).)
Three binary-class trees estimate the probability of “late payment” as \(0.90\), \(0.40\), and \(0.40\). Each tree predicts the class with probability above \(0.50\), if any. Find the majority-vote prediction and the probability-aggregation prediction. (Answer: Votes are one late payment and two not late, so majority vote predicts not late; the mean late-payment probability is \(1.70/3=0.5667\), so probability aggregation predicts late payment.)
Four trees give probabilities for classes A and B of \((0.60,0.40)\), \((0.55,0.45)\), \((0.20,0.80)\), and \((0.20,0.80)\). Calculate both vote proportions and averaged class probabilities. (Answer: Votes are A: \(2/4=0.5000\), B: \(2/4=0.5000\); averaged probabilities are A: \(1.55/4=0.3875\), B: \(2.45/4=0.6125\), so probability aggregation predicts B.)
Three terminal nodes contain, respectively, 3, 2, and 4 positive cases out of 4, 8, and 5 observations. Calculate the equal-weight average of the three positive-class proportions and the pooled positive proportion. (Answer: Equal-weight average \(=(0.7500+0.2500+0.8000)/3=0.6000\); pooled proportion \(=(3+2+4)/(4+8+5)=9/17=0.5294\).)

What You'll Learn