Why Classification Forests Have Two Summaries
A random forest classifier combines predictions from many classification trees. For a given predictor profile, each tree can contribute a class label, a set of class probabilities, or both. These outputs lead to two related but distinct summaries: class votes and averaged class probabilities. They need not identify the same class, so a candidate must determine which summary a question asks for before making a classification.
Suppose a forest contains \(B\) trees and the possible response classes are \(1,\ldots,K\). For a predictor profile \(x\), tree \(b\) predicts a class label \(c_b(x)\). It may also provide a probability \(p_{bk}(x)\) for class \(k\). For example, a tree’s terminal node might contain observations from several classes, so the tree can assign a probability to each class rather than a probability of exactly zero or one.
Here, \(I\{c_b(x)=k\}\) equals 1 if tree \(b\) predicts class \(k\), and 0 otherwise. Each vote fraction is between 0 and 1, and the vote fractions across classes sum to 1. If each tree’s probabilities sum to 1 across the \(K\) classes, then the averaged probabilities also sum to 1.
How the Two Summaries Are Formed
The vote calculation first reduces each tree’s output to a single label, then counts those labels. Averaged probabilities retain more information: a tree assigning probabilities \(0.51\) and \(0.49\) to two classes contributes the same one vote as a tree assigning \(0.99\) and \(0.01\), but the probability averages preserve the difference in strength.
For a binary classification problem, a common rule is to predict the positive class when its vote fraction or its averaged probability exceeds \(0.5\), with an explicit tie rule needed at exactly \(0.5\). The threshold \(0.5\) is not mandatory for every application. It corresponds to a particular classification rule; a problem may specify a different threshold or a cost-sensitive rule. For multiple classes, the usual rule is to choose the class with the largest summary, again specifying how ties are handled.
A tree-level probability is often based on the class proportions among the training observations in the terminal node reached by \(x\). Implementations can define or report forest probabilities differently, so calculations must follow the tree outputs or probability definition given in the problem. When per-tree probabilities are supplied, the forest average is an equal-weight average across trees unless tree weights are explicitly stated.
Worked Example: A Majority Vote That Disagrees With the Probability Average
A hypothetical insurer uses a five-tree forest to classify an application as either “review” (R) or “standard” (S). For one application, the trees’ probabilities of R are \(0.51,\ 0.51,\ 0.51,\ 0.01,\ 0.01\). Each tree predicts the class with the larger probability. Find the vote fractions, the averaged probabilities, and the predictions under each summary. Use a tie rule only if a tie occurs.
Method. First convert each tree’s probabilities to a class label; count those labels for the vote fractions. Then average the five probabilities for R. Since each tree’s two probabilities sum to 1, its probability of S is 1 minus its probability of R.
The first three trees predict R because \(0.51>0.49\). The last two predict S because \(0.01<0.99\). Thus the vote counts are 3 for R and 2 for S, giving
The mean probability of R is
Therefore, \(\bar p_S=1-0.31=0.69\). The majority vote predicts R, whereas the larger averaged probability predicts S. The calculations check because both pairs of summaries sum to 1: \(0.60+0.40=1\) and \(0.31+0.69=1\).
Common mistake avoided: Do not infer that a class with the most tree labels must also have the largest averaged probability. Three trees give R only a small margin, while the two trees favoring S do so strongly.
Why Averaging Preserves a Valid Probability Summary
The averaged probabilities are computed class by class, but they remain a probability distribution across the classes. The justification follows directly from the fact that each tree’s class probabilities sum to 1.
For each tree \(b\), its probabilities satisfy \(\sum_{k=1}^{K}p_{bk}(x)=1\), with each \(p_{bk}(x)\) between 0 and 1.
Substitute the definition of \(\bar p_k(x)\) and interchange the finite sums: \(\sum_{k=1}^{K}\bar p_k(x)=\sum_{k=1}^{K}\frac{1}{B}\sum_{b=1}^{B}p_{bk}(x)=\frac{1}{B}\sum_{b=1}^{B}\sum_{k=1}^{K}p_{bk}(x)\).
Each inner sum equals 1, so the result is \(\frac{1}{B}\sum_{b=1}^{B}1=1\). Also, an average of numbers between 0 and 1 remains between 0 and 1.
This proves that \((\bar p_1(x),\ldots,\bar p_K(x))\) is a valid set of class probabilities in the numerical sense: its entries are nonnegative and sum to 1. It does not, by itself, prove that the probabilities are calibrated to observed long-run frequencies. Probability validity and calibration are different claims.
The connection between votes and probabilities can also be stated precisely. Replace each tree’s probability vector by a one-hot vector: it has value 1 for the tree’s predicted class and 0 for every other class. Averaging these vectors produces exactly the vote fractions. Thus votes are averages of hard, one-class outputs, while averaged probabilities use each tree’s full probability vector.
Worked Example: Three-Class Votes and Probability Averages
A hypothetical claims triage forest has four trees and three possible labels: “routine” (R), “investigate” (I), and “urgent” (U). For a claim, the tree probability vectors in the order R, I, U are \((0.51,0.48,0.01)\), \((0.51,0.48,0.01)\), \((0.51,0.48,0.01)\), and \((0.01,0.98,0.01)\). Find the vote fractions and averaged probabilities, then identify the class selected by each summary. Each tree predicts its largest-probability class.
Method. In each of the first three trees, R has the largest probability, so those trees vote R. In the fourth tree, I has the largest probability, so it votes I. No tree votes U. The vote fractions are therefore
Now average each probability column across the four trees:
The majority vote selects R, while the largest averaged probability selects I. The averaged probabilities check: \(0.385+0.605+0.010=1.000\). The vote fractions also sum to 1. The disagreement occurs because the fourth tree gives I a very high probability, even though only one tree selects I as its label.
Common mistake avoided: Do not average only the probabilities for the class that wins the vote. Each class needs its own column average before the probability-based class is selected.
Terminal-Node Counts: Average Trees, Not Observations Pooled Across Trees
If a tree assigns a class probability using class proportions in its terminal node, the node’s counts can be used to obtain that tree’s probability. For example, if 8 of 10 observations in a node belong to class A, its estimated probability for A is \(8/10=0.8\). When a forest combines these tree-level probabilities, each tree contributes one probability value to the average. A tree with a larger terminal node does not automatically receive more weight.
Worked Example: Unequal Terminal-Node Sizes
A hypothetical forest has three trees. For a policy profile, the reached terminal nodes contain the following counts of classes A and B: tree 1 has 8 A and 2 B; tree 2 has 2 A and 8 B; tree 3 has 1 A and 0 B. Estimate each tree’s probability of A from its terminal-node proportions, then calculate the forest’s averaged probability of A and the corresponding class prediction. Use the largest probability as the decision rule.
Method. Calculate the A proportion separately in each node, then take the arithmetic mean of the three proportions. Tree 1 gives \(p_{1A}=8/(8+2)=0.8\). Tree 2 gives \(p_{2A}=2/(2+8)=0.2\). Tree 3 gives \(p_{3A}=1/(1+0)=1\).
The averaged probability of A is
to four decimal places. Since the averaged probability of B is \(1-0.6667=0.3333\), the probability-based prediction is A. The tree labels are A, B, and A, so the vote fractions are \(V_A=2/3\) and \(V_B=1/3\), also selecting A.
Pooling the counts would instead give \(11\) A observations out of \(21\), or \(11/21=0.5238\) for A. That calculation weights the trees according to their terminal-node sizes, which is not the stated equal-weight forest average. The correct forest average is \(0.6667\), as verified by directly averaging the three tree probabilities.
Common mistake avoided: Do not combine all terminal-node observations into one large pool unless the problem explicitly defines that weighting scheme. First compute each tree’s probability, then average across trees.
Interpreting and Reporting a Forest Classification
A clear answer states which summary was used and reports enough intermediate information to make the decision auditable. For a binary problem, a response might report both the fraction of trees voting for the positive class and the mean positive-class probability, followed by the applicable threshold rule. For a multiclass problem, report the class-specific values when the distinction matters.
The two summaries answer different questions. The vote fraction describes how often a tree’s hard class prediction is a particular class. The averaged probability describes the mean of the tree-level probability estimates for that class. Neither should be silently substituted for the other. A stated question may request only the class label, but if it specifies “majority vote,” use labels; if it specifies “average the tree probabilities,” average probabilities.
- Check the output type. A list of labels supports vote counts; probability vectors support probability averages.
- Check the decision rule. Use the specified threshold, tie rule, or largest-class rule rather than assuming one.
- Check the averaging unit. For equal-weight trees, divide by the number of trees, not by the total number of observations across terminal nodes.
- Check arithmetic. Vote fractions should sum to 1, and averaged class probabilities should sum to 1 when each tree’s vector does.
These calculations are aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus. In a multi-step classification problem, the main exam trap is often not difficult arithmetic but applying the right summary to the information supplied.
Check Your Understanding
For each problem, distinguish the requested summary from the other available forest output.
- A three-tree binary forest gives positive-class probabilities \(0.62,\ 0.48,\ 0.46\). Each tree predicts its larger-probability class. Find the positive-class vote fraction and averaged probability, and state the prediction under each rule. (Answer: Vote fraction \(1/3\); averaged probability \(0.5200\); vote rule predicts negative, probability rule with threshold \(0.5\) predicts positive.)
- A four-tree, three-class forest has hard predictions A, B, B, C. Find the vote fractions. (Answer: \(V_A=0.25,\ V_B=0.50,\ V_C=0.25\); majority vote predicts B.)
- Three trees report class-A probabilities \(0.30,\ 0.60,\ 0.90\). Find the averaged probability of A and the corresponding probability of B in a binary problem. State the probability-based prediction using threshold \(0.5\). (Answer: \(\bar p_A=0.6000,\ \bar p_B=0.4000\); predict A.)
- Two tree terminal nodes contain, respectively, 9 A and 1 B, and 1 A and 1 B. Using equal tree weights, find the forest average probability of A. Compare it with the probability obtained by pooling all node counts. (Answer: Equal-tree average \(=(0.90+0.50)/2=0.7000\); pooled probability \(=10/12=0.8333\).)
- Three trees give probability vectors for classes A and B of \((0.55,0.45)\), \((0.20,0.80)\), and \((0.51,0.49)\). Find both vote fractions and averaged probabilities, and identify the selected class under each summary. (Answer: Votes A \(2/3\), B \(1/3\), so vote rule selects A; averaged probabilities are A \(0.4200\), B \(0.5800\), so probability rule selects B.)