Two Ways to Combine Classification Trees
A bagged classifier uses multiple classification trees to predict a class for the same observation. Each tree can contribute a predicted class, a set of estimated class probabilities, or both. These contributions lead to two distinct ensemble predictions: one based on votes and one based on averaged class probabilities. They need not agree.
Suppose there are \(T\) trees and \(K\) possible classes. For a particular observation \(x\), tree \(b\) predicts class \(g_b(x)\) and estimates the probability \(p_{bk}(x)\) for class \(k\). Assume throughout that all trees receive equal weight and that each tree’s probability estimates are nonnegative and sum to one across the classes.
Here \(I\{\cdot\}\) is an indicator: it equals one when the condition inside the braces holds and zero otherwise. In a binary problem, a strict majority requires more than \(T/2\) votes. In a multiclass problem, the class with the most votes is often called the majority-vote winner, but it may have fewer than half of all votes. In that case, plurality winner is the precise description.
Why Averaging Produces a Class-Probability Estimate
Averaging the tree estimates creates a valid probability vector, provided each tree supplies a valid probability vector for the same set of classes. This is a property of the arithmetic, not a claim that the estimates are perfectly calibrated or that they equal the true conditional class probabilities.
Each \(p_{bk}(x)\) is nonnegative, so their sum divided by \(T\) is also nonnegative: \(\bar p_k(x)\geq 0\).
Interchange the finite sums over trees and classes to find the total averaged probability.
Each tree’s estimates sum to one, so the total across classes after averaging is also one.
The vote-based and probability-based methods agree in a special case: if each tree reports a one-hot probability vector, assigning probability one to its predicted class and zero to every other class, then the average probability for class \(k\) is \(N_k(x)/T\). Real tree probability estimates need not be one-hot. A tree can predict one class while assigning substantial probability to another, so vote counts and averaged probabilities can rank classes differently.
Worked Calculations
Worked Example: A Plurality Winner and Its Averaged Probability
A hypothetical insurer uses five equally weighted classification trees to assign an application to one of three review classes: Standard (S), Further Review (F), or Specialist Review (R). For a particular application, the trees report the following probabilities. Each tree predicts the class with its largest probability.
| Tree | S | F | R | Tree prediction |
|---|---|---|---|---|
| 1 | 0.60 | 0.25 | 0.15 | S |
| 2 | 0.45 | 0.40 | 0.15 | S |
| 3 | 0.35 | 0.50 | 0.15 | F |
| 4 | 0.30 | 0.25 | 0.45 | R |
| 5 | 0.55 | 0.30 | 0.15 | S |
Method: count votes, then average each probability column. Standard receives votes from Trees 1, 2, and 5, so its count is 3. Further Review receives one vote, and Specialist Review receives one vote. The vote proportions are therefore \(3/5=0.60\), \(1/5=0.20\), and \(1/5=0.20\). Standard is the plurality winner and, here, also has a strict majority.
For the averaged probability of Standard, add its five estimates and divide by five: \(0.60+0.45+0.35+0.30+0.55=2.25\), so \(\bar p_S=2.25/5=0.45\).
For Further Review: \(0.25+0.40+0.50+0.25+0.30=1.70\), so \(\bar p_F=1.70/5=0.34\).
For Specialist Review: \(0.15+0.15+0.15+0.45+0.15=1.05\), so \(\bar p_R=1.05/5=0.21\).
Check: \(0.45+0.34+0.21=1.00\). Standard also has the largest averaged probability, but that probability is 0.45, not the vote proportion 0.60. The common mistake is to report the fraction of trees voting Standard as the averaged probability; these are different summaries.
Worked Example: A Vote Tie With Unequal Averaged Probabilities
Four equally weighted trees classify a hypothetical equipment claim as Routine (R), Investigate (I), or Escalate (E). Their probability estimates and predictions are:
| Tree | R | I | E | Tree prediction |
|---|---|---|---|---|
| 1 | 0.55 | 0.30 | 0.15 | R |
| 2 | 0.60 | 0.25 | 0.15 | R |
| 3 | 0.20 | 0.65 | 0.15 | I |
| 4 | 0.25 | 0.60 | 0.15 | I |
Method: handle the hard-vote tie under the stated rule, then calculate soft averages separately. Routine receives two votes and Investigate receives two; Escalate receives none. No class has a strict majority, and Routine and Investigate tie for the largest vote count. Suppose the system resolves vote ties using the fixed class order Routine, Investigate, Escalate. Under this rule, the vote-based prediction is Routine.
The average for Routine is \((0.55+0.60+0.20+0.25)/4=1.60/4=0.40\). For Investigate it is \((0.30+0.25+0.65+0.60)/4=1.80/4=0.45\). For Escalate it is \((0.15+0.15+0.15+0.15)/4=0.60/4=0.15\).
Check: \(0.40+0.45+0.15=1.00\). The probability-based prediction is Investigate, even though the stated vote tie rule selects Routine. A tie-breaking convention chooses among tied vote counts; it does not change any tree’s probability estimate or the averaged probability ranking. Without a stated rule, the tied vote prediction should be reported as a tie rather than silently resolved.
Worked Example: The Vote Winner Has a Lower Averaged Probability
Five equally weighted trees classify a hypothetical pension inquiry as Routine (R), Review (V), or Escalate (E). Trees 1–3 predict Routine and each reports probabilities \((0.51,0.48,0.01)\). Trees 4–5 predict Review and each reports probabilities \((0.01,0.98,0.01)\). All entries are ordered R, V, E.
Method: first determine the hard-vote winner, then pool the probability estimates. Routine receives three votes and Review receives two. Thus Routine wins the vote by a strict majority: \(3/5=0.60\) of the trees predict it.
Across all five trees, the Routine probabilities sum to \(3(0.51)+2(0.01)=1.53+0.02=1.55\), so \(\bar p_R=1.55/5=0.31\). The Review probabilities sum to \(3(0.48)+2(0.98)=1.44+1.96=3.40\), so \(\bar p_V=3.40/5=0.68\). The Escalate probabilities sum to \(5(0.01)=0.05\), so \(\bar p_E=0.05/5=0.01\).
Check: \(0.31+0.68+0.01=1.00\). The vote winner is Routine, but the largest averaged probability belongs to Review. The vote counts only each tree’s top-ranked class; averaging also uses how strongly each tree favors each class. Here, the two Review-predicting trees assign 0.98 to Review, outweighing the small per-tree preference for Routine among the other three. The common mistake is to assume that the class winning the votes must also have the largest averaged probability.
Interpreting the Two Outputs
A vote proportion, such as \(N_k(x)/T\), answers: What fraction of the trees predicted class \(k\)? An averaged probability, \(\bar p_k(x)\), answers: What is the mean of the trees’ estimated probabilities for class \(k\)? The two quantities have different meanings, even when they lead to the same predicted class.
Neither number should automatically be interpreted as the true probability that the observation belongs to a class. The averaged probability is an ensemble estimate constructed from the trees. Whether it is well calibrated is a separate empirical question. Likewise, the vote proportion describes agreement among these trees; it is not, by definition, a calibrated class probability.
When reporting a classification result, state which aggregation rule is being used. If the requested rule is hard voting, report the vote counts and identify any tie or plurality. If it is probability averaging, calculate the mean for every class and select the largest mean. If both are requested, calculate both rather than assuming they agree. For multiclass votes, also distinguish a plurality from a strict majority.
Check Your Understanding
For each problem, distinguish votes from averaged probabilities and apply any stated tie rule.
- Three equally weighted trees estimate probabilities for classes A and B as follows: Tree 1 reports \((0.70,0.30)\), Tree 2 reports \((0.40,0.60)\), and Tree 3 reports \((0.60,0.40)\). Each predicts its largest-probability class. Find the vote winner and both averaged probabilities. (Answer: A receives 2 of 3 votes; averaged probabilities are A = 0.5667 and B = 0.4333, so both methods select A.)
- Four trees estimate probabilities for classes A, B, and C: \((0.50,0.30,0.20)\), \((0.20,0.60,0.20)\), \((0.10,0.70,0.20)\), and \((0.20,0.30,0.50)\). Each predicts its largest-probability class. Find the vote winner and averaged probabilities. (Answer: B wins with 2 votes; the averaged probabilities are A = 0.2500, B = 0.4750, C = 0.2750, so the probability-based prediction is also B.)
- Four trees produce probability rows, ordered A, B, C: \((0.60,0.25,0.15)\), \((0.20,0.65,0.15)\), \((0.25,0.60,0.15)\), and \((0.10,0.20,0.70)\). The vote tie rule selects B when A and B tie. Find the vote result and the probability-based prediction. (Answer: A, B, and C each receive one vote except that A and B? No; the predictions are A, B, B, C, so B receives 2 votes and is the unique vote winner; averages are A = 0.2875, B = 0.4250, C = 0.2875, so B wins both.)
- Five trees have predictions A, A, A, B, B. The three A-predicting trees each report \((0.52,0.47,0.01)\), and the two B-predicting trees each report \((0.05,0.90,0.05)\). Find the vote winner and averaged probability for each class. (Answer: A wins the vote 3–2; averaged probabilities are A = 0.3320, B = 0.6420, C = 0.0260, so B wins by averaged probability.)
- In a six-tree, four-class problem, the vote counts are A = 3, B = 2, C = 1, and D = 0. Is there a strict majority, and what is the plurality winner? Can the averaged probability winner be determined from these counts alone? (Answer: There is no strict majority because 3 is not greater than 6/2; A is the plurality winner; the averaged probability winner cannot be determined from vote counts alone.)