Random Forest Architecture From Trees To Predictions

From One Tree to a Random Forest

A single tree gives a prediction by routing a case through its split rules to a terminal node. Random forests extend that idea by building many trees with deliberate variation in both the records and the predictors available during tree construction. The trees then contribute to a combined prediction. The result is an ensemble: a model whose output is assembled from multiple fitted models.

The architecture matters because two different sources of randomness are involved. In the standard random-forest approach, each tree is trained on a bootstrap sample of the training records. In addition, at each split, the tree considers a randomly selected subset of the predictors rather than searching all predictors. These steps affect how the trees are built; aggregation combines their predictions afterward.

Definition: A random forest is an ensemble of decision trees in which each tree is fitted using a randomized training process. In the standard approach, that process uses a bootstrap sample of the training records and a newly selected random subset of predictors as candidates at each split. A regression forest combines numerical tree predictions; a classification forest combines predicted classes or class scores.

For a training data set with $n$ rows and $p$ predictors, a standard bootstrap sample for one tree consists of $n$ draws from the $n$ training rows, made with replacement. Therefore, a row can appear more than once or not at all in a particular tree’s sample. At a node, the tree selects a subset of the $p$ predictors as split candidates, then chooses its split from the candidates available there. The subset is selected again at a later node; it is not one fixed subset for the entire forest.

The random selection does not mean that a split is chosen without regard to fit. The tree still selects the best split according to its splitting rule among the predictors offered at that node. A predictor that was not offered cannot be selected at that node, even if it might have produced a better split had it been considered.

How the Pieces Fit Together

The steps below describe the standard architecture conceptually. Particular software can offer implementation options, so the precise sample size or other settings should be checked for the model being fitted.

Generate a tree-specific training sample.
Draw training rows with replacement to create the bootstrap sample for that tree. Repeated rows contribute repeatedly when the tree is fitted.

Grow a tree using randomized split candidates.
At each node, select a subset of predictors, evaluate possible splits among those candidates, and apply the chosen split. Continue according to the tree-growing rules in use.

Repeat the randomized construction.
Build additional trees using new bootstrap samples and new predictor subsets. The trees may consequently differ in their training rows and split structure.

Aggregate predictions for a case.
For regression, average the trees’ numerical predictions. For classification, combine their predicted classes or class scores according to the model’s aggregation rule.

Worked Example: Tracking a Bootstrap Sample into a Terminal Node

A hypothetical training set has five records. Their observed response values are R1: 12, R2: 21, R3: 18, R4: 27, and R5: 30. One tree receives the bootstrap draw R1, R3, R3, R5, R1. Assume that, after the tree’s splits are made, the terminal node reached by a case contains the sampled copies of R1, R3, and R5, and no other sampled records. Find the node’s mean response prediction.

Method. A regression tree’s terminal-node prediction is the mean response among the training records assigned to that node. In this tree, repeated bootstrap draws count as repeated records. The relevant multiplicities are R1 twice, R3 twice, and R5 once.

The weighted response total is $2(12)+2(18)+1(30)=24+36+30=90$. The number of sampled records in the node, counting repetitions, is $2+2+1=5$. Therefore, the node prediction is $90/5=18$.

Check. The mean of the five node responses, listed with repetition, is $(12+18+18+30+12)/5=90/5=18$. The common mistake this avoids is treating the two appearances of R1 or R3 as if each were only one training contribution.

Random Predictor Selection at a Split

Randomly limiting split candidates changes the tree-building search, not the definition of a useful split. For example, a regression tree can compare candidate splits using the reduction in residual sum of squares at a node. It can choose the largest reduction among the candidates it was given, but cannot compare against predictors outside that candidate subset at that node.

Worked Example: Choosing a Split from the Available Predictors

At a node of a hypothetical regression tree, the parent node has residual sum of squares 120. The randomly selected candidate predictors are driving distance and prior-claim count. For the best available distance split, the two child nodes have residual sums of squares 38 and 42. For the best available prior-claim-count split, the child values are 55 and 35. A third predictor, repair severity, is not selected as a candidate at this node; its best possible split would have child residual sums of squares 30 and 35. Which split is selected?

Method. For each eligible split, calculate the reduction as the parent residual sum of squares minus the sum for its two children. The tree compares only the two predictors selected as candidates.

For distance, the child total is $38+42=80$, so the reduction is $120-80=40$. For prior-claim count, the child total is $55+35=90$, so the reduction is $120-90=30$. Among the available candidates, distance gives the larger reduction, 40 rather than 30, and is selected.

Repair severity would have a child total of $30+35=65$, giving a reduction of $120-65=55$. That is larger than 40, but repair severity was not offered as a candidate at this node. It therefore cannot be selected here. The common mistake is to choose the best split among all predictors instead of the best split among the randomly selected candidates.

Aggregating Predictions

For regression, suppose the fitted forest contains $B$ trees, and tree $b$ predicts $\hat f_b(x)$ for a case with predictors $x$. The standard average prediction is

\hat f_{\mathrm{forest}}(x)=\frac{1}{B}\sum_{b=1}^{B}\hat f_b(x).

This formula applies when the forest’s regression rule is the arithmetic mean of its tree predictions. Each tree contributes one prediction for the case; the denominator is the number of trees, not the number of rows in the original training data.

Worked Example: Averaging Regression-Tree Predictions

A hypothetical forest predicts the repair cost for one policyholder using four trees. The trees predict $72, $84, $90, and $114, respectively. Find the forest prediction.

Method. Use the arithmetic mean of the four tree predictions. Their total is $72+84+90+114=360$ dollars, and there are four trees, so the forest prediction is $360/4=\$90$.

Check and trap. The result, $90, lies between the smallest and largest tree predictions. Dividing the total by the number of training records would use the wrong denominator. For instance, if the training set had 40 rows, $360/40=\$9$ would be an incorrect calculation: the aggregation averages four tree outputs, not 40 training responses.

For classification, the aggregation rule depends on what each tree supplies. If every tree supplies one class label, a common rule is to assign the class with the most votes. If trees supply class scores or estimated class probabilities, a model may instead combine those scores. The output and rule should be identified before interpreting a result: a vote share is not automatically the same quantity as a probability calibrated to observed outcomes.

Worked Example: Combining Classification Votes

A hypothetical classification forest uses five trees to predict whether a claim should be flagged for review. Four trees predict “flag” and one predicts “do not flag.” Using plurality voting, determine the forest’s predicted class and report the vote shares.

Method. Count the votes for each class and divide each count by the total number of trees to obtain the vote shares. “Flag” receives 4 votes and “do not flag” receives 1, out of 5 trees.

The vote shares are $4/5=0.8$ for “flag” and $1/5=0.2$ for “do not flag.” Since 4 is greater than 1, the plurality prediction is “flag.” The shares sum to $0.8+0.2=1.0$, as expected.

The common mistake is to report 0.8 as a guaranteed 80% chance that the claim truly should be flagged. It is the share of trees voting for that class in this calculation. Interpreting a forest output as a probability requires knowing how the fitted model defines that output and, where relevant, assessing how well its probabilities correspond to outcomes.

Why Averaging Represents the Randomized Ensemble

The forest’s average can also be understood as an approximation to the average prediction over the random tree-building process. Fix the training data $D$ and a case $x$. Let $\Theta_b$ represent the random choices used to construct tree $b$, including its bootstrap sample and predictor subsets, and write that tree’s prediction as $g(x;D,\Theta_b)$. If the trees use independent randomizations drawn from the same construction process, then

E\left[\frac{1}{B}\sum_{b=1}^{B}g(x;D,\Theta_b)\,\middle|\,D\right] = \frac{1}{B}\sum_{b=1}^{B}E[g(x;D,\Theta_b)\mid D] = E[g(x;D,\Theta)\mid D].

The first equality follows from linearity of expectation. The second follows because each $\Theta_b$ has the same distribution as a generic randomization $\Theta$. Thus, the average of the fitted trees is a Monte Carlo estimate of the mean prediction under the randomized tree-building process, conditional on the fixed training data. This justification explains the role of aggregation; it does not claim that any finite forest equals that expectation exactly.

What to Keep Straight

The two randomization steps and the aggregation step have different jobs. Bootstrap sampling changes which training-record copies a tree sees. Random predictor selection changes which variables it can consider at a particular split. Aggregation combines the predictions after the trees have been constructed. Confusing these stages can lead to incorrect descriptions of how a forest arrived at its output.

A bootstrap sample is drawn with replacement; repeated records have repeated influence during tree fitting.
Predictor candidates are selected at each split, so a predictor can be unavailable at one node and available at another.
The selected split is the best under the tree’s splitting rule among the candidates offered at that node.
A regression forest commonly averages numerical predictions; a classification forest commonly combines class votes or scores.
The number of trees, not the number of training records, is the denominator when averaging one prediction from each tree.

These construction and aggregation steps are aligned with the SOA Exam PA (Predictive Analytics) syllabus. In a written explanation, identify what is randomized, what rule selects a split, and what the ensemble combines; do not describe the forest as one unusually large tree.

Check Your Understanding

For each problem, identify the relevant stage of forest construction or prediction before calculating.

A bootstrap sample for one tree contains records A, B, B, D, and E. Their response values are 10, 14, 14, 22, and 30, respectively. If all five sampled copies fall in one regression terminal node, what is its mean prediction? (Answer: 18)
A regression node has parent residual sum of squares 90. The selected candidate predictors produce best child totals of 58 and 45, respectively. Which predictor is selected, and what is its reduction? (Answer: The predictor with child total 45; reduction $90-45=45$)
Six regression trees predict 40, 50, 50, 70, 80, and 90 for a case. What is the forest prediction using the arithmetic mean? (Answer: $380/6=63.33$, rounded to two decimal places)
Seven classification trees cast five votes for “renew” and two for “nonrenew.” Under plurality voting, what class is predicted, and what are the vote shares? (Answer: “Renew”; $5/7=0.7143$ and $2/7=0.2857$, to four decimal places)

What You'll Learn