Random Forest Regression Predictions

How a Regression Forest Predicts

A regression tree predicts a numerical response by assigning a case to a terminal node and using a value for that node. A random forest makes its prediction by combining the outputs of many such trees. The calculation is simple, but it is important to keep the two levels separate: first find one prediction from each tree, then average those tree predictions.

The trees can differ because they are fitted to bootstrap samples and, at each split, consider a randomly selected subset of predictors. Thus, the same new case can follow different paths and reach different terminal nodes in different trees. The forest prediction is an average of those results, not a vote and not a prediction from a single “typical” tree.

Definition: Let $x$ be a predictor vector for a case to be predicted, and let $\hat f_b(x)$ be the prediction from regression tree $b$, for $b=1,\ldots,B$. In the standard regression forest, the prediction is the equally weighted arithmetic mean $$ \hat f_{\mathrm{RF}}(x)=\frac{1}{B}\sum_{b=1}^{B}\hat f_b(x). $$ For a regression tree whose terminal-node prediction is the mean response among the training records in that node, $\hat f_b(x)$ is that node’s response mean.

The response and the prediction remain on the original response scale. If the response is annual cost in dollars, the tree predictions and their forest average are in dollars. The averaging formula is for a quantitative response; classification forests use a different prediction summary.

From a Terminal Node to a Forest Prediction

To predict for a case, apply the split rules in each tree in turn. The case follows one branch at every split until it reaches a terminal node. If the tree uses the mean of the training responses in that node, that mean is the tree’s prediction. Repeat for all $B$ trees and take the arithmetic mean of the resulting $B$ values.

Route the case through each tree.
Use the case’s predictor values to follow the split rules until the case reaches a terminal node in that tree.

Record each tree prediction.
For a mean-response regression tree, take the mean response in the reached terminal node. Repeat separately for every tree.

Average the tree predictions.
Add the $B$ predictions and divide by $B$. Give every tree equal weight in the standard forest prediction.

Worked Example: Averaging Five Tree Predictions

A hypothetical insurer fits a regression forest to predict annual claim cost per policy, measured in dollars. A new policyholder follows the tree paths to terminal nodes whose response means are $44, $40, $49, $40, and $48. Find the forest prediction and compare it with the individual tree predictions.

Method. Compute the tree-level predictions first, then take their equal-weight mean:

\hat f_{\mathrm{RF}}(x) =\frac{44+40+49+40+48}{5} =\frac{221}{5} =44.2.

The forest prediction is $44.20 per policy. The individual trees predict between $40 and $49, so the forest’s $44.20 prediction lies within their range. It is not the prediction of any one tree in this example.

Check. The five predictions sum to $221, and $221 divided by five is $44.20. The common mistake this calculation avoids is reporting the output of one convenient tree instead of the forest average.

Why the Arithmetic Mean Appears

Equal averaging is the standard regression forest rule. One useful justification is that the arithmetic mean is the value closest, in squared-distance terms, to the collection of tree predictions. This explains the aggregation rule; it does not claim that the tree predictions are independent or that their average must equal the unknown true response.

For fixed predictions $t_1,\ldots,t_B$ from the $B$ trees, consider a single summary value $a$ and the sum of its squared deviations from those predictions:

Q(a)=\sum_{b=1}^{B}(a-t_b)^2.

Differentiate with respect to $a$, set the derivative equal to zero, and check that the stationary point is a minimum:

Q'(a)=2\sum_{b=1}^{B}(a-t_b) =2Ba-2\sum_{b=1}^{B}t_b=0, \qquad a=\frac{1}{B}\sum_{b=1}^{B}t_b.

Since $Q''(a)=2B>0$, this value minimizes $Q(a)$. Substituting $t_b=\hat f_b(x)$ gives the forest’s arithmetic mean prediction. This is a justification for a summary of the tree outputs, not a guarantee that the forest has lower prediction error than every possible alternative.

Equal Weight for Trees, Not for Records

Each tree receives equal weight in the standard forest average, even if the terminal nodes reached in different trees contain different numbers of training records. The forest does not generally pool every response from all reached nodes and average those records together. Pooling would give greater weight to trees whose reached nodes contain more records, which is a different calculation.

Worked Example: Tree Averages Versus Pooled Terminal Records

A hypothetical maintenance-cost forest predicts cost in hundreds of dollars. For a new facility, the response values in the reached terminal nodes are as follows: Tree 1 has 18, 22, and 26; Tree 2 has 16, 20, 24, and 28; Tree 3 has 25 and 29; Tree 4 has 14, 18, and 22. Find the forest prediction, then calculate the pooled-record mean to see why it is not the standard forest prediction.

Step 1: Find each terminal-node mean. The reached-node predictions are:

\begin{aligned} \hat f_1(x)&=(18+22+26)/3=66/3=22,\\ \hat f_2(x)&=(16+20+24+28)/4=88/4=22,\\ \hat f_3(x)&=(25+29)/2=54/2=27,\\ \hat f_4(x)&=(14+18+22)/3=54/3=18. \end{aligned}

Step 2: Average the four tree predictions.

\hat f_{\mathrm{RF}}(x)=\frac{22+22+27+18}{4} =\frac{89}{4}=22.25.

The forest predicts 22.25 hundreds of dollars, or $2,225.

Step 3: Calculate the different pooled-record quantity. Across the four reached nodes there are $3+4+2+3=12$ listed responses, with total $66+88+54+54=262$. Their pooled mean is $262/12=21.8333\ldots$ hundreds of dollars, not 22.25 hundreds of dollars.

The pooled calculation weights the tree predictions according to the numbers of records in their nodes. The standard forest prediction instead gives each of the four trees weight $1/4$. The common mistake is to pool records across trees when the requested quantity is the forest prediction.

Out-of-Bag Predictions and Error

A tree’s bootstrap sample is drawn with replacement from the training data. A training observation not selected into a particular tree’s bootstrap sample is out of bag (OOB) for that tree. For a training observation $i$, its OOB prediction averages only the trees for which $i$ was OOB. It must not include trees whose bootstrap samples contained that observation.

Definition: Let $\mathcal{B}_i$ be the set of trees for which training observation $i$ was out of bag, and let $B_i$ be the number of those trees. When $B_i>0$, the OOB prediction is $$ \hat f_{\mathrm{OOB},i}=\frac{1}{B_i}\sum_{b\in\mathcal{B}_i}\hat f_b(x_i). $$ For observations with OOB predictions, the regression OOB mean squared error is the average squared difference between the observed response and its OOB prediction.

\operatorname{MSE}_{\mathrm{OOB}} =\frac{1}{n_{\mathrm{covered}}} \sum_{i:\,B_i>0} \bigl(y_i-\hat f_{\mathrm{OOB},i}\bigr)^2,

where $n_{\mathrm{covered}}$ is the number of training observations with at least one OOB tree. With enough trees, OOB predictions are typically available for all training observations. If some observations have no OOB trees, they cannot contribute an OOB prediction and are not included in this average.

Worked Example: Calculating Regression OOB Error

A hypothetical portfolio model predicts annual claim cost in hundreds of dollars. Four training observations have actual responses $10, 15, 20,$ and $25$. The OOB tree predictions for the observations are:

Observation A: 8 and 12.
Observation B: 13, 17, 21, and 17.
Observation C: 18, 22, 26, and 22.
Observation D: 23 and 27.

For each observation, average only the listed OOB tree predictions:

\begin{aligned} \hat f_{\mathrm{OOB},A}&=(8+12)/2=10,\\ \hat f_{\mathrm{OOB},B}&=(13+17+21+17)/4=68/4=17,\\ \hat f_{\mathrm{OOB},C}&=(18+22+26+22)/4=88/4=22,\\ \hat f_{\mathrm{OOB},D}&=(23+27)/2=25. \end{aligned}

The squared errors are $ (10-10)^2=0$, $ (15-17)^2=4$, $ (20-22)^2=4$, and $ (25-25)^2=0$. Thus,

\operatorname{MSE}_{\mathrm{OOB}}=\frac{0+4+4+0}{4}=2.

The OOB mean squared error is 2 squared hundreds-of-dollars units. Its square root is $\sqrt{2}=1.4142\ldots$ hundreds of dollars, or $141.42 to the nearest cent. The check is that the four squared errors sum to 8 and their mean is 2. A common mistake is to average all-tree predictions for a training observation; that would no longer be its OOB prediction.

Interpreting the Prediction and OOB Error

For a new case, the usual forest prediction uses all $B$ fitted trees. OOB error is different: it summarizes prediction errors on training observations, where each observation’s prediction was formed using only trees that left that observation out of their bootstrap samples. An OOB prediction for a training observation is therefore not the same quantity as that observation’s full-forest prediction.

OOB error is a useful internal estimate of predictive performance, but it is not a guarantee of test-set performance. Its reliability depends on such factors as the training sample, the number of OOB trees available per observation, and the modelling choices made during development. If model choices are repeatedly adjusted in response to the same OOB result, the result may become optimistic as a final assessment. A separate test set can provide an additional evaluation when one is available.

Prediction: average the outputs of all trees for a new case.
OOB prediction: average only the trees that excluded the particular training case.
OOB error: compare each covered training response with its OOB prediction, then average the squared errors for regression MSE.

These calculations are aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus. On an exam problem, identify which trees are included before averaging, preserve the response units, and distinguish the requested forest prediction from any individual-tree output or pooled set of terminal-node records.

Check Your Understanding

For each problem, first identify the tree-level or OOB predictions that belong in the calculation. Answers are given at the end of each item.

A three-tree regression forest predicts a new case’s response as 31, 35, and 40. What is the forest prediction? (Answer: $106/3=35.3333\ldots$)
A case reaches terminal nodes with response values 4 and 8 in Tree 1, 5, 7, and 9 in Tree 2, and 10 and 14 in Tree 3. Each tree predicts its terminal-node mean. What is the forest prediction? (Answer: $25/3=8.3333\ldots$)
A five-tree forest gives a new case individual predictions of 12, 15, 18, 21, and 24. Calculate the forest prediction and state whether it is an individual tree output. (Answer: 18; yes, it is also the third tree’s output.)
Three observations have actual responses 6, 10, and 14. Their OOB predictions are 7, 8, and 16, respectively. Calculate the OOB MSE and OOB RMSE. (Answer: MSE $=3$; RMSE $=\sqrt{3}\approx1.7321$ response units.)
Two observations have actual responses 20 and 30. The first has OOB tree predictions 18 and 22; the second has OOB tree predictions 27, 33, and 36. Find both OOB predictions and the OOB MSE. (Answer: OOB predictions 20 and 32; MSE $=(0^2+(-2)^2)/2=2$ squared response units.)

What You'll Learn