Why a Regression Forest Averages
A regression tree assigns a numerical prediction to the terminal node reached by an observation. A forest sends the same predictor profile through many trees, which may have different splits and terminal nodes, and combines the resulting numerical predictions. For a standard regression forest, that combination is the arithmetic mean across trees.
This is different from pooling every response value found in every reached terminal node and taking one overall mean. Each tree contributes one prediction, regardless of how many training observations are in its terminal node. Keeping that distinction clear is essential when interpreting the forest’s output.
The formula assumes equal weight for each tree. It applies whether the trees were fitted using bootstrap samples or another sampling scheme; the sampling affects the fitted trees and their predictions, not the arithmetic aggregation rule defined here. If a method explicitly assigns different tree weights, its aggregation formula must reflect those weights instead.
What the Forest Prediction Means
Each tree prediction is on the response scale used to fit the regression forest. If the response is claim cost in dollars, the forest output is in dollars. If costs were recorded in thousands of dollars, the output is in thousands. The average is not a vote, and it is not a probability unless the response itself represents a probability-valued quantity and the model is being interpreted that way.
In the standard squared-error tree described above, a leaf prediction is a mean of responses in that leaf. Such a mean cannot be below the smallest or above the largest response in that leaf. Consequently, each tree prediction lies within the range of training responses used to fit that tree. Averaging those predictions also keeps the forest output within the range of its tree predictions. If all trees use responses from the same overall training data, the forest prediction is therefore within the overall training-response range.
This range property is one reason a tree forest does not extrapolate beyond its fitted response range in the way a regression equation can. It is a property of the standard mean-in-leaf prediction rule and the averaging step, not a guarantee that every prediction will be close to the true value. If a method uses a different terminal-node estimator, such as a local model that can predict beyond the responses in a node, the range conclusion need not hold.
Worked Example: Averaging Three Claim-Cost Predictions
A hypothetical insurer uses a regression forest to predict claim cost for one risk profile. Three trees return terminal-node predictions of $10,000, $10,000, and $14,000. Calculate the forest prediction and check its scale and bounds.
Method. Use the equal-weight arithmetic mean of the three tree predictions. The fact that two predictions are equal does not change the rule: each tree contributes once.
First add the predictions: $$ 10{,}000+10{,}000+14{,}000=34{,}000. $$
Then divide by the number of trees: $$ \hat f_{\mathrm{forest}}(x)=\frac{34{,}000}{3} =11{,}333.333\ldots. $$
Rounded to the nearest cent, the forest prediction is $11,333.33. It is in dollars, the same units as the tree predictions. It is also between the smallest and largest tree predictions, $10,000 and $14,000, as an arithmetic mean must be.
Common mistake avoided. Reporting the smallest tree prediction, $10,000, does not average the trees. The correct calculation uses all three predictions, including the $14,000 result.
Why Averaging Preserves Bounds
The range property follows directly from the definition of an average. Let \(m\) be the smallest tree prediction and \(M\) the largest. For every tree \(b\), \(m\leq \hat f_b(x)\leq M\). Adding these \(B\) inequalities and dividing by the positive number \(B\) gives
Thus, the forest prediction is a convex combination of the tree predictions: each receives the same nonnegative weight \(1/B\), and the weights sum to one. More generally, any nonnegative weights that sum to one would keep a weighted average within the smallest and largest component predictions. For the standard equal-weight forest, the result is specifically the arithmetic mean.
When each tree uses the ordinary mean response in a terminal node, that leaf mean is also bounded by the smallest and largest responses in the leaf. If all leaves are formed from observations in the training data, both the leaf predictions and their forest average are bounded by the overall training-response minimum and maximum.
Worked Example: Different Leaf Sizes, Equal Tree Weights
A hypothetical claim-cost forest uses costs measured in thousands of dollars. The overall training responses range from $2,000 to $12,000. For a particular risk profile, Tree A places the case in a leaf with training responses 2 and 4 (thousands of dollars). Tree B places it in a leaf with responses 2, 4, and 10 (thousands). Find the forest prediction and check it against the response range.
Step 1: Find Tree A’s terminal-node prediction. Under the mean-in-leaf rule, $$ \hat f_A(x)=\frac{2+4}{2}=3 $$ thousand dollars.
Step 2: Find Tree B’s terminal-node prediction. $$ \hat f_B(x)=\frac{2+4+10}{3} =\frac{16}{3} =5.3333\ldots $$ thousand dollars.
Step 3: Average the two tree predictions. There are two trees, so each gets weight \(1/2\): $$ \hat f_{\mathrm{forest}}(x) =\frac{3+\frac{16}{3}}{2} =\frac{\frac{9}{3}+\frac{16}{3}}{2} =\frac{25}{6} =4.1666\ldots $$ thousand dollars.
The result is $4.167 thousand dollars, or $4,166.67 to the nearest cent. It lies between the two tree predictions, 3 and \(16/3\) thousand, and within the overall training-response range of 2 to 12 thousand dollars.
Common mistake avoided. Pooling the five listed leaf responses would give \((2+4+2+4+10)/5=4.4\) thousand dollars. That weights Tree B more heavily because its leaf contains more observations. The forest instead gives Tree A and Tree B equal weight, so the correct prediction is 4.167 thousand dollars.
Averaging and Smoothing
A single regression tree usually gives a piecewise-constant prediction: nearby predictor profiles may receive the same prediction until a split boundary is crossed, at which point the prediction can jump. In a forest, different trees can place split boundaries in different locations. Averaging their outputs can soften a jump in one tree when other trees give predictions that do not jump at that same boundary.
This is a smoothing effect, not a guarantee of a mathematically smooth or continuous prediction function. A forest prediction can still jump when enough tree predictions change at the same location. Nor does averaging force the predictions to increase with a predictor, remove all sensitivity to unusual data, or correct a poorly specified prediction problem. It combines the fitted trees; it does not rewrite their rules.
Worked Example: How Tree Jumps Can Be Softened
A hypothetical forest predicts claim cost in thousands of dollars using a risk score. At five increasing score values, four trees return the following predictions. Each row represents one score value, and each column represents one tree.
| Risk score | Tree 1 | Tree 2 | Tree 3 | Tree 4 |
|---|---|---|---|---|
| 1 | 2 | 2 | 4 | 2 |
| 2 | 2 | 5 | 4 | 2 |
| 3 | 8 | 5 | 4 | 6 |
| 4 | 8 | 9 | 7 | 6 |
| 5 | 8 | 9 | 12 | 10 |
Method. Average across each row, because a row contains the four tree predictions for one profile. At score 1, the sum is \(2+2+4+2=10\), so the forest prediction is \(10/4=2.50\) thousand dollars. At score 2, it is \((2+5+4+2)/4=13/4=3.25\). At score 3, it is \((8+5+4+6)/4=23/4=5.75\). At score 4, it is \((8+9+7+6)/4=30/4=7.50\). At score 5, it is \((8+9+12+10)/4=39/4=9.75\).
Tree 1 jumps from 2 to 8 between scores 2 and 3, while other trees have different prediction patterns. The forest predictions rise from 3.25 to 5.75 over that step rather than taking Tree 1’s full jump alone. The forest sequence is less affected by that single tree’s change, but the example does not establish that every forest must be smooth or monotone.
Common mistake avoided. A forest prediction at a score is the average across that score’s row, not an average down one tree’s column. The latter would combine predictions for different risk profiles.
Aggregation Traps and Interpretation Checks
Before reporting a regression forest prediction, verify what is being averaged and on which scale. These checks prevent several plausible but incorrect interpretations.
- Average tree predictions, not all leaf observations. A terminal node with many training cases does not give its tree extra weight in the standard equal-weight forest.
- Keep profiles aligned. Every \(\hat f_b(x)\) in the sum must refer to the same predictor profile \(x\). Averaging outputs for different profiles does not produce the forest prediction for either one.
- Use the correct units. If the response and tree predictions are in thousands of dollars, the average is also in thousands of dollars. Convert to dollars only after averaging if needed.
- Do not confuse averaging with inverse transformation. If trees predict on a transformed response scale, averaging those outputs and then transforming back is generally not the same as transforming each tree output back and averaging. State which scale the model uses before interpreting the result.
- Do not infer more than the average guarantees. The output lies between the tree predictions, but that fact alone does not establish accuracy, calibration, fairness, or a causal effect.
The central calculation is compact, but its interpretation depends on a careful reading of the response definition, the units, and the aggregation rule. In a standard regression forest, each tree contributes one prediction for the same case, and their arithmetic mean is the forest’s prediction.
Check Your Understanding
For each problem, calculate the requested regression-forest prediction and interpret it using the stated units.
- A forest predicting repair cost in dollars has four tree predictions of $7,000, $11,000, $8,000, and $14,000 for one vehicle. Find the forest prediction. (Answer: $10,000)
- Two trees use mean responses in their terminal nodes. The first leaf contains responses 6 and 10; the second contains 4, 8, and 12. All values are thousands of dollars. Find the forest prediction, giving each tree equal weight. (Answer: $8,000)
- For one risk profile, three trees predict 5, 7, and 12 thousand dollars. A fourth tree predicts 8 thousand dollars. Find the four-tree prediction and state whether it lies within the range of tree predictions. (Answer: $8,000; yes, it lies between $5,000 and $12,000)
- For a particular case, Tree A predicts 3 thousand dollars from a leaf containing responses 2 and 4. Tree B predicts 5 thousand dollars from a leaf containing responses 1, 4, and 10. Calculate the forest prediction. Explain why pooling the five leaf responses is not the standard forest calculation. (Answer: $4,000; the trees receive equal weight, so the forest averages 3 and 5 rather than pooling the leaf observations)
- A forest predicts a numeric response in percentage points. At one profile, its five trees return 18, 22, 25, 25, and 30. Calculate the prediction, then state whether it is within the range of tree predictions. (Answer: 24 percentage points; yes, it is between 18 and 30)