Hand-Calculating Bagged Regression Predictions

Why Bagged Predictions Require Tree-by-Tree Calculation

A regression tree can give different predictions for the same input depending on the bootstrap sample used to fit it. The tree’s split locations can change, and the observations that fall into a terminal leaf can change. Bagging combines these fitted trees by averaging their predictions, not by combining their splits or averaging the original responses first.

The calculation is manageable by hand when each tree has only a few leaves: find the prediction from each tree at the requested input, then take the arithmetic mean across trees. This tutorial focuses on that calculation and on interpreting how the resulting prediction changes as the input changes. The method is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.

Definition: Suppose there are $n$ training observations. A bootstrap sample of size $n$ is drawn from those observations with replacement. A regression tree is fitted to each bootstrap sample. For $B$ fitted trees with prediction functions $\hat f_1,\ldots,\hat f_B$, the bagged prediction at input $x$ is $$ \hat f_{\mathrm{bag}}(x)=\frac{1}{B}\sum_{b=1}^{B}\hat f_b(x). $$ Each tree receives the same weight, and each tree is evaluated at the same input $x$.

In a regression tree fitted using squared-error loss, the prediction in a terminal leaf is the mean of the response values assigned to that leaf in the tree’s fitting sample. A bootstrap sample can contain the same training observation more than once. Each occurrence counts in the leaf mean: if a response appears twice in the bootstrap sample, it contributes twice.

The prediction is piecewise constant in $x$: all inputs routed to a given terminal leaf receive that leaf’s mean. Since different trees can have different splits, an input must be routed separately through every tree. Do not average the split points or use one tree’s route for the whole ensemble.

Why the Leaf Prediction Is a Mean

The leaf-mean rule follows from the squared-error fitting criterion. If a terminal leaf contains responses $y_1,\ldots,y_m$, a constant prediction $c$ in that leaf has total squared error $\sum_{j=1}^{m}(y_j-c)^2$. Differentiating with respect to $c$ gives

\frac{d}{dc}\sum_{j=1}^{m}(y_j-c)^2 =-2\sum_{j=1}^{m}(y_j-c).

Setting this derivative equal to zero yields $mc=\sum_{j=1}^{m}y_j$, so $c=\bar y$, the arithmetic mean of the responses in the leaf. The second derivative is $2m>0$, confirming that this value minimizes the squared error. In a bootstrap-fitted tree, the list $y_1,\ldots,y_m$ includes repeated responses according to their multiplicities in that bootstrap sample.

This justification applies to a leaf fitted by minimizing squared error with a constant prediction. If a different loss function or a different leaf-prediction rule is used, the leaf output need not be the arithmetic mean. The bagged prediction is still the average of the fitted trees’ outputs, but those individual outputs must be calculated using the rule appropriate to the model.

A Reliable Calculation Routine

Read each tree’s bootstrap sample.
Keep repeated observations as separate entries. Do not reduce a sample to its distinct observation labels before calculating a leaf mean.

Find the relevant leaf in each tree.
Apply that tree’s split rules to the requested input. Check carefully which side receives equality at a split.

Calculate each tree’s output.
For squared-error regression, add the bootstrap response values in the reached leaf and divide by their count.

Average the tree outputs.
Add the $B$ predictions and divide by $B$, regardless of differences in leaf sizes or bootstrap multiplicities.

Worked Calculations

Worked Example: Bootstrap Multiplicities and Two Inputs

A hypothetical insurer models a continuous claim-cost index $y$ from a rating score $x$. The training data are $(1,2),(2,6),(3,8),(4,14)$, where each pair is $(x,y)$. Two bootstrap samples, each of size four, are used to fit squared-error regression trees. Tree A’s sample contains observations $1,2,2,4$; Tree B’s contains observations $1,3,3,4$. Both trees split at $x=3$, sending $x<3$ left and $x\geq 3$ right. Calculate the bagged predictions at $x=2.5$ and $x=3$.

Method. Calculate each terminal-leaf mean using the observations in that tree’s bootstrap sample, including repeats, then average the two tree predictions at each input.

Tree A’s left leaf. This leaf contains bootstrap responses $2,6,6$. Its prediction is $ (2+6+6)/3=14/3 $. Tree A’s right leaf contains only response $14$, so its prediction is $14$.

Tree B’s left leaf. This leaf contains response $2$, so its prediction is $2$. Tree B’s right leaf contains responses $8,8,14$, giving prediction $ (8+8+14)/3=30/3=10 $.

At $x=2.5$, both trees route left. Therefore, $ \hat f_{\mathrm{bag}}(2.5)=(14/3+2)/2=(20/3)/2=10/3 $, or approximately $3.333$.

At $x=3$, equality goes right in both trees. Therefore, $ \hat f_{\mathrm{bag}}(3)=(14+10)/2=12 $.

Check. The left-side average is between the left predictions $2$ and $14/3$; $10/3$ is between them. At $x=3$, the average of $14$ and $10$ is $12$. The calculation preserves the repeated responses in both bootstrap samples.

Worked Example: A Repeated Response Changes a Leaf Mean

A hypothetical severity model is fitted to three training responses $2,8,14$. At a particular input, Tree 1 reaches a leaf whose size-three bootstrap sample contains the first response twice and the second response once: $2,2,8$. Tree 2 reaches a leaf containing all three responses $2,8,14$. Find the bagged prediction at this input.

Tree 1. Its leaf mean is $ (2+2+8)/3=12/3=4 $. The repeated response $2$ is counted twice because it occurs twice in the bootstrap sample.

Tree 2. Its leaf mean is $ (2+8+14)/3=24/3=8 $.

There are two tree predictions, $4$ and $8$, so the bagged prediction is $ (4+8)/2=12/2=6 $.

Common mistake and correction. If the repeated $2$ in Tree 1’s bootstrap sample is mistakenly discarded, the leaf mean is calculated as $ (2+8)/2=5 $. Averaging that incorrect value with Tree 2’s prediction gives $ (5+8)/2=6.5 $, which is wrong. Tree 1’s leaf has three sampled responses, not two, so its correct prediction is $4$, and the correct bagged prediction is $6$.

Check. The final prediction $6$ lies halfway between the two fitted tree outputs, $4$ and $8$, as required for an equal-weight average of two trees.

Worked Example: Tree-Specific Split Boundaries

Two fitted regression stumps use different split points and send equality right. Tree A splits at $x=2$: its left leaf has mean response $3$, and its right leaf has mean response $5$. Tree B splits at $x=4$: its left leaf has mean response $11$, and its right leaf has mean response $15$. Calculate the bagged predictions at $x=3$ and $x=4$, and identify which tree changes leaf between the two inputs.

The rule “equality right” means the left leaf is reached when $x<c$, and the right leaf when $x\geq c$, for split point $c$.

At $x=3$, Tree A routes right because $3\geq2$, and predicts $5$. Tree B routes left because $3<4$, and predicts $11$. Thus $ \hat f_{\mathrm{bag}}(3)=(5+11)/2=8 $.

At $x=4$, Tree A still routes right because $4\geq2$, and still predicts $5$. Tree B routes right because equality at its split goes right, and predicts $15$. Thus $ \hat f_{\mathrm{bag}}(4)=(5+15)/2=10 $.

Only Tree B changes leaf between $x=3$ and $x=4$. Tree A sends both inputs to its right leaf. The bagged prediction changes from $8$ to $10$ because Tree B’s output changes from $11$ to $15$, while Tree A’s output remains $5$.

Common mistake and correction. It is incorrect to say Tree A changes leaf at $x=4$: its split is at $2$, so both inputs satisfy $x\geq2$. It is also incorrect to send $x=4$ left in Tree B; equality goes right. Applying each split rule to each input gives the consistent predictions $8$ and $10$.

What Averaging Does—and Does Not Do

A bagged prediction is an equally weighted average of the fitted tree predictions at a single input. This means it lies between the smallest and largest of those predictions. For example, if three trees predict $4,9,13$, their bagged prediction is $ (4+9+13)/3=26/3 $, which is between $4$ and $13$. It need not equal any individual tree prediction.

Averaging can make the ensemble’s output less dependent on the particular fitted tree at that input, but it does not turn the trees into one tree with an averaged split. In particular, the average of predictions at $x_1$ and $x_2$ need not be the prediction at an input between $x_1$ and $x_2$: the tree outputs are piecewise constant, and the ensemble can change whenever one or more trees cross a split.

Keep two averages distinct. First, each tree’s leaf mean averages responses from that tree’s bootstrap sample, with repeats counted. Second, the bagged prediction averages the $B$ tree predictions, with each tree counted once. A large leaf does not give its tree extra weight in the second average.

Key takeaway: At every requested input, route through each tree separately, compute the reached leaf’s prediction using that tree’s fitting rule, and take the equal-weight mean of the $B$ outputs. Preserve bootstrap multiplicities and apply each split’s equality rule exactly.

Check Your Understanding

For each problem, calculate the requested prediction by keeping the tree-level and leaf-level averages separate.

At one input, three regression trees predict $7,11,$ and $14$. Find the bagged prediction. (Answer: $32/3$, approximately $10.6667$)
At a query input, Tree A reaches a bootstrap leaf with responses $4,4,10,14$. Tree B predicts $12$ there. Find the bagged prediction. (Answer: $10$)
Tree A splits at $x=3$, with left and right predictions $2$ and $8$. Tree B splits at $x=5$, with left and right predictions $10$ and $16$. Both send equality right. Find the bagged predictions at $x=4$ and $x=5$. (Answer: $9$ at $x=4$ and $12$ at $x=5$)
At the same input, three trees have bootstrap-leaf responses $1,1,7,11$; $3,7,7,11$; and $5,5,9,13$, respectively. Find each tree prediction and then the bagged prediction. (Answer: tree predictions $5,7,8$; bagged prediction $20/3$, approximately $6.6667$)
Four trees predict $2,6,10,14$ at an input. A fifth tree is added and predicts $20$. Find the original four-tree bagged prediction and the new five-tree bagged prediction. (Answer: $8$ and $10.4$, respectively)

What You'll Learn