Bagging Regression Trees

Why Bagging Uses Many Regression Trees

A regression tree can represent nonlinear relationships and interactions, but small changes in its training data can lead to different splits and terminal nodes. Bagging—short for bootstrap aggregating—builds many trees on bootstrap samples of the training data, then averages their predictions. The result is an ensemble: a prediction formed from multiple fitted trees, not a newly fitted single tree.

The procedure is useful when a tree-fitting method is sensitive to the particular observations it receives. The bootstrap samples create alternative versions of the training data; averaging the resulting predictions reduces reliance on any one version. Bagging regression trees is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.

Definition: Let the training data contain $n$ observational units, with predictor vectors $x_i$ and numerical responses $y_i$, for $i=1,\ldots,n$. A standard bootstrap sample is formed by drawing $n$ observations independently with replacement from those $n$ training units. A regression tree is fitted to each bootstrap sample. If $\hat f_b^*(x)$ is the prediction from the tree fitted to bootstrap sample $b$, the bagged regression prediction at predictor value $x$ is $$ \hat f_{\mathrm{bag}}(x)=\frac{1}{B}\sum_{b=1}^{B}\hat f_b^*(x), $$ where $B$ is the number of bootstrap samples and fitted trees.

The star indicates that a tree was fitted using a bootstrap sample rather than the original data exactly once. The $B$ samples are generated separately, so a training observation can appear several times in one sample and not appear at all in another. The number of draws in each sample is $n$ under the standard procedure, but the number of distinct original units in that sample is generally less than $n$.

A regression tree partitions predictor space into regions called terminal nodes. Under the usual squared-error criterion, the prediction in a terminal node is the mean response of the training observations assigned to that node. For a bootstrap-fitted tree, repeated copies of a training observation contribute repeatedly: the tree fitting treats them as repeated rows. The same split-selection and stopping rules are applied to each sample. Those rules may specify a maximum tree size, a minimum node size, or another stopping condition.

The Bagging Procedure

The central steps are straightforward, but it is important to keep resampling, fitting, and prediction separate. In particular, a bootstrap sample is used to fit one tree; bagging does not pool all bootstrap samples into a single large training set and fit one tree to that pool.

Fix the training data and fitting rule.
Record the $n$ training units and specify the regression-tree fitting and stopping rules. Apply the same rules for each bootstrap fit unless the procedure explicitly says otherwise.

Draw a bootstrap sample.
Select $n$ training units with replacement. Keep repeated selections as repeated observations; do not replace them with distinct units or silently discard them.

Fit a regression tree.
Fit one tree to that bootstrap sample. Its splits and terminal-node predictions are determined by the sampled rows and the specified fitting rule.

Repeat the resampling and fitting.
Generate $B$ bootstrap samples and fit one tree to each. The individual trees need not have the same splits or terminal nodes.

Average predictions at the requested input.
For a predictor value $x$, obtain a prediction from each fitted tree and take their arithmetic mean. The result is the bagged regression prediction at $x$.

Bagging is often paired with large or unpruned trees because it is intended to stabilize predictions from trees that can change substantially when the data change. This is a common choice, not a mathematical requirement: the tree-growing and stopping rules are part of the specified algorithm. Bagging does not, by itself, prescribe one unique tree depth or guarantee that a particular degree of complexity is best.

Why the Average Is the Bagged Prediction

For fixed training data, imagine generating bootstrap samples repeatedly according to the bootstrap sampling rule. At a particular input $x$, the fitted tree prediction varies from sample to sample. Its conditional bootstrap average is the expected prediction over those possible samples. A finite bagged ensemble estimates this average by taking the sample mean of the $B$ fitted predictions.

Justification: Conditional on the observed training data, let $\hat f^*(x)$ denote the prediction from one tree fitted to a newly generated bootstrap sample. If the $B$ bootstrap samples are generated independently by the same procedure, then the predictions $\hat f_1^*(x),\ldots,\hat f_B^*(x)$ have the same conditional expectation. Linearity of expectation gives $$ E_*\!\left[\frac{1}{B}\sum_{b=1}^{B}\hat f_b^*(x)\,\middle|\,\text{training data}\right] =\frac{1}{B}\sum_{b=1}^{B}E_*\!\left[\hat f_b^*(x)\mid\text{training data}\right] =E_*\!\left[\hat f^*(x)\mid\text{training data}\right]. $$ Thus the ensemble average is an unbiased Monte Carlo estimate of the conditional bootstrap-average prediction. Here $E_*$ denotes expectation over bootstrap resampling, conditional on the observed training data. This argument explains the averaging step; it does not say that a finite ensemble equals the exact average over every possible bootstrap sample.

The average is computed at each input value. It is not generally a single tree’s prediction, nor does it imply that a tree with the averaged structure has been fitted. Since each tree can partition predictor space differently, its prediction can come from a different terminal node than the predictions from the other trees.

Worked Example: Fitting Two Bootstrap Trees on a Small Data Set

Consider four training observations with predictor $x$ and response $y$: $(1,2)$, $(2,4)$, $(3,8)$, and $(4,10)$. For this illustration, fit a regression stump to each bootstrap sample: a tree with at most two terminal nodes, choosing a split that minimizes the sum of squared deviations from the terminal-node means. Two bootstrap samples are $(1,2,2,4)$ and $(1,1,3,4)$, where the entries denote the original observation numbers. Find the two tree predictions at $x=2.5$ and their bagged average.

First sample. The sampled responses, in order, are $2,4,4,10$. A split between $x=1$ and $x=2$ gives a left-node mean of $2$ and a right-node mean of $(4+4+10)/3=6$. The within-node sum of squared errors is $0+(4-6)^2+(4-6)^2+(10-6)^2=24$.

A split between $x=2$ and $x=4$, at threshold $x=3$, gives a left-node mean of $(2+4+4)/3=10/3$ and a right-node mean of $10$. Its sum of squared errors is \[ (2-10/3)^2+(4-10/3)^2+(4-10/3)^2+(10-10)^2 =16/9+4/9+4/9+0=8/3. \] Since $8/3<24$, this split is preferred. The first tree predicts $10/3$ at $x=2.5$, which falls in the left node.

Second sample. Its responses are $2,2,8,10$, at predictor values $1,1,3,4$. A split at $x=2$ gives a left-node mean of $2$ and a right-node mean of $(8+10)/2=9$. The sum of squared errors is $0+0+(8-9)^2+(10-9)^2=2$. The alternative split between $x=3$ and $x=4$ has left-node mean $(2+2+8)/3=4$, right-node mean $10$, and sum of squared errors $ (2-4)^2+(2-4)^2+(8-4)^2+0=24$. Thus the split at $x=2$ is preferred, and this tree predicts $9$ at $x=2.5$.

Bagged prediction. With these two trees, the ensemble prediction is \[ \frac{10/3+9}{2}=\frac{37}{6}=6.1667 \] to four decimal places. The individual predictions are $10/3$ and $9$; the ensemble prediction is their average, not either tree’s prediction. This example uses two specified bootstrap samples to illustrate the procedure, not to suggest that two trees are generally sufficient.

Method and mistake avoided. Fit each tree to its own resample, evaluate both trees at the same $x$, and then average. A common error is to use the second tree alone because it was fitted last. That would give $9$, not the bagged prediction $37/6$.

Interpreting Component Predictions and the Ensemble

Each component tree is a fitted function of the predictors. The bagged model forms a new prediction function by averaging those component functions point by point. Even if every individual tree has a piecewise-constant prediction, the ensemble’s regions of constant prediction need not match the regions of any one tree: boundaries from different trees can overlap in different ways.

The averaging rule gives equal weight to each fitted tree in standard bagging. It does not weight a tree more heavily because its bootstrap sample contains more distinct observations, because it has more terminal nodes, or because its prediction happens to be closer to a response that will later be observed. A different weighting scheme would be a different procedure and would need to be specified.

Worked Example: Averaging Predictions From Four Fitted Trees

A hypothetical insurer uses four bootstrap-fitted regression trees to predict a numerical response representing a future claim-cost measure for a particular risk profile. The fitted predictions, in dollars, are $120$, $150$, $90$, and $140$. Find the bagged prediction and explain why it is not the prediction of a single component tree.

Step 1: Add the component predictions. $120+150+90+140=500$ dollars.

Step 2: Divide by the number of trees. There are $B=4$ trees, so the bagged prediction is $500/4=125$ dollars.

Step 3: Compare with the component predictions. The four component values are $120$, $150$, $90$, and $140$; none is $125$. Therefore, the ensemble prediction is not the output of any of these four trees at this risk profile. It is the equal-weight average of their outputs.

Method and mistake avoided. Average the four predictions at the same risk profile. Do not average predictions evaluated at different profiles, and do not report the prediction from a selected component tree as if it were the ensemble output. The calculation is exact for the supplied values; no rounding is needed.

Worked Example: Recognizing an Ensemble Output Between Tree Outputs

For a second hypothetical prediction task, five fitted regression trees give predictions of $4$, $8$, $8$, $12$, and $18$ thousand dollars at the same input $x$. Calculate the ensemble prediction, state whether it equals a component prediction, and identify what would change if the final tree were omitted.

Step 1: Sum all five outputs. $4+8+8+12+18=50$ thousand dollars.

Step 2: Average over the five trees. The bagged prediction is $50/5=10$ thousand dollars. None of the five component predictions is $10$, so this ensemble output lies between component outputs without matching any of them.

Step 3: Recalculate if the final tree is omitted. The remaining four outputs sum to $4+8+8+12=32$ thousand dollars. Their average is $32/4=8$ thousand dollars. Omitting a tree changes both the sum and the divisor; it does not mean that the original five-tree average is still used.

Method and mistake avoided. Apply the stated average using exactly the component trees included in the ensemble. A frequent arithmetic error is to divide the original total by four after one tree has been omitted. That would give $50/4=12.5$, which incorrectly retains the omitted tree’s prediction in the total.

Assumptions, Scope, and Common Traps

The standard row bootstrap is most directly motivated when observational units can reasonably be treated as independent draws from a common population. The algorithm itself can be run on many data sets, but ordinary row-wise resampling may not respect dependence among observations. For example, repeated measurements on one person or serially dependent time-series observations should not automatically be resampled as though every row were independent. The resampling unit and method must match the data structure.

No normality assumption for the response is required to define the bagging algorithm. Under the squared-error tree criterion, terminal-node predictions are means, but that fitting rule does not assert that responses have a normal distribution. Other regression-tree criteria or response summaries can be specified; the prediction-averaging principle remains the arithmetic mean for standard regression bagging.

The following distinctions prevent frequent errors:

Bootstrap sample versus original data. A tree is fitted to one resampled data set at a time. Repeated rows remain repeated observations in that fit.
Tree prediction versus ensemble prediction. A component tree produces one prediction; bagging averages the predictions from all included trees.
Resampling versus fitting. Drawing bootstrap samples does not itself create predictions. A tree must be fitted to each sample before its prediction can contribute to the ensemble.
Number of draws versus number of distinct units. A standard sample has $n$ draws, but typically fewer than $n$ distinct training units because sampling is with replacement.
Bagging versus one pooled tree. Fitting one tree after combining all bootstrap rows is not the same procedure as fitting separate trees and averaging their predictions.

The main exam trap is to identify the bootstrap replicate correctly but then report only its tree prediction, rather than averaging predictions across the ensemble. Another is to average terminal-node response values without first determining which node each tree assigns the requested input to. The required operation is always: evaluate every included fitted tree at the same input, then average those outputs.

Key takeaway: Bagging regression trees means repeatedly drawing bootstrap samples, fitting one regression tree to each sample, and averaging the trees’ numerical predictions at the requested input. The ensemble prediction belongs to the collection of fitted trees; it need not equal the prediction from any individual tree.

Check Your Understanding

For each problem, follow the stated resampling and prediction rules carefully.

A training set has $n=60$ rows. Under the standard bootstrap, how many draws are in each bootstrap sample, and can an observation appear more than once? (Answer: 60 draws; yes, an observation can appear more than once.)
Three fitted trees predict $11$, $17$, and $14$ units at the same input. Find the bagged prediction and state whether it equals a component prediction. (Answer: 14 units; yes, it equals the third tree’s prediction.)
Four trees give predictions of $3$, $7$, $10$, and $12$ thousand dollars. Find the ensemble prediction. A student reports $12$ thousand dollars because that is the largest prediction. Correct the student. (Answer: $8$ thousand dollars; bagging uses the mean $(3+7+10+12)/4=8$, not the largest component output.)
A bootstrap sample contains original row numbers $2,2,4,5,5,5$. How many draws and how many distinct original rows does it contain? When fitting its tree, should the repeated rows be deleted? (Answer: 6 draws and 3 distinct rows; no, the repeated rows remain repeated in the bootstrap fit.)
At a specified input, five trees predict $2$, $5$, $5$, $9$, and $14$. One tree with prediction $14$ is removed from the ensemble. Calculate the original and revised bagged predictions. (Answer: original $7$; revised $21/4=5.25$.)

What You'll Learn