Why Tree Depth Matters in Boosting
Boosting builds a prediction by adding the contributions of a sequence of trees. The depth allowed for each tree controls what patterns an individual update can express. A very shallow tree may capture a simple change associated with one predictor; a deeper tree can make a prediction depend on combinations of predictors. That extra flexibility can represent useful interactions, but it can also fit noise in the training data.
The distinction is structural. Increasing the number of boosting updates adds more terms to the prediction, while increasing maximum tree depth changes the kinds of terms each tree can represent. These are different ways to increase model flexibility and should not be confused.
For squared-error boosting, a fitted model can be written as \(F_M(x)=F_0(x)+\sum_{m=1}^{M}\eta f_m(x)\), where \(F_0\) is the initial prediction, \(f_m\) is the tree contribution at update \(m\), and \(\eta\) is the shrinkage factor. In the squared-error residual-fitting procedure, the next tree is fitted to the current residuals \(Y_i-F_{m-1}(x_i)\). The depth restriction applies to each \(f_m\), not to the number \(M\) of updates.
Depth and Interaction Structure
A split uses a predictor to divide the observations, and each path through a tree records a sequence of split conditions. A leaf prediction is therefore selected by the combined conditions on that path. With depth one, a tree can split on only one predictor along any path. Adding many such trees can produce a flexible sum of one-predictor effects, but the resulting prediction function remains additive in those predictors.
A depth-two tree can split first on one predictor and then, within a branch, on a second. The prediction can then depend on which combination of the two predictor values applies. More generally, a path with at most \(d\) splits can involve at most \(d\) distinct predictors and can support interactions among up to \(d\) predictors. If a predictor is used more than once on a path, the number of distinct predictors on that path is smaller than the depth.
Why Stumps Cannot Fit a Two-Predictor Interaction
A short algebraic check makes the restriction precise. Let \(x_1\) and \(x_2\) each take values 0 or 1. For a function \(f\), define its two-factor contrast as \(f(1,1)-f(1,0)-f(0,1)+f(0,0)\). If \(f\) is additive, meaning \(f(x_1,x_2)=c+a(x_1)+b(x_2)\), then this contrast is zero: the constant and each one-variable contribution cancel.
Each stump that splits on \(x_1\) contributes a function of \(x_1\) alone; each stump that splits on \(x_2\) contributes a function of \(x_2\) alone. Their sum is still additive, so its two-factor contrast remains zero. A depth-two tree can instead isolate the four combinations of the two binary predictors, allowing a nonzero contrast. This is why adding more stumps does not, by itself, remove their additive structural restriction.
Worked Example: A Depth-Two Tree Fits an Interaction Pattern
A hypothetical model predicts whether a device requires a particular adjustment. Let \(x_1\) indicate high temperature and \(x_2\) indicate high vibration; both are coded 0 or 1. Four training observations have outcomes \(Y\) as follows: \((0,0)\) has \(Y=0\), \((0,1)\) has \(Y=1\), \((1,0)\) has \(Y=1\), and \((1,1)\) has \(Y=0\). Starting from the squared-error initial prediction, compare what one depth-one tree and one depth-two tree can fit. Use a full update, \(\eta=1\).
Method. The squared-error initial prediction is the sample mean. Each tree is fitted to residuals \(Y-F_0(x)\). Use the residual convention \(Y-F_0(x)\), not its negative.
Step 1: Initial prediction and residuals. The outcome mean is \((0+1+1+0)/4=0.5\), so \(F_0(x)=0.5\) for all four observations. In the order listed, residuals are \(0-0.5=-0.5\), \(1-0.5=0.5\), \(1-0.5=0.5\), and \(0-0.5=-0.5\). Their squared sum is \(0.25+0.25+0.25+0.25=1\).
Step 2: Fit a depth-one tree. If the stump splits on \(x_1\), the residual mean in the \(x_1=0\) group is \((-0.5+0.5)/2=0\); the residual mean in the \(x_1=1\) group is \((0.5-0.5)/2=0\). Thus both leaves predict zero. Splitting on \(x_2\) also gives a residual mean of zero in each leaf. Either stump leaves all four residuals unchanged, so the squared residual sum remains 1.
Step 3: Fit a depth-two tree. Split first on \(x_1\), then split each child on \(x_2\). Each of the four leaves then corresponds to one predictor combination and has one observation. Its residual prediction is that observation’s residual: \(-0.5, 0.5, 0.5,\) or \(-0.5\). With \(\eta=1\), the updated predictions are \(0.5-0.5=0\), \(0.5+0.5=1\), \(0.5+0.5=1\), and \(0.5-0.5=0\). They match the four outcomes, so the new squared residual sum is zero.
Check and interpretation. The observed outcomes have two-factor contrast \(0-1-1+0=-2\), which is nonzero. The stump’s additive prediction cannot match that pattern, while the depth-two tree can. The example shows representational capacity, not a guarantee that a deeper tree will improve predictions on new data. The common mistake is to say that enough stumps must eventually fit this pattern; their sum still has zero two-factor contrast.
Training Fit: A Deeper Tree Can Reduce Error
For squared-error fitting, a leaf prediction is the mean outcome among training observations assigned to that leaf. Splitting a leaf into smaller groups can reduce or leave unchanged the minimum training sum of squared errors, because the unsplit prediction remains available by assigning the same mean to both children. This training-fit property does not establish that a deeper tree will predict better outside the training sample.
Worked Example: Comparing Depth-One and Depth-Two Fits
A hypothetical insurer studies an adjustment score \(Y\) using two binary risk indicators, \(A\) and \(B\). The observed scores are: for \((A,B)=(0,0)\), 2 and 4; for \((0,1)\), 8 and 10; for \((1,0)\), 6 and 8; and for \((1,1)\), 12 and 14. Compare the training sum of squared errors for a root-only prediction, the best stump using either indicator, and a depth-two tree that splits on both indicators.
Step 1: Root-only prediction. The eight observations sum to \(2+4+8+10+6+8+12+14=64\), so the root prediction is \(64/8=8\). The squared errors sum to \((-6)^2+(-4)^2+0^2+2^2+(-2)^2+0^2+4^2+6^2=112\).
Step 2: Stump splitting on \(A\). For \(A=0\), the mean is \((2+4+8+10)/4=6\), and its squared error sum is \(16+4+4+16=40\). For \(A=1\), the mean is \((6+8+12+14)/4=10\), and its squared error sum is \(16+4+4+16=40\). Total: \(40+40=80\).
Step 3: Stump splitting on \(B\). For \(B=0\), the mean is \((2+4+6+8)/4=5\); its squared error sum is \(9+1+1+9=20\). For \(B=1\), the mean is \((8+10+12+14)/4=11\); its squared error sum is \(9+1+1+9=20\). Total: \(20+20=40\), so this is the better of the two stumps.
Step 4: Depth-two tree. Split on \(A\), then split each branch on \(B\). The four leaf means are 3, 9, 7, and 13, respectively. Each leaf contains two observations differing from its mean by \(-1\) and \(1\), so each contributes \(1+1=2\) to squared error. Total: \(4(2)=8\).
Check and interpretation. The training sums of squared errors are 112 at the root, 80 for the best stump on \(A\), 40 for the stump on \(B\), and 8 for the depth-two tree. The depth-two tree fits more closely, although these cell means happen to be additive: they can be written as \(3+4A+6B\). Thus a deeper tree permits interactions but does not require them. The common mistake is to infer from a lower training error alone that the deeper structure is preferable for prediction.
Depth Is a Bias–Variance Choice
Restricting depth can reduce the range of patterns a tree can fit. If the true relationship contains an interaction that shallow trees cannot represent, the restriction can create systematic approximation error. Allowing greater depth can reduce that limitation, but small leaves may be determined by very few observations, making fitted values sensitive to the training sample. In that sense, depth is a complexity control: more depth can reduce bias while increasing variability.
The practical comparison should therefore use data not used to fit the trees, such as a validation set or cross-validation folds. Compare the same performance measure across candidate depth limits, using a consistent fitting procedure and the same data partitions. Training error is useful for diagnosing fit, but choosing the depth solely because it has the smallest training error favors the most flexible candidate.
Worked Example: Choosing a Depth Using Cross-Validation
For a hypothetical boosted regression analysis, a practitioner compares maximum depths 1, 2, and 3. The training mean squared errors are 9, 5, and 2, respectively. The validation mean squared errors over five folds are:
| Maximum depth | Fold 1 | Fold 2 | Fold 3 | Fold 4 | Fold 5 |
|---|---|---|---|---|---|
| 1 | 10 | 12 | 11 | 9 | 13 |
| 2 | 8 | 9 | 8 | 10 | 9 |
| 3 | 7 | 10 | 12 | 9 | 14 |
Step 1: Average the validation errors. For depth 1, the mean is \((10+12+11+9+13)/5=55/5=11\). For depth 2, it is \((8+9+8+10+9)/5=44/5=8.8\). For depth 3, it is \((7+10+12+9+14)/5=52/5=10.4\).
Step 2: Compare candidates. Depth 2 has the lowest average validation mean squared error, 8.8. Although depth 3 has the lowest training error, its average validation error is 10.4, higher than the depth-two result. On this comparison, select depth 2.
Check and interpretation. The chosen depth is based on validation performance, not on the smallest training error or the largest available depth. The fold results also vary, so the comparison is evidence about this fitting exercise rather than a universal rule that depth 2 is best. The common mistake is to choose depth 3 because its training error is lowest.
Exam Traps and Practical Interpretation
- Confusing depth with the number of trees. A model with many depth-one trees is still a sum of one-predictor effects. A single depth-two tree has the capacity to make predictions based on combinations of two split conditions.
- Reading maximum depth as an interaction guarantee. A depth-three tree can use only one predictor repeatedly, or may have branches that stop early. Its maximum depth describes what is allowed, not what the fitted model necessarily uses.
- Equating a lower training error with better generalization. Deeper trees can lower training error while increasing sensitivity to the training data. Use validation evidence to assess the tradeoff.
- Ignoring the path structure. Interaction capacity is determined by the predictors appearing together along a path. A tree’s total number of splits is not the same as the number of predictors in an interaction.
- Changing depth and update count at once. If both the number of trees and their depth change, a performance difference cannot be attributed to either choice alone without a controlled comparison.
Check Your Understanding
For each problem, track the path structure as well as the numerical comparison.
- A boosted model uses trees with maximum depth 1, and every stump splits on either \(x_1\) or \(x_2\). Can their sum have a nonzero two-factor contrast? (Answer: No; the sum is additive in \(x_1\) and \(x_2\), so the two-factor contrast is zero.)
- A depth-four tree has a path that splits on \(x_1\), then \(x_2\), then \(x_1\) again, then \(x_3\). What is the maximum number of distinct predictors on this path, and what is the highest interaction order this path could support? (Answer: Three distinct predictors; at most a three-way interaction.)
- A current prediction for one observation is 40. The next tree predicts 15 for that observation and the shrinkage factor is \(\eta=0.2\). What is the updated prediction? (Answer: \(40+0.2(15)=43\).)
- Three candidate depths have validation mean squared errors across four folds of \((8,10,12,10)\), \((7,8,9,8)\), and \((6,11,12,11)\), respectively. Calculate each average and select the depth with the lowest average. (Answer: The averages are 10, 8, and 10; select the second candidate depth.)
- For binary predictors, a fitted prediction table has values \(f(0,0)=2\), \(f(0,1)=5\), \(f(1,0)=4\), and \(f(1,1)=10\). Calculate its two-factor contrast and state whether it can be represented by a sum of stumps splitting on these predictors alone. (Answer: \(10-4-5+2=3\); no, because a sum of those stumps has contrast zero.)