Why a Leaf Needs Its Own Loss-Minimizing Update
In boosting, a tree divides the predictor space into terminal regions, or leaves. The tree structure determines which observations share an update. Once those regions are fixed, each leaf still needs a numerical prediction. Choosing that number to minimize the selected loss is not always the same as averaging the responses or averaging pseudo-residuals.
The appropriate update depends on the loss, the scale on which the model is updated, and any weights used in the objective. The mean is the optimizer for squared error; the median is an optimizer for absolute error. For other losses, the update is found by minimizing the leaf’s objective, often with a one-dimensional line search.
The expression \(\gamma_j^*\) is a full leaf update. With shrinkage, the model’s actual change is scaled by \(0<\nu\leq 1\). If the update is additive on the response scale, the new prediction in the leaf is \(F_{m-1}(x_i)+\nu\gamma_j^*\). If the model uses a transformed scale, such as a log-rate, the addition occurs on that scale; transforming back to the response scale is a separate step.
Squared Error: The Mean Residual
Suppose the objective within a leaf is the unweighted sum of squared errors, \(\sum_{i\in R_j}(y_i-F_{m-1}(x_i)-\gamma)^2\). Define the current residual \(r_i=y_i-F_{m-1}(x_i)\). Then the leaf objective is \(\sum_{i\in R_j}(r_i-\gamma)^2\).
To justify the formula, differentiate the objective with respect to \(\gamma\). Its derivative is \(-2\sum_{i\in R_j}(r_i-\gamma)\); setting the derivative equal to zero gives \(\sum r_i-|R_j|\gamma=0\), hence the stated mean. The second derivative is \(2|R_j|>0\) for a nonempty leaf, so this stationary point is the unique minimum.
Worked Example: Squared-Error Update From Current Residuals
A regression tree places four hypothetical policy records in one leaf. Their observed outcomes are \(9,14,10,16\), while the current model predictions are \(7,15,8,16\). Find the full squared-error leaf update and the minimized sum of squared errors. Use learning rate \(\nu=0.4\) to find the increment actually added.
Method. Calculate residuals, average them to obtain the full update, and then apply the learning rate. This avoids treating the average response as the update when the current model is already nonzero.
The residuals are \(9-7=2\), \(14-15=-1\), \(10-8=2\), and \(16-16=0\). Their sum is \(2-1+2+0=3\), so the full leaf update is \(\gamma^*=3/4=0.75\).
The residuals after the full update are \(2-0.75=1.25\), \(-1-0.75=-1.75\), \(2-0.75=1.25\), and \(0-0.75=-0.75\). The minimized sum of squared errors is \(1.25^2+(-1.75)^2+1.25^2+(-0.75)^2=1.5625+3.0625+1.5625+0.5625=6.75\). Before the update it was \(2^2+(-1)^2+2^2+0^2=9\), so the loss has decreased.
With \(\nu=0.4\), the increment added is \(0.4(0.75)=0.30\). The model’s updated predictions in this leaf are therefore \(7.30,15.30,8.30,16.30\). The check is that every prediction changes by the same \(0.30\), as required for a constant additive leaf update.
Absolute Error: The Median Residual
For absolute-error loss, the leaf objective is \(\sum_{i\in R_j}|r_i-\gamma|\). A mean need not minimize this sum: large residuals can pull the mean away from the value that gives the smallest total absolute deviation. Instead, any median of the residuals is a minimizer.
The reason follows from how the objective changes as \(\gamma\) moves. Away from residual values, each residual below \(\gamma\) contributes slope \(+1\), while each residual above \(\gamma\) contributes slope \(-1\). The objective decreases while more than half the residuals are above \(\gamma\), and increases once more than half are below it. A median is precisely a point where neither side contains more than half the observations. At ties, the objective can be flat over an interval, which explains the range of minimizers for an even-sized leaf.
Worked Example: Absolute-Error Update With an Extreme Residual
Five hypothetical claim records share a leaf. Their observed outcomes are \(8,11,14,25,10\), and their current predictions are \(7,9,12,15,11\). Find the absolute-error minimizing update and compare its loss with the loss at the mean residual. Use learning rate \(\nu=0.5\) to calculate the applied increment.
Method. Compute and order the residuals, then use their median for absolute error. This avoids assuming that the squared-error mean formula applies to every loss.
The residuals are \(1,2,2,10,-1\). In increasing order they are \(-1,1,2,2,10\), so the median and full update are \(\gamma^*=2\). The total absolute error after this update is \(|1-2|+|2-2|+|2-2|+|10-2|+|-1-2|=1+0+0+8+3=12\).
The mean residual is \((1+2+2+10-1)/5=14/5=2.8\). Its total absolute error is \(|1-2.8|+|2-2.8|+|2-2.8|+|10-2.8|+|-1-2.8|=1.8+0.8+0.8+7.2+3.8=14.4\). Thus the median gives the smaller absolute-error objective.
With \(\nu=0.5\), the actual increment is \(0.5(2)=1\), not the full update of 2. The residuals after the shrunken increment are \(0,1,1,9,-2\), giving absolute error \(0+1+1+9+2=13\). This is less than the original total \(1+2+2+10+1=16\), though greater than the loss at the unshrunk minimizer, as expected.
Other Losses: Minimize the Leaf Objective Directly
For a general loss, there is no universal mean-or-median rule. The full update is the value of \(\gamma\) that minimizes the sum of the selected losses over the observations in that leaf. If the loss is differentiable and the minimum is an interior point, a candidate satisfies \[ \sum_{i\in R_j}\frac{\partial}{\partial \gamma}\ell\!\left(y_i,F_{m-1}(x_i)+\gamma\right)=0. \] This condition identifies stationary points, not automatically a global minimum. Convexity, a second-derivative check, or direct comparison of objective values is needed to establish that the selected point minimizes the objective. For losses without a convenient closed-form solution, a one-dimensional numerical line search can evaluate the objective over candidate values of \(\gamma\).
A useful actuarial example is a leaf modeling counts with a log-rate prediction. Suppose the counts \(Y_i\) are independent Poisson random variables with means \(\lambda_i=\exp(F_{m-1}(x_i)+\gamma)\) after a candidate update. The Poisson negative log-likelihood, omitting the term \(\log(Y_i!)\) that does not depend on the model, is \(\exp(F_{m-1}(x_i)+\gamma)-Y_i(F_{m-1}(x_i)+\gamma)\). The leaf update minimizes the sum of these terms. The update is additive on the log-rate scale, not on the count or rate scale.
Worked Example: Poisson Line Search on the Log-Rate Scale
Two hypothetical exposure cells in one leaf have observed claim counts \(0\) and \(4\). The current log-rate prediction is \(F_{m-1}=0\) for both cells, so each current rate is \(\exp(0)=1\). Assume independent Poisson counts and use the negative log-likelihood with the factorial terms omitted. Find the full minimizing update, verify it is a minimum, and calculate the increment for learning rate \(\nu=0.5\).
Method. Write the sum of the Poisson losses as a function of the shared log-rate update, differentiate, and solve. This is a loss-specific line search; the squared-error mean residual is not the relevant formula.
For a candidate update \(\gamma\), the loss is \(Q(\gamma)=\exp(\gamma)-0\gamma+\exp(\gamma)-4\gamma=2\exp(\gamma)-4\gamma\). Its first derivative is \(Q'(\gamma)=2\exp(\gamma)-4\). Setting this to zero gives \(\exp(\gamma)=2\), hence \(\gamma^*=\log 2\approx0.6931\). The second derivative is \(Q''(\gamma)=2\exp(\gamma)>0\) for every \(\gamma\), so this is the unique minimum.
At the minimum, \(Q(\log 2)=2(2)-4\log 2=4-2.7726\approx1.2274\). At the original prediction, \(Q(0)=2\), confirming that the full update lowers the objective. With learning rate \(0.5\), the increment on the log-rate scale is \(0.5\log 2\approx0.3466\), rounded to four decimal places. It is not \(0.5\).
The updated log-rate prediction is \(0.3466\), and the corresponding rate is \(\exp(0.5\log 2)=\sqrt{2}\approx1.4142\). The applied rate change is therefore multiplicative, not an addition of \(0.3466\) to the original rate of 1. This check confirms that shrinkage was applied on the model’s log scale.
Separate the Loss Minimizer From the Applied Update
The examples use different optimizers because their objectives differ. The squared-error example gave a mean residual; the absolute-error example gave a median residual; and the Poisson example required minimizing a likelihood on the log-rate scale. These are not competing versions of one formula. Each follows from a different loss and model scale.
| Leaf objective and scale | Full update | Common check |
|---|---|---|
| Unweighted squared error; additive response-scale update | Mean of current residuals | Confirm residuals are \(y_i-F_{m-1}(x_i)\) |
| Unweighted absolute error; additive response-scale update | Any median of current residuals | Check total absolute deviation, especially with an extreme residual |
| Other loss, including a likelihood on a transformed scale | Minimizer of the leaf-specific objective | Check the scale, derivative, and minimum condition |
For weighted losses, the formulas change accordingly: for example, squared error gives a weighted mean residual, while absolute error gives a weighted median. The weights must be part of the stated objective; they should not be added or ignored after calculating an unweighted optimizer. Likewise, if an objective includes a penalty on leaf values, minimizing the sum of observation-level losses alone is not sufficient—the penalty must also be included.
Common Exam Trap: Shrinkage Is Not the Leaf Prediction
A frequent error is to find the correct full update and then confuse the learning rate with either that update or the new prediction. In the Poisson example, the full update was \(\log 2\approx0.6931\). With learning rate \(0.5\), the added log-rate increment is \(0.5\log 2\approx0.3466\). Writing \(0.5\) as the increment is incorrect: the learning rate multiplies the full update; it does not replace it.
A second common error is to apply an additive update on the wrong scale. In the same example, adding \(0.3466\) to the original rate of 1 would give \(1.3466\), but the model’s prediction is a log-rate. The correct updated rate is \(\exp(0+0.3466)\approx1.4142\). Always identify whether the model adds on the response, log, or another scale before interpreting the numerical update.
Check Your Understanding
For each problem, identify the relevant leaf objective before choosing an update.
- A squared-error leaf has current residuals \(3,-1,2,0\). Find the full leaf update. (Answer: \(\gamma^*=1\).)
- An absolute-error leaf has residuals \(-3,0,1,4,4\). Find a minimizing update. (Answer: any median; \(\gamma^*=1\).)
- Two independent Poisson counts in a leaf are \(1\) and \(3\), and both current rates are 1. The model adds a shared update to log-rate. Find the full minimizing update. (Answer: \(\gamma^*=\log 2\approx0.6931\).)
- A weighted squared-error leaf has residuals \(2\) and \(5\), with respective weights \(3\) and \(1\). Find the full minimizing update using the weighted sum of squared errors. (Answer: \(\gamma^*=(3(2)+1(5))/(3+1)=2.75\).)
- A squared-error leaf has residuals \(4,0,-2\), and the learning rate is \(0.3\). Find both the full leaf update and the increment actually added. (Answer: \(\gamma^*=2/3\approx0.6667\); applied increment \(0.3(2/3)=0.2\).)