Why a Tree Leaf Needs Its Own Prediction
In gradient boosting, a tree partitions the observations into terminal nodes, or leaves. Each leaf assigns the same proposed update to every observation it contains. Once the tree structure is fixed, choosing a leaf value is an optimization problem: select the constant update that gives the smallest loss for the observations in that leaf.
The best value depends on the loss function. For squared error, it is the mean of the current residuals in the leaf. For binary logistic loss, it generally does not equal the mean negative gradient; instead, the exact update solves a nonlinear equation. That distinction matters because the tree structure and the value assigned to each leaf are separate parts of the fit.
Throughout, assume observations have equal weight, the tree structure is already fixed, and there is no penalty on the magnitude of \(\gamma\). The result is the unshrunk minimizing update for that leaf. If the objective includes observation weights, a penalty, or a constraint on the leaf value, the optimization problem changes.
Squared-Error Loss: The Mean Residual
For a quantitative response \(y_i\), define the current residual as \(r_i=y_i-F_i\). Use the squared-error loss with a factor of one-half:
The factor of one-half does not change which value minimizes the loss; it makes the derivative simpler. Differentiating with respect to \(\gamma\) and setting the derivative to zero gives the minimizing value. The second derivative is positive whenever the leaf contains at least one observation, so this stationary point is the unique minimum.
Why the mean appears. The first-order condition says that the residuals after the update must sum to zero: \(\sum_{i\in R}(r_i-\widehat{\gamma}_R)=0\). The only constant whose subtraction makes that sum zero is the arithmetic mean of the original residuals. Also, \(d^2L_R/d\gamma^2=|R|>0\), confirming that the critical point minimizes the loss rather than maximizes it.
Worked Example: Minimizing Squared Error in a Leaf
A hypothetical pricing model has four training observations in one leaf. Their responses and current predictions produce residuals \(3,-1,2,0\). Find the optimal leaf update and verify its effect on the loss. Use one-half squared error.
Method. The leaf value is the mean residual. The residual sum is \(3+(-1)+2+0=4\), and the leaf has four observations, so
Before the update, the loss is
After adding \(1\) to each current prediction, the residuals become \(3-1=2\), \(-1-1=-2\), \(2-1=1\), and \(0-1=-1\). Thus,
The updated residuals sum to \(2-2+1-1=0\), as required by the first-order condition. The loss falls from \(7\) to \(5\). This check confirms the calculation; the common mistake it avoids is using the sum of residuals, \(4\), as the leaf value instead of dividing by the number of observations.
Logistic Loss: Solve for the Score Update
For binary outcomes, let \(y_i\in\{0,1\}\), and let \(F_i\) be the current raw score. The score corresponds to a probability through the logistic function. Define the probability after a candidate leaf update by \(p_i(\gamma)\).
The binary logistic loss for observation \(i\), evaluated at score \(F_i+\gamma\), is
The leaf objective is \(L_R(\gamma)=\sum_{i\in R}\ell_i(\gamma)\). Differentiating each contribution with respect to the shared update gives \(p_i(\gamma)-y_i\). Therefore, the exact optimal update must satisfy the following equation:
In words, after the update, the sum of the fitted probabilities in the leaf must equal the number of positive outcomes in that leaf. The equation is usually nonlinear, so a numerical root-finding method may be needed when the starting scores differ.
A useful approximation comes from expanding the objective to second order around \(\gamma=0\). The first derivative at zero is \(\sum_{i\in R}(p_i(0)-y_i)\), and the second derivative is \(\sum_{i\in R}p_i(0)(1-p_i(0))\). Minimizing this quadratic approximation gives a Newton step:
This step is exact only when it happens to solve the original first-order condition. In general, it minimizes a local quadratic approximation, not the full logistic objective. The denominator adjusts for the curvature of logistic loss; simply averaging \(y_i-p_i(0)\) omits that curvature.
Worked Example: Exact Logistic Update From a Common Starting Score
A hypothetical binary underwriting model has four observations in one leaf. All four current scores are \(F_i=0\), and the outcomes are \(1,1,1,0\). Find the exact logistic leaf update, compare it with one Newton step from zero, and check the loss. Use the logistic loss defined above.
Exact method. Since all starting scores are zero, \(p_i(0)=1/(1+\exp(0))=0.5\). After the common update, all four probabilities are \(p(\gamma)\). The first-order condition requires their sum to equal the number of positive outcomes, which is three:
The score corresponding to probability \(3/4\) is the log-odds \(\log((3/4)/(1/4))=\log 3\). Because the starting score is zero, the exact update is \(\widehat{\gamma}_R=\log 3\), approximately \(1.0986\).
Newton comparison. At zero, the numerator of the Newton step is \(3-4(0.5)=1\). The denominator is \(4(0.5)(0.5)=1\). Thus \(\gamma_{\mathrm{Newton}}=1/1=1\), which is close to but not equal to \(\log 3\).
Loss check. Before the update, each observation contributes \(\log 2\), so \(L_R(0)=4\log 2\), approximately \(2.7726\). At the exact update, each positive outcome contributes \(-\log(3/4)=\log(4/3)\), and the one negative outcome contributes \(-\log(1/4)=\log 4\). Therefore,
The exact update reduces the loss, and its derivative is \(4(3/4)-3=0\). The Newton value is an approximation from the initial score, not the solution to that exact derivative equation.
Worked Example: Exact Update When Starting Scores Differ
A second hypothetical binary model has five observations in a leaf. Two have current probability \(1/3\), current score \(-\log 2\), and outcomes \(1,1\). Three have current probability \(1/2\), current score \(0\), and outcomes \(0,0,1\). Find the exact update and compare it with the Newton step.
Exact method. Test \(\gamma=\log 2\) in the first-order condition. For each of the first two observations, the starting odds are \(1/2\). Adding \(\log 2\) doubles the odds to \(1\), giving updated probability \(1/2\). For each of the other three observations, the starting odds are \(1\); doubling them gives odds \(2\) and updated probability \(2/3\). The sum of updated probabilities is
There are three positive outcomes, so the first-order condition is satisfied. Since this leaf contains both outcome classes and the minimizer is unique, \(\widehat{\gamma}_R=\log 2\), approximately \(0.6931\).
Newton comparison. The starting probability sum is \(2(1/3)+3(1/2)=2/3+3/2=13/6\). The Newton numerator is \(3-13/6=5/6\). Its denominator is
Consequently, \(\gamma_{\mathrm{Newton}}=(5/6)/(43/36)=30/43\), approximately \(0.6977\). This differs slightly from \(\log 2\); substitution into the exact first-order condition, rather than closeness to the Newton step, determines whether an update is the exact optimum.
What the Comparison Means in Practice
| Loss | Condition for the leaf optimum | Interpretation |
|---|---|---|
| One-half squared error | \(\widehat{\gamma}_R=|R|^{-1}\sum_{i\in R}(y_i-F_i)\) | The update is the mean current residual. |
| Binary logistic loss | \(\sum_{i\in R}p_i(\widehat{\gamma}_R)=\sum_{i\in R}y_i\) | The updated probability sum matches the positive-outcome count. |
For squared error, the objective is quadratic in the leaf value, so its minimizer is available directly. For logistic loss, the scores enter through the sigmoid function, so an exact update generally requires solving the first-order condition. When all starting scores in the leaf are equal, the solution can be found by setting the updated probability equal to the observed positive fraction and converting that probability to log-odds. When scores differ, the probability sum must be solved using the individual starting scores.
A common exam and implementation trap is to treat the negative gradient as if it were always the optimal leaf value. For squared error, the negative gradients are the residuals, and their mean is indeed the exact leaf update. For logistic loss, the negative gradients are \(y_i-p_i(0)\). Their sum identifies the direction of improvement at zero, but does not by itself determine the minimizing update.
For example, in the preceding four-observation logistic example, the mean negative gradient is \((0.5+0.5+0.5-0.5)/4=0.25\). Calling \(0.25\) the exact leaf update is incorrect. At \(\gamma=0.25\), every updated probability is \(1/(1+\exp(-0.25))\), approximately \(0.5622\). The derivative of the total loss is therefore \(4(0.5622)-3\), approximately \(-0.7513\), which is not zero. The exact update is \(\log 3\), while the Newton step is \(1\). The three quantities answer different questions: the mean negative gradient describes a simple tree-fitting target, the Newton step optimizes a local quadratic approximation, and the exact update minimizes the logistic objective over the leaf value.
The key discipline is to write down the objective for the constant leaf value and differentiate that objective. This makes clear whether a closed-form mean is justified, whether a nonlinear equation must be solved, and whether a proposed update is exact or approximate.
Check Your Understanding
For each problem, identify the relevant leaf optimization and check that your final value satisfies the stated condition.
- A squared-error leaf has residuals \(2,-1,4,1\). Find the minimizing update and the sum of squared residuals after the update. (Answer: \(\widehat{\gamma}_R=1.5\); updated residual sum of squares \(=13\).)
- A squared-error leaf has residuals \(0,2,4,6,8\). Find the minimizing update and compare the sum of squared residuals before and after. (Answer: \(\widehat{\gamma}_R=4\); sum of squares falls from \(120\) to \(40\).)
- Four observations in a logistic leaf all have current probability \(1/4\). Their outcomes are \(1,1,1,0\). Find the exact update by first finding the required updated probability, then changing from starting log-odds to target log-odds. (Answer: target probability \(3/4\); \(\widehat{\gamma}_R=\log 9\approx2.1972\).)
- A logistic leaf has four observations, all with current probability \(1/2\), and outcomes \(1,1,1,0\). Calculate one Newton step and the exact update. State why they need not agree. (Answer: Newton step \(=1\); exact update \(=\log 3\approx1.0986\), because Newton minimizes the local quadratic approximation.)
- A logistic leaf has two observations with current probability \(1/2\), two with current probability \(1/5\), and exactly two positive outcomes. Find an exact update by testing \(\gamma=\log 2\), and calculate the Newton step from zero. (Answer: \(\widehat{\gamma}_R=\log 2\approx0.6931\), since the updated probabilities sum to \(2(2/3)+2(1/3)=2\); Newton step \(=(2-1.4)/(2(0.25)+2(0.16))=0.6/0.82\approx0.7317\).)