Logistic Loss Gradients For Binary Boosting

From a Binary Outcome to a Score Update

A boosted classifier does not usually add a tree directly to a probability. It adds a tree to a score, also called a logit or raw prediction, and converts the resulting score to a probability. The direction of that update matters: a tree should tend to increase scores for cases whose observed outcome is higher than their current predicted probability, and decrease scores for cases whose outcome is lower.

The negative gradient of binary logistic loss gives a precise measure of that direction and magnitude. These values are called pseudo-residuals. A regression tree fitted to them partitions the feature space into leaves whose updates move the current model toward lower logistic loss.

Definition: For observation $i$, let the binary outcome be $y_i\in\{0,1\}$, the current score be $F_i\in\mathbb{R}$, and the corresponding probability be $p_i=1/(1+\exp(-F_i))$. The binary logistic loss is $$ \ell(y_i,F_i)=-y_i\log(p_i)-(1-y_i)\log(1-p_i). $$ Its negative gradient with respect to the score is the pseudo-residual $r_i=y_i-p_i$.

Equivalently, the loss can be written directly in terms of the score:

\ell(y_i,F_i)=\log(1+\exp(F_i))-y_iF_i.

This form is convenient for differentiation and makes clear that the score is on the log-odds scale: $F_i=\log(p_i/(1-p_i))$. The probability $p_i$ is always between zero and one, while the score can be any real number.

Deriving the Pseudo-Residual

For a Bernoulli response with probability $p_i$, the probability model assigns probability $p_i$ to $y_i=1$ and $1-p_i$ to $y_i=0$. The negative log-likelihood for that observation is the logistic loss above. No special distributional approximation is needed to differentiate it.

Differentiate the score form of the loss.
Since the derivative of $\log(1+\exp(F_i))$ is $\exp(F_i)/(1+\exp(F_i))=p_i$, and the derivative of $-y_iF_i$ is $-y_i$, the gradient is $p_i-y_i$.

Take the negative gradient.
Gradient boosting fits a function to the direction of steepest decrease, which is the negative gradient: $r_i=-(p_i-y_i)=y_i-p_i$.

Interpret its sign.
If $y_i=1$, then $r_i=1-p_i\geq 0$, so the score should rise unless the predicted probability is already one. If $y_i=0$, then $r_i=-p_i\leq 0$, so the score should fall unless the predicted probability is already zero.

The residual is not the ordinary classification error $y_i-\mathbf{1}(p_i\geq 0.5)$. It is a continuous quantity that reflects both the outcome and the current model's confidence. For example, a positive case predicted with probability $0.51$ has residual $0.49$, while one predicted with probability $0.95$ has residual $0.05$. Both call for an increase, but the first has a stronger gradient signal.

How a Tree Uses These Values

At a boosting iteration, calculate one pseudo-residual for each training observation using the current score. Fit a regression tree with those residuals as its response. A leaf groups observations with similar predictor patterns; the fitted leaf value indicates the direction and, under the usual squared-error fit to the gradients, an average magnitude of the proposed score movement.

If tree $h_m$ assigns observation $i$ to leaf $j$, a simple gradient update is

F_i^{(m)}=F_i^{(m-1)}+\nu h_m(x_i),

where $\nu$ is the learning rate. Shrinkage scales the tree's proposed movement; it does not change how the pseudo-residuals were calculated. Depending on the boosting method, a leaf value may instead be chosen by minimizing the logistic loss within that leaf or by a Newton approximation. Those choices are related but not interchangeable: fitting a tree to negative gradients is one step, and selecting the final leaf increment is another.

Worked Example: Residuals Direct Two Leaf Updates

A hypothetical model currently assigns three accounts scores corresponding to probabilities $0.20$, $0.80$, and $0.50$. Their observed outcomes are $1,1,0$, respectively. A regression tree puts the first two accounts in one leaf and the third account in another. Fit leaf values as the average pseudo-residual in each leaf and use learning rate $\nu=0.40$. Calculate the updated probabilities and check the loss direction.

Method. Calculate $r_i=y_i-p_i$, average within each leaf, then add the learning-rate-scaled leaf value to the score. This follows the negative gradient and avoids treating a hard class prediction as a residual.

For the first account, $r_1=1-0.20=0.80$. For the second, $r_2=1-0.80=0.20$. For the third, $r_3=0-0.50=-0.50$. Thus the first leaf's fitted value is $(0.80+0.20)/2=0.50$; the second leaf's value is $-0.50$.

The score increments are $0.40(0.50)=0.20$ for each of the first two accounts and $0.40(-0.50)=-0.20$ for the third. To convert scores to probabilities, it is convenient to multiply the original odds by the exponential of the score increment.

For account 1, the initial odds are $0.20/0.80=0.25$. Updated odds are $0.25\exp(0.20)\approx0.3054$, so the new probability is $0.3054/(1+0.3054)\approx0.2339$. For account 2, the initial odds are $0.80/0.20=4$; updated odds are $4\exp(0.20)\approx4.8856$, giving probability $4.8856/(1+4.8856)\approx0.8300$. For account 3, the initial odds are $1$; updated odds are $\exp(-0.20)\approx0.8187$, giving probability $0.8187/(1+0.8187)\approx0.4502$.

The individual losses before the update are $-\log(0.20)\approx1.6094$, $-\log(0.80)\approx0.2231$, and $-\log(0.50)\approx0.6931$. After the update, they are approximately $-\log(0.2339)=1.4528$, $-\log(0.8300)=0.1862$, and $-\log(1-0.4502)=0.5981$. The total falls from approximately $2.5257$ to $2.2371$. This check is consistent with the update directions: probabilities for the two positive outcomes rose, while the probability for the negative outcome fell.

Leaf Increments: Gradient Fit and Loss Minimization

A regression tree fitted to pseudo-residuals by squared error gives each leaf the mean residual among the observations in that leaf. This is the least-squares constant approximation to those gradients. A different approach chooses a constant score increment $\gamma_j$ for each leaf $R_j$ to minimize the actual logistic loss there:

\gamma_j=\underset{\gamma}{\operatorname{argmin}}\; \sum_{i:x_i\in R_j}\ell(y_i,F_i+\gamma).

At a finite minimum, differentiation with respect to $\gamma$ gives the condition

\sum_{i:x_i\in R_j}\left[\frac{1}{1+\exp(-(F_i+\gamma_j))}-y_i\right]=0.

In words, the sum of the updated probabilities in the leaf must equal the sum of its observed outcomes. This follows because each updated probability is the derivative contribution from the loss, and a differentiable loss at an interior minimum has derivative zero. The condition is not generally solved by taking the mean pseudo-residual: the probabilities change as $\gamma$ changes.

A Newton step gives a useful local approximation. At the current scores, the gradient of the leaf loss is $-\sum r_i$, and its second derivative is $\sum p_i(1-p_i)$. A one-step Newton increment from zero is therefore

\gamma_j^{\text{Newton}}\approx \frac{\sum_{i:x_i\in R_j}(y_i-p_i)} {\sum_{i:x_i\in R_j}p_i(1-p_i)}.

This approximation uses curvature as well as direction. It is not the same as the average pseudo-residual used by a least-squares tree, and a finite Newton step need not equal the exact loss-minimizing increment.

Worked Example: Compare a Gradient Leaf Value With a Newton Step

A leaf contains two observations. The first has $y_1=1$ and current probability $p_1=0.25$; the second has $y_2=0$ and current probability $p_2=0.50$. Find the pseudo-residuals, their leaf average, the Newton increment, and the exact loss-minimizing common score increment.

Method. First calculate the gradients and curvature at the current scores. Then solve the zero-derivative equation for the exact common increment. This comparison avoids assuming that the mean residual is the exact logistic-loss minimizer.

The residuals are $r_1=1-0.25=0.75$ and $r_2=0-0.50=-0.50$. Their sum is $0.25$, and their mean is $0.25/2=0.125$. Thus a least-squares tree's constant leaf fit is $0.125$.

The curvature terms are $p_1(1-p_1)=0.25(0.75)=0.1875$ and $p_2(1-p_2)=0.50(0.50)=0.25$. Their sum is $0.4375$. The Newton increment is $0.25/0.4375=4/7\approx0.5714$.

For the exact increment, the initial odds are $1/3$ and $1$. Let $t=\exp(\gamma)$. After a common score increase, the probabilities are $t/(3+t)$ and $t/(1+t)$. The zero-derivative condition requires their sum to equal $y_1+y_2=1$:

\frac{t}{3+t}+\frac{t}{1+t}=1 \quad\Longrightarrow\quad 2t+2t^2=t^2+4t+3 \quad\Longrightarrow\quad t^2=3.

Since $t=\exp(\gamma)>0$, $t=\sqrt{3}$ and $\gamma=\log(\sqrt{3})\approx0.5493$. The updated probabilities are $p_1'=\sqrt{3}/(3+\sqrt{3})\approx0.3660$ and $p_2'=\sqrt{3}/(1+\sqrt{3})\approx0.6340$. Their sum is $1.0000$, checking the derivative condition.

Before the update, total loss is $-\log(0.25)-\log(1-0.50)=\log(4)+\log(2)\approx2.0794$. Afterward, both the positive-case probability and the negative-case complement equal $1/(1+\sqrt{3})\approx0.3660$, so total loss is $-2\log(0.3660)\approx2.0101$. The exact increment, Newton increment, and mean residual differ, as expected; each is based on a different leaf-update rule.

Checking Signs and Probabilities

The score-to-probability transformation provides a direct check on an update. If a score changes by $\Delta$, its odds are multiplied by $\exp(\Delta)$. A positive increment increases the probability; a negative increment decreases it. The outcome determines whether that direction tends to reduce the individual loss.

Worked Example: Check a Small Update Against Logistic Loss

A positive case has current probability $0.80$, so its score is $\log(4)$. Its pseudo-residual is $1-0.80=0.20$. Suppose a leaf rule uses half of this residual as the score increment. Calculate the new probability and loss. Then check what happens if the increment has the wrong sign.

Method. Apply the stated increment to the score, transform using the logistic function, and evaluate the positive-case loss $-\log(p)$. This avoids estimating the new probability by an inaccurate mental approximation.

The proposed increment is $0.5(0.20)=0.10$, so the new score is $\log(4)+0.10$. The new odds are $4\exp(0.10)\approx4.4207$, and the new probability is $4.4207/(1+4.4207)\approx0.8155$. The loss changes from $-\log(0.80)\approx0.2231$ to $-\log(0.8155)\approx0.2039$, a decrease.

With the wrong-sign increment $-0.10$, the new odds are $4\exp(-0.10)\approx3.6193$, and the probability is $3.6193/(1+3.6193)\approx0.7836$. The loss is $-\log(0.7836)\approx0.2439$, an increase relative to $0.2231$.

Common trap. Substituting $\log(4)+0.10$ into the logistic function gives approximately $0.8155$, not $0.8182$; its loss is approximately $0.2039$, not $0.2007$. Similarly, $\log(4)-0.10$ gives approximately $0.7836$, not $0.7750$, and its loss is approximately $0.2439$, not $0.2549$. The reliable check is to multiply odds by $\exp(\Delta)$ and then calculate the loss from the resulting probability.

For a negative case, the relevant loss is $-\log(1-p)$, not $-\log(p)$. This distinction is a frequent arithmetic trap. More generally, the full expression $-y\log(p)-(1-y)\log(1-p)$ automatically selects the correct term for either binary outcome.

Key takeaway: At each boosting iteration, calculate $r_i=y_i-p_i$ from the current probability, fit a tree to those pseudo-residuals, and add a suitably scaled leaf update to the score. Check that positive residuals move scores upward and negative residuals move them downward. A gradient-fit leaf value, a Newton step, and an exact loss-minimizing increment are distinct methods.

Check Your Understanding

For each problem, use the current scores or probabilities stated; do not substitute hard class labels for probabilities.

A binary response is $y=1$, and the current score is $F=0$. Calculate the current probability, the negative gradient, and the direction of a score update. (Answer: $p=0.5000$, pseudo-residual $0.5000$, score should increase.)
A case has $y=0$, current probability $0.60$, and a score increment of $-0.15$. Multiply the initial odds by $\exp(-0.15)$ to find the new probability. (Answer: approximately $0.5635$.)
A leaf contains three cases with outcomes $(1,0,1)$ and probabilities $(0.40,0.20,0.70)$. Find the pseudo-residuals, their mean, and the score increment with learning rate $0.30$ when the leaf prediction is their mean. (Answer: residuals $(0.60,-0.20,0.30)$, mean $0.2333$, increment $0.0700$.)
A leaf contains two cases with outcomes $(1,0)$ and probabilities $(0.20,0.50)$. Calculate the one-step Newton leaf increment using the current residual sum and curvature sum. (Answer: $0.30/(0.16+0.25)=0.7317$, approximately.)
A positive case has current probability $0.80$ and receives a score increment of $0.10$. Calculate the updated probability and positive-case loss to four decimal places, and state whether the loss fell from its starting value. (Answer: updated probability $0.8155$, updated loss $0.2039$, down from $0.2231$.)

What You'll Learn