Logistic Gradient Boosting And Class Scores

Logistic Scores as Class Scores

In binary gradient boosting, a weak learner does not directly need to predict a class label. Instead, it contributes an increment to a real-valued score, and that score is converted into a probability. The connection is especially useful for interpreting a boosting model: the score records the log-odds of the positive class, while its negative gradient supplies the targets that the next learner attempts to fit.

Use labels $y_i\in\{0,1\}$, where $1$ denotes the positive class and $0$ the negative class. For an input $x$, let $F(x)$ be the score and let $\sigma(t)=1/(1+\exp(-t))$ be the logistic function. The model assigns positive-class probability $p(x)=\sigma(F(x))$. This tutorial uses the Bernoulli negative log-likelihood, also called binary logistic loss.

Definition: For a binary label $y\in\{0,1\}$ and real-valued score $F$, the logistic loss is $$ \ell(y,F)=\log(1+\exp(F))-yF. $$ The associated positive-class probability is $p=\sigma(F)$. Thus $F$ is a class score measured on the log-odds scale, not a probability itself.

For $y=1$, the loss is $\log(1+\exp(-F))$; for $y=0$, it is $\log(1+\exp(F))$. Both are small when the score supports the observed class and grow when the score strongly contradicts it. The score's sign determines the predicted class under a threshold of one half: $F>0$ implies $p>1/2$, and $F<0$ implies $p<1/2$.

Why the Score Is a Log-Odds

The probability-score relationship can be inverted directly. Since $p=\exp(F)/(1+\exp(F))$, its complement is $1-p=1/(1+\exp(F))$. Their ratio is therefore $\exp(F)$, so taking logarithms recovers the score.

Result: For $0<p<1$, the score corresponding to positive-class probability $p$ is the log-odds $$ F=\log\left(\frac{p}{1-p}\right). $$ Conversely, $p=\sigma(F)$. Increasing $F$ by $c$ multiplies the odds $p/(1-p)$ by $\exp(c)$.

A score of zero means equal odds and probability one half. A score of $\log(3)$ means odds of three to one and probability $3/4$; a score of $-\log(3)$ means odds of one to three and probability $1/4$. This interpretation is exact for the chosen zero-one convention.

Worked Example: Turning Probabilities Into Scores

Suppose a model assigns positive-class probability $0.8$ to one observation and $0.25$ to another. Find both scores and interpret them.

For $p=0.8$, the odds are $0.8/(1-0.8)=0.8/0.2=4$. The score is $F=\log(4)$, approximately $1.386$. The positive class is four times as likely as the negative class under these fitted probabilities.

For $p=0.25$, the odds are $0.25/0.75=1/3$. The score is $F=\log(1/3)=-\log(3)$, approximately $-1.099$. The negative class is three times as likely as the positive class.

Checking the inverse transformation, $\sigma(\log(4))=4/(1+4)=0.8$, while $\sigma(-\log(3))=(1/3)/(1+1/3)=0.25$.

Negative-Gradient Targets

For training observations $(x_i,y_i)$, a current score function $F$ gives fitted probabilities $p_i=\sigma(F(x_i))$. Functional gradient boosting forms a pseudo-residual at each observation by taking the negative derivative of the loss with respect to its current score. The next regression learner is fitted to these targets, so that its output approximates a direction in which the logistic loss decreases.

Definition: At boosting round $m$, the logistic negative-gradient target for observation $i$ is $$ r_{im}=-\left.\frac{\partial \ell(y_i,F(x_i))}{\partial F(x_i)}\right|_{F=F_{m-1}} =y_i-p_{i,m-1}, $$ where $p_{i,m-1}=\sigma(F_{m-1}(x_i))$.

The derivative follows by differentiating $\log(1+\exp(F))-yF$: its first term has derivative $\exp(F)/(1+\exp(F))=\sigma(F)$, and its second term has derivative $-y$. Thus the derivative is $p-y$, and its negative is $y-p$. A positive target asks for a higher score; a negative target asks for a lower one. Its magnitude is the discrepancy between the observed zero-one label and the current fitted probability.

Result: For each fixed label $y\in\{0,1\}$, the logistic loss is differentiable and convex in $F$, with $$ \frac{\partial \ell(y,F)}{\partial F}=\sigma(F)-y, \qquad \frac{\partial^2 \ell(y,F)}{\partial F^2}=\sigma(F)\{1-\sigma(F)\}>0. $$ Therefore $y-\sigma(F)$ is the exact negative-gradient target.

To verify the second derivative, differentiate $\sigma(F)$: $\sigma'(F)=\sigma(F)\{1-\sigma(F)\}$. For every finite $F$, the logistic probability is strictly between zero and one, making this derivative positive. The loss is consequently strictly convex as a function of the score. This calculation establishes the gradient targets; it does not mean that a fitted weak learner will match every target exactly.

Worked Example: Computing Logistic Pseudo-Residuals

Three observations have labels $y=(1,0,1)$ and current fitted probabilities $p=(0.2,0.7,0.5)$. Calculate the negative-gradient targets.

For the first observation, $r_1=y_1-p_1=1-0.2=0.8$. The model underestimates the probability of its positive label, so the target points toward increasing its score.

For the second, $r_2=0-0.7=-0.7$. The model assigns too much probability to the positive class for a negative observation, so the target points toward decreasing its score.

For the third, $r_3=1-0.5=0.5$. The current probability is uncertain, and the positive label gives a positive target. In order, the pseudo-residuals are $(0.8,-0.7,0.5)$.

Choosing a Constant Starting Score

A common initialization uses one constant score for every training observation. The best constant under the empirical logistic loss is determined by the observed proportion of positive labels. This provides a principled starting probability rather than requiring the model to begin at an arbitrary class score.

Result: Suppose the training set has $n$ labels in $\{0,1\}$, with $n_1=\sum_{i=1}^n y_i$ positive labels and $0<n_1<n$. The unique constant score minimizing average logistic loss is $$ F_0=\log\left(\frac{n_1}{n-n_1}\right), $$ and its fitted probability is $n_1/n$.

To prove this, write the average loss for a constant $F$ as $$ L(F)=\frac{1}{n}\sum_{i=1}^n\{\log(1+\exp(F))-y_iF\} =\log(1+\exp(F))-\frac{n_1}{n}F. $$ Differentiation gives $L'(F)=\sigma(F)-n_1/n$. Since $\sigma$ is strictly increasing from zero to one, $L'(F)=0$ has exactly one solution, namely $\sigma(F)=n_1/n$. Inverting the logistic function gives the displayed $F_0$; strict convexity makes it the unique minimizer. $\square$

Worked Example: Initializing From Class Prevalence

A training sample contains 10 observations, of which 6 have label 1 and 4 have label 0. Find the optimal constant initial score and verify the gradient balance.

The positive-label proportion is $6/10=0.6$. The best constant score is the logit of this proportion: $F_0=\log(0.6/0.4)=\log(1.5)$, approximately $0.4055$. Transforming back gives $\sigma(F_0)=1.5/(1+1.5)=0.6$.

At this initialization, each positive observation has target $1-0.6=0.4$, and each negative observation has target $0-0.6=-0.6$. The sum of all ten targets is $6(0.4)+4(-0.6)=2.4-2.4=0$, as required at a constant-loss minimum.

How Additive Updates Change Probabilities

At round $m$, gradient boosting adds a fitted function $g_m(x)$ to the current score, usually with a learning-rate multiplier $\nu>0$: $F_m(x)=F_{m-1}(x)+\nu g_m(x)$. At a particular input, write the score increment as $\delta=\nu g_m(x)$. The update is additive on the log-odds scale, but its effect on probability is nonlinear.

Result: If the current probability is $p=\sigma(F)$ and the score changes to $F+\delta$, then the new probability is $$ p_{\mathrm{new}}=\sigma(F+\delta) =\frac{p\exp(\delta)}{1-p+p\exp(\delta)}. $$ Equivalently, the new odds equal the old odds multiplied by $\exp(\delta)$.

For the derivation, the old odds satisfy $p/(1-p)=\exp(F)$. The updated odds are $\exp(F+\delta)=\exp(\delta)\exp(F)$, so they equal $\exp(\delta)p/(1-p)$. Solving $p_{\mathrm{new}}/(1-p_{\mathrm{new}})=\exp(\delta)p/(1-p)$ for $p_{\mathrm{new}}$ gives the stated expression. In particular, a positive increment raises the probability, a negative increment lowers it, and a zero increment leaves it unchanged. $\square$

Worked Example: Following Two Boosting Updates

Consider two observations, one with label 1 and one with label 0. Both start at score zero, so each has probability $0.5$. A fitted weak learner returns $g_1=0.4$ for the positive observation and $g_1=-0.4$ for the negative observation. Use learning rate $\nu=0.5$, then suppose the next fitted learner returns $g_2=0.3$ and $-0.3$, respectively, with the same learning rate. Trace the probabilities.

Initially, both scores are zero. After the first update, the score increments are $0.5(0.4)=0.2$ and $0.5(-0.4)=-0.2$. The probabilities become $\sigma(0.2)\approx0.5498$ for the positive observation and $\sigma(-0.2)\approx0.4502$ for the negative observation.

The new negative-gradient targets are $1-0.5498=0.4502$ and $0-0.4502=-0.4502$. The second learner's stated outputs are fitted approximations to these targets. Its score increments are $0.5(0.3)=0.15$ and $0.5(-0.3)=-0.15$, so the updated scores are $0.2+0.15=0.35$ and $-0.2-0.15=-0.35$.

The resulting probabilities are $\sigma(0.35)\approx0.5866$ and $\sigma(-0.35)\approx0.4134$. Each update moves the probability in the direction indicated by the label, but neither the target nor the learner output is itself the updated probability. The probabilities are obtained only after summing score increments and applying $\sigma$.

Interpretation and a Common Pitfall

Logistic boosting separates three quantities that are easy to conflate: the label $y_i$, the negative-gradient target $y_i-p_i$, and the score $F(x_i)$. The label is zero or one; the target is a real-valued residual; and the score is a log-odds value. A learner fitted to the targets contributes an increment to the score, after which the logistic transformation gives a probability.

A common mistake is to treat a score as though it were already a probability, or to add a score increment directly to the probability. For example, if $p=0.5$ and the score increment is $0.2$, the new probability is not $0.5+0.2=0.7$. The old score is zero, so the correct update gives $\sigma(0.2)\approx0.5498$. The addition happens on the log-odds scale, not on the probability scale.

Another source of errors is changing label conventions without changing the loss convention. For labels $0$ and $1$, the Bernoulli logistic loss used here has negative gradient $y-p$, and its score is the full log-odds. Signed labels in $\{-1,+1\}$ are often paired with a different-looking margin loss; the score scaling and gradient formula must be derived from that stated loss rather than copied without adjustment. Under the convention used here, the interpretation is unambiguous: the score is log-odds, and the pseudo-residual is observed label minus fitted probability.

Check Your Understanding

Use the zero-one label and logistic-loss convention developed above.

A fitted probability is $0.9$. What is its score, and what are the corresponding odds?
For a negative observation with fitted probability $0.35$, calculate its negative-gradient target and explain its sign.
A training sample has 8 positive labels and 12 negative labels. Find the optimal constant initial probability and score.
If a score increases by $\log(2)$, by what factor do the odds change? Does the probability increase by a fixed amount?
Starting from score $F=0.4$, a learner contributes an increment of $-0.1$. State the new score and express the new probability using the logistic function.