Pseudo-Residuals Under Nonquadratic Losses

Why Boosting Uses Pseudo-Residuals

For squared-error boosting, the negative gradient with respect to a model’s current prediction is the familiar response residual. That equality depends on the particular loss function. With another loss, the direction that most rapidly reduces loss can be different from the raw difference between the observed response and the current prediction. Boosting uses this loss-based direction to decide what signal a new learner should approximate.

This distinction matters when the outcome is resistant to large errors or is binary. Under absolute-error loss, the gradient depends on whether a prediction is below or above the observation, not on how far away it is. Under logistic loss, it depends on the current predicted probability as well as the observed class. In each case, calculating the pseudo-residual requires knowing which score and loss convention is being used.

Definition: At boosting iteration $m$, let $F_i^{(m-1)}$ be the current model score for observation $i$, and let $\ell(y_i,F_i^{(m-1)})$ be its loss. The pseudo-residual is the negative derivative of that loss with respect to the current score: $$ r_i^{(m)} = -\left. \frac{\partial \ell(y_i,F)}{\partial F} \right|_{F=F_i^{(m-1)}}. $$ It is a direction of local loss reduction in score space. It is not, in general, the raw response residual $y_i-F_i^{(m-1)}$.

The calculation is observation-by-observation: hold the observed response fixed, differentiate the chosen loss with respect to the model score, then take the negative and evaluate at the current score. A tree-based boosting procedure may use these values as a response for the next learner. The calculation of the pseudo-residual itself does not determine how a tree’s terminal predictions are subsequently chosen.

Absolute-Error Loss

For a quantitative response $y_i$ and a real-valued score $F$, define the absolute-error loss by $\ell(y_i,F)=|y_i-F|$. The raw response residual is $y_i-F$. Away from an exact fit, differentiating the loss gives a value determined by the sign of that residual.

\ell(y_i,F)=|y_i-F|, \qquad -\frac{\partial \ell(y_i,F)}{\partial F} = \begin{cases} +1, & y_i>F,\\ -1, & y_i<F. \end{cases}

For example, if $y_i>F$, then $|y_i-F|=y_i-F$, whose derivative with respect to $F$ is $-1$. Its negative derivative is $+1$: increasing the score locally reduces the loss. If $y_i<F$, then $|y_i-F|=F-y_i$, whose derivative is $+1$, so the negative derivative is $-1$. The magnitude of the raw residual does not change either result.

At $F=y_i$, absolute value has a corner, so its ordinary derivative does not exist. The subgradient of $|y_i-F|$ with respect to $F$ at that point is the interval $[-1,1]$; consequently the negative subgradient is also $[-1,1]$. A common convention is to assign a pseudo-residual of zero at an exact fit. That is a valid subgradient choice, but it is a convention at the nondifferentiable point, not a derivative value obtained by ordinary differentiation.

Worked Example: Absolute-Error Pseudo-Residuals

A hypothetical reserve-estimation model has current scores $F_i$ for three segments, and the observed responses $y_i$ are 10, 7, and 13. The respective scores are 8, 9, and 13. Calculate the raw residuals and the absolute-error pseudo-residuals, using zero at an exact fit.

Method. First calculate each raw residual $y_i-F_i$. For a positive residual, the absolute-error negative gradient is $+1$; for a negative residual, it is $-1$. At zero, use the stated zero subgradient convention.

For segment 1, $y_1-F_1=10-8=2$. The score is below the response, so the pseudo-residual is $+1$.

For segment 2, $y_2-F_2=7-9=-2$. The score is above the response, so the pseudo-residual is $-1$.

For segment 3, $y_3-F_3=13-13=0$. This is the nondifferentiable point; under the stated convention, the pseudo-residual is $0$.

The raw residuals are therefore $(2,-2,0)$, while the pseudo-residuals are $(1,-1,0)$. The first two raw residuals have magnitude 2, but the corresponding pseudo-residuals have magnitude 1. This is precisely the difference between an error’s size and the local gradient direction. The check is that the pseudo-residual is positive when the score must increase, negative when it must decrease, and zero here by convention at an exact fit.

Logistic Loss for Binary Outcomes

For binary classification, the score is commonly an unrestricted real number rather than a probability. With zero-one labels $y_i\in\{0,1\}$, take $F$ to be a logit score and define the logistic probability by $$ p(F)=\frac{1}{1+\exp(-F)}. $$ The corresponding per-observation logistic loss, also called binary cross-entropy, is $$ \ell(y_i,F)=-y_i\log p(F)-(1-y_i)\log(1-p(F)). $$ Here the logarithm is natural. The score $F$ can take any real value, while $p(F)$ lies strictly between zero and one.

The negative gradient has a concise form:

-\frac{\partial \ell(y_i,F)}{\partial F}=y_i-p(F).

Thus for a positive observation the pseudo-residual is $1-p(F)$, and for a negative observation it is $-p(F)$. These are not usually $y_i-F$: $F$ is a logit, not a fitted response probability. Even if the fitted probability is reported, the pseudo-residual is the observed label minus that probability, not the label minus the logit.

Justification: Since $p(F)=1/(1+\exp(-F))$, differentiation gives $dp/dF=p(1-p)$. Differentiating the loss first with respect to $p$ gives $$ \frac{\partial \ell}{\partial p} =-\frac{y_i}{p}+\frac{1-y_i}{1-p} =\frac{p-y_i}{p(1-p)}. $$ Applying the chain rule, $$ \frac{\partial \ell}{\partial F} = \frac{\partial \ell}{\partial p}\frac{dp}{dF} = \frac{p-y_i}{p(1-p)}p(1-p) =p-y_i. $$ Taking the negative yields $y_i-p$, as claimed.

A second common convention encodes the classes as $y_i\in\{-1,+1\}$. With score $F$, use the margin-based logistic loss $\ell(y_i,F)=\log(1+\exp(-y_iF))$. Differentiation gives $$ -\frac{\partial \ell(y_i,F)}{\partial F} = \frac{y_i}{1+\exp(y_iF)}. $$ This convention and the zero-one-label convention express the same logistic modeling idea when their labels and score are matched consistently. Their formulas look different because the labels are encoded differently. Do not insert a zero-one label into the minus-one-plus-one formula, or vice versa.

Worked Example: Logistic Pseudo-Residuals on the Logit Scale

A hypothetical model scores three binary claims with zero-one labels $y=(1,0,1)$. Its current logits are $F=(\log 3,-\log 4,0)$. Calculate the fitted probabilities, the pseudo-residuals, and the raw label-minus-logit differences.

Method. Convert each logit to a probability using $p=1/(1+\exp(-F))$, then calculate $y-p$. The logarithms are chosen to give exact probabilities.

For observation 1, $F_1=\log 3$, so $\exp(-F_1)=1/3$ and $p_1=1/(1+1/3)=3/4=0.75$. Its pseudo-residual is $y_1-p_1=1-0.75=0.25$. Its raw label-minus-logit difference is $1-\log 3$, approximately $-0.0986$.

For observation 2, $F_2=-\log 4$, so $\exp(-F_2)=4$ and $p_2=1/(1+4)=1/5=0.20$. Its pseudo-residual is $y_2-p_2=0-0.20=-0.20$. Its raw label-minus-logit difference is $0-(-\log 4)=\log 4$, approximately $1.3863$.

For observation 3, $F_3=0$, so $p_3=1/(1+1)=0.50$. Its pseudo-residual is $y_3-p_3=1-0.50=0.50$. Its raw label-minus-logit difference is $1-0=1$.

The fitted probabilities are $(0.75,0.20,0.50)$ and the pseudo-residuals are $(0.25,-0.20,0.50)$. The raw label-minus-logit differences are approximately $(-0.0986,1.3863,1.0000)$. The signs even disagree for the first observation: its logit exceeds the label numerically, but its probability is below the positive label, so the logistic loss decreases locally if the logit increases. The check is to interpret the score on the logit scale and the gradient through the probability map, rather than subtracting the logit from the class label.

Worked Example: Minus-One-Plus-One Logistic Labels

A hypothetical event detector uses labels $y_i\in\{-1,+1\}$ and the loss $\log(1+\exp(-y_iF_i))$. One negative case has $y_1=-1$ and score $F_1=\log 2$; one positive case has $y_2=+1$ and score $F_2=\log 3$. Calculate both pseudo-residuals and compare them with $y_i-F_i$.

Method. Use the negative gradient $y_i/[1+\exp(y_iF_i)]$, which is defined for this label encoding.

For the negative case, $y_1F_1=-\log 2$, so $\exp(y_1F_1)=1/2$. Therefore $$ r_1=\frac{-1}{1+1/2}=-\frac{2}{3}\approx -0.6667. $$ The raw difference is $-1-\log 2$, approximately $-1.6931$.

For the positive case, $y_2F_2=\log 3$, so $\exp(y_2F_2)=3$. Therefore $$ r_2=\frac{1}{1+3}=\frac{1}{4}=0.2500. $$ The raw difference is $1-\log 3$, approximately $-0.0986$.

The pseudo-residuals are approximately $(-0.6667,0.2500)$, not the raw differences. The first pseudo-residual is negative because increasing the score would worsen the negative case’s margin; the second is positive because increasing the score would improve the positive case’s margin. This sign check confirms the direction of the loss gradients. The common mistake avoided is using the zero-one formula $y-p$ with labels coded as $-1,+1$.

Interpreting and Checking the Calculation

Pseudo-residuals depend on both the observed outcome and the current score, through the specified loss. For absolute error, observations on the same side of the score receive the same gradient magnitude, regardless of their raw residual magnitudes. For logistic loss with zero-one labels, the pseudo-residual magnitude depends on how far the fitted probability is from the observed class. A confidently wrong prediction can therefore have a large logistic pseudo-residual, while a confidently correct prediction has a small one.

The phrase “negative gradient” also gives a useful sign check. A positive pseudo-residual means increasing the score locally decreases the loss; a negative pseudo-residual means decreasing the score locally decreases the loss. This is a local statement about the loss as a function of the score. It does not mean a pseudo-residual is itself a probability, a raw error, or necessarily the final update to the model.

Common exam trap: For a binary response, do not calculate $y_i-F_i$ when $F_i$ is a logit. For example, if $y_i=1$ and $F_i=\log 3$, the raw difference is $1-\log 3$, but the logistic pseudo-residual is $1-0.75=0.25$. The correction is to first map the logit to $p=1/(1+\exp(-F_i))$, then use $y_i-p$. Also check the label encoding: the formula changes when classes are coded $-1,+1$.

A reliable procedure is to write down the per-observation loss and the meaning of its score before differentiating. Then take the derivative with respect to the score, negate it, and substitute the current score. For absolute error, explicitly note an exact fit because the loss is not differentiable there. For logistic loss, identify the label convention and distinguish a logit from a probability. These checks prevent sign errors and scale errors when interpreting gradient-based boosting steps aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.

Check Your Understanding

For each question, identify the stated loss and score convention before calculating the pseudo-residual.

Under absolute-error loss, an observation has response $y=14$ and current score $F=11$. Give its raw residual and pseudo-residual. (Answer: raw residual $3$; pseudo-residual $+1$.)
Under absolute-error loss, an observation has $y=6$ and $F=9$. Give the pseudo-residual and explain its sign. (Answer: $-1$; the score is above the response, so decreasing it locally reduces loss.)
Under zero-one logistic loss, $y=0$ and $F=\log 3$. Calculate $p$ and the pseudo-residual. (Answer: $p=0.75$; pseudo-residual $-0.75$.)
Under zero-one logistic loss, $y=1$ and $F=-\log 3$. Calculate $p$ and the pseudo-residual. (Answer: $p=0.25$; pseudo-residual $0.75$.)
Under the minus-one-plus-one logistic loss, $y=-1$ and $F=0$. Calculate the pseudo-residual and compare it with $y-F$. (Answer: pseudo-residual $-0.5$; raw difference $-1$, so they are not equal.)

What You'll Learn