Functional Gradient Descent And Negative Gradients

From Parameter Gradients to Prediction Functions

In gradient boosting, a model is built by adding functions in stages. At a given stage, the current model supplies a prediction $F(x_i)$ for each training case. The next tree is not necessarily fitted directly to the original response. Instead, it is used to approximate a direction in which changing the predictions would reduce the chosen loss.

This is the functional gradient descent viewpoint: treat the collection of predictions as values of a function $F$, calculate how the empirical loss changes when that function is perturbed, and use a tree to represent an approximate improving perturbation. The gradient is evaluated at the current predictions, so the targets used to fit the next tree generally change from one boosting iteration to the next.

Definition: For training data $(x_i,y_i)$, $i=1,\ldots,n$, and a differentiable loss $L(y,F(x))$, the empirical risk of a prediction function $F$ is $$ R(F)=\frac{1}{n}\sum_{i=1}^n L\bigl(y_i,F(x_i)\bigr). $$ The negative loss gradient at observation $i$, also called its pseudo-response, is $$ r_i=-\left.\frac{\partial L(y_i,F(x_i))}{\partial F(x_i)}\right|_{F(x_i)}. $$ It indicates the local direction in which that observation's prediction should move to reduce its loss. For a nondifferentiable loss, a suitable subgradient may be used instead; the derivative formulas below apply to differentiable losses.

The values $r_i$ are not necessarily residuals in the ordinary response-minus-prediction sense. That familiar residual is the negative gradient for squared-error loss, but a different loss generally produces different pseudo-responses. The tree-fitting step uses the current pseudo-responses as targets, often by fitting a regression tree with squared-error splitting and leaf predictions.

Why a Negative Gradient Gives a Descent Direction

Consider a candidate direction $h(x)$ and a positive step size $\gamma$. The updated prediction function is $F_\gamma(x)=F(x)+\gamma h(x)$. To determine whether this direction initially reduces empirical risk, differentiate risk with respect to $\gamma$ at zero. The chain rule gives

\left.\frac{d}{d\gamma}R(F+\gamma h)\right|_{\gamma=0} = \frac{1}{n}\sum_{i=1}^n \left.\frac{\partial L(y_i,F(x_i))}{\partial F(x_i)}\right|_{F(x_i)} h(x_i) = -\frac{1}{n}\sum_{i=1}^n r_i h(x_i).

This equation is the key test. A direction is locally descending when the directional derivative is negative, equivalently when $\sum_i r_i h(x_i)>0$. If the derivative is positive, a sufficiently small positive step increases risk. If it is zero, first-order information alone does not establish improvement. Even when the derivative is negative, a large finite step can overshoot and increase risk; a local descent direction is not a guarantee for every $\gamma$.

Derivation: At each observation, the prediction changes from $F(x_i)$ to $F(x_i)+\gamma h(x_i)$. Differentiating its loss with respect to $\gamma$ multiplies the derivative of loss with respect to prediction by $h(x_i)$, by the chain rule. Summing the $n$ observation-level derivatives and dividing by $n$ gives the displayed directional derivative. Replacing the loss derivative by its negative, $r_i$, yields the final minus sign.

A tree is a restricted function: it divides the predictor space into regions and gives a constant direction within each terminal region. Fitting a regression tree to $r_i$ seeks a tree whose values $h(x_i)$ approximate the observation-level negative gradients. For a fixed partition, the least-squares value in a leaf is the mean pseudo-response in that leaf. If the resulting tree has positive aggregate alignment $\sum_i r_i h(x_i)$, it gives a locally descending direction. The approximation may be imperfect, and a candidate tree should not be assumed to have positive alignment merely because it was called a gradient tree.

Worked Example: A Tree Direction That Reduces Squared-Error Risk

Suppose four hypothetical claims have responses $y=(3,1,-1,-3)$, and the current prediction is $F(x_i)=0$ for each claim. Use half-squared-error loss $L(y,F)=\tfrac12(y-F)^2$. A tree splits the first two claims into one leaf and the last two into another. Find its pseudo-responses, its least-squares leaf directions, and the risk after a step of $\gamma=0.5$.

Method and gradients. For this loss, $\partial L/\partial F=F-y$, so $r_i=y_i-F(x_i)$. The pseudo-responses are therefore $3,1,-1,-3$. In the first leaf their mean is $(3+1)/2=2$; in the second leaf it is $(-1-3)/2=-2$. Thus the fitted direction is $h=(2,2,-2,-2)$.

Check the direction. The alignment is $\sum_i r_i h(x_i)=3(2)+1(2)+(-1)(-2)+(-3)(-2)=6+2+2+6=16$. The directional derivative is $-16/4=-4$, which is negative.

Initial risk. The four losses are $\tfrac12(3^2)=4.5$, $\tfrac12(1^2)=0.5$, $\tfrac12((-1)^2)=0.5$, and $\tfrac12((-3)^2)=4.5$. Hence $R(F)=(4.5+0.5+0.5+4.5)/4=2.5$.

Updated risk. The new predictions are $0.5(2)=1,1,-1,-1$. The response-minus-prediction values are $2,0,0,-2$; the losses are $2,0,0,2$. Therefore $R(F+0.5h)=(2+0+0+2)/4=1$. The risk falls from $2.5$ to $1$, consistent with the negative directional derivative.

Check. The update moves each group in the sign of its average residual, and the computed finite-step risk is below the initial risk. The method is tree fitting to negative gradients; the check avoids assuming that a tree update improves risk without evaluating its direction and step.

Squared-Error Loss as a Special Case

For half-squared-error loss, differentiating with respect to the prediction gives $\partial L/\partial F=F-y$. Its negative is $y-F$, exactly the ordinary response residual. This explains why, under this loss, fitting a tree to residuals is not a separate heuristic: the residuals are the negative gradients of empirical risk. The correspondence depends on the loss function and the half-squared-error convention.

The following example shows why the tree direction must be checked across all observations, not just one at a time. A direction can look plausible as a function while providing no aggregate first-order improvement.

Worked Example: Cancellation Makes a Constant Direction Fail

Two hypothetical observations have current predictions of zero and responses $2$ and $-2$. Use half-squared-error loss. A candidate constant tree direction is $h(x_i)=1$ for both observations, and the proposed step is $\gamma=0.25$. Calculate initial and updated mean risks, and determine whether this is a descent direction.

Initial risk. Each loss is $\tfrac12(2-0)^2=2$, so the mean risk is $(2+2)/2=2$.

Gradient check. The pseudo-responses are $r=(2,-2)$. Since the direction is constant, its alignment is $2(1)+(-2)(1)=0$. The directional derivative is $-0/2=0$. The direction is not a strict descent direction to first order: the two observations' contributions cancel.

Finite update. Both predictions increase by $0.25$. The new predictions are $0.25,0.25$, so the response-minus-prediction residuals are $1.75$ and $-2.25$. The losses are $\tfrac12(1.75)^2=\tfrac12(3.0625)=1.53125$ and $\tfrac12(-2.25)^2=\tfrac12(5.0625)=2.53125$. The updated mean risk is $(1.53125+2.53125)/2=2.03125$.

Check and trap avoided. The updated risk is $2.03125$, which exceeds the initial risk of $2$. A common mistake is to overlook that a positive prediction update helps the observation with response $2$ but hurts the observation with response $-2$. The zero directional derivative rules out a strict first-order descent conclusion, and direct calculation confirms that this finite step increases risk.

Gradient Approximation Is Not the Same as a Guarantee

The functional gradient at the training observations is the vector $(r_1,\ldots,r_n)$. A tree cannot generally reproduce an arbitrary vector: observations that land in the same leaf receive the same direction, and the tree's split structure restricts which patterns can be represented. Thus boosting uses a weak learner to approximate the gradient direction rather than moving freely in every observation-level coordinate.

Least-squares fitting to pseudo-responses is a practical way to find a tree that resembles the negative gradient. For a fixed partition with leaf means, the alignment is nonnegative: in a leaf $A$ with $n_A$ observations and mean pseudo-response $\bar r_A$, its contribution is $\sum_{i\in A}r_i\bar r_A=n_A\bar r_A^2$, which is at least zero. This fact explains why fitting leaf means on a fixed partition aligns with the gradient. It does not establish that every tree-building procedure, constrained fit, or subsequent finite step will lower the original loss. The actual loss and chosen step still matter.

\sum_{i\in A} r_i h(x_i) = \sum_{i\in A} r_i\bar r_A = n_A\bar r_A^2 \geq 0 \qquad\text{when }h(x_i)=\bar r_A\text{ for }i\in A.

Worked Example: A Different Loss Produces Different Pseudo-Responses

For two hypothetical cases, let the responses be $y=(2,0)$ and the current predictions both be zero. Use the differentiable quartic loss $L(y,F)=\tfrac14(y-F)^4$. Fit a constant direction by least squares to the negative gradients, then calculate the directional derivative and risk after a step of $\gamma=0.1$.

Method and pseudo-responses. Differentiating gives $\partial L/\partial F=-(y-F)^3$, so $r=(y-F)^3=(8,0)$. The least-squares constant fit is their mean, $h=(8+0)/2=4$, for each case.

Direction check. The alignment is $8(4)+0(4)=32$, and the directional derivative is $-32/2=-16$. This is a strict descent direction.

Initial risk. The losses are $\tfrac14(2^4)=4$ and $0$. The mean risk is $(4+0)/2=2$.

Updated risk. The new prediction for each case is $0.1(4)=0.4$. The residuals are $2-0.4=1.6$ and $0-0.4=-0.4$. Their losses are $\tfrac14(1.6^4)=\tfrac14(6.5536)=1.6384$ and $\tfrac14((-0.4)^4)=\tfrac14(0.0256)=0.0064$. The updated mean risk is $(1.6384+0.0064)/2=0.8224$.

Check. The risk decreases from $2$ to $0.8224$. Under this loss the pseudo-responses are cubes of the residuals, not the residuals themselves. The method avoids carrying the squared-error residual formula over to a different loss.

Practical Interpretation and Common Exam Traps

The functional-gradient view gives a disciplined way to reason about a boosting update: first identify the loss, then differentiate with respect to the prediction, negate the derivative, fit a tree to those values, and assess whether the resulting direction and step reduce the objective. This reasoning applies to differentiable losses in general, while the specific gradients and any useful approximations depend on the selected loss.

Keep these distinctions clear when interpreting a calculation:

Gradient versus residual: residuals equal negative gradients for half-squared-error loss, but not for every loss.
Direction versus step: a negative directional derivative describes local behavior near a zero step; it does not guarantee that an arbitrary positive step improves risk.
Individual versus aggregate movement: a direction can help some observations and hurt others. The empirical-risk derivative sums their contributions, so cancellation matters.
Approximation versus exact optimization: the fitted tree approximates the negative-gradient vector within the tree's restricted function class. It need not equal that vector or minimize the original loss after the update.
Sign convention: the pseudo-response is the negative derivative. Reversing the sign reverses the proposed direction.

These checks are particularly useful when a problem gives a candidate tree direction rather than asking for a complete tree fit. Compute the gradient alignment and, when a finite step is specified, calculate the updated predictions and loss directly. The result is aligned with the SOA Exam PA (Predictive Analytics) syllabus.

Check Your Understanding

For each problem, use the stated loss and current predictions; distinguish the local direction check from the finite-step risk calculation.

Two observations have responses $3$ and $1$, current predictions zero, and half-squared-error loss. A constant direction $h=1$ is used with $\gamma=0.25$. Find the initial and updated mean risks and the directional derivative. (Answer: Initial risk $2.5$; updated risk $2.03125$; derivative $-2$.)
Two observations have responses $2$ and $-2$, predictions zero, and half-squared-error loss. The direction is $h=(1,-1)$, with $\gamma=0.5$. Find the initial and updated mean risks and directional derivative. (Answer: Initial risk $2$; updated risk $1.125$; derivative $-2$.)
Use $L(y,F)=\tfrac14(y-F)^4$, responses $1$ and $3$, and current predictions zero. Find the pseudo-responses and the least-squares constant direction. For $\gamma=0.05$, calculate initial and updated mean risk and the directional derivative. (Answer: Pseudo-responses $(1,27)$; direction $14$; initial risk $10.25$; updated risk $3.499025$; derivative $-196$.)
Under half-squared-error loss, the current residuals are $(4,2,-2,-4)$. A tree has two leaves, with the first two observations in one leaf and the last two in the other. Find the least-squares direction in each leaf and its directional derivative. (Answer: Directions $3$ and $-3$; derivative $-9$.)
Under half-squared-error loss, residuals are $r=(1,-1)$, and a proposed direction is $h=(2,-1)$. With $\gamma=0.2$, calculate the directional derivative, initial mean risk, and updated mean risk. (Answer: Derivative $-1.5$; initial risk $0.5$; updated risk $0.25$.)

What You'll Learn