Pseudo-Residuals In Least-Squares Boosting

Why Fit a Learner to Residuals?

In gradient boosting, each new learner is meant to improve the current predictor in a direction that reduces the chosen loss. For squared-error loss, that direction has a particularly direct interpretation: at each training observation, it is the difference between the observed response and the current fitted value. Those differences are the residuals. Least-squares boosting turns this observation into a practical procedure: calculate residuals, fit a learner to them, and add the fitted learner to the current prediction.

The word “pseudo” emphasizes that this is an instance of a more general gradient-based construction. For squared-error loss, however, the pseudo-residuals are exactly the ordinary response residuals. The key is to calculate the gradient with respect to the current fitted value, not with respect to the original response.

Definition and Gradient Calculation

Let the training data be \((x_i,y_i)\), \(i=1,\ldots,n\), and let \(F\) be the current prediction function. Use the half-squared-error loss for one observation,

The factor \(1/2\) simplifies the derivative. The empirical risk is the average of these losses, \(R(F)=n^{-1}\sum_{i=1}^n L(y_i,F(x_i))\). The pseudo-residual at observation \(i\) is the negative derivative of its individual loss with respect to its current fitted value.

The scaling in the empirical risk should not be confused with the derivative of an individual loss. For one observation, the derivative is \(-[y_i-F(x_i)]\). By contrast, differentiating the empirical risk with respect to the single fitted-value coordinate \(F(x_i)\) gives \(-[y_i-F(x_i)]/n\), because that observation’s loss is weighted by \(1/n\). These derivatives differ by the same positive factor for every observation, so they give the same direction of descent.

The Least-Squares Boosting Step

A learner is fitted by least squares to the pseudo-residuals, not to the original \(y_i\) values. If the available learner class is \(\mathcal H\), the fitting step seeks a function \(h\in\mathcal H\) that minimizes \(\sum_i\{r_i-h(x_i)\}^2\). In practice, \(\mathcal H\) might consist of constant functions or regression trees of a chosen form. After fitting, the update adds the learner’s predictions to the current predictions. A step factor may be included; this tutorial uses \(\alpha\) to make that scaling explicit, without requiring a particular method for choosing it.

The initial constant predictor is commonly chosen to minimize the empirical squared-error risk among all constants. Differentiating \(\sum_i\frac12(y_i-c)^2\) with respect to \(c\) gives \(-\sum_i(y_i-c)\). Setting this derivative to zero yields \(c=\bar y\), where \(\bar y=n^{-1}\sum_i y_i\). Thus \(F_0(x)=\bar y\) is a least-squares constant initialization.

Worked Updates

Why a Least-Squares Fit Reduces Risk

The decrease in the tree example follows from a general projection calculation. Fix a partition of the training observations into leaves and fit a constant residual prediction within each leaf. This is least-squares projection onto the vector space of vectors that are constant within each leaf. More generally, the same calculation applies whenever the fitted prediction vector is the least-squares projection of the residual vector onto a linear subspace. The partition or subspace must be fixed for the projection calculation; it may have been selected using the data, but the claimed identity concerns the resulting fitted vector and its least-squares fit.

Proof. Write \(h_i=h(x_i)\). Since \(h\) is the least-squares projection of \(r\) onto \(S\), the projection residual \(r-h\) is orthogonal to every vector in \(S\). In particular, \(h\in S\), so \(\sum_i(r_i-h_i)h_i=0\). Rearranging gives \(\sum_i r_i h_i=\sum_i h_i^2\).

The updated residual for observation \(i\) is \(y_i-\{F(x_i)+\alpha h_i\}=r_i-\alpha h_i\). Expand the difference in empirical risks: \[ R(F_{\mathrm{new}})-R(F)=\frac{1}{2n}\sum_i\{(r_i-\alpha h_i)^2-r_i^2\} =\frac{1}{2n}\left(-2\alpha\sum_i r_i h_i+\alpha^2\sum_i h_i^2\right). \] Substitute \(\sum_i r_i h_i=\sum_i h_i^2\) to obtain the stated identity. The coefficient \(-\alpha+\alpha^2/2\) is negative exactly when \(0<\alpha<2\); when \(h\) is nonzero, its squared norm is positive, so the risk strictly decreases. At \(\alpha=1\), the coefficient is \(-1/2\), giving the asserted decrease. \(\square\)

For a fixed tree partition, the vectors that are constant within each leaf form a linear subspace. The least-squares fit in each leaf is its mean residual, so the theorem applies to that fit. If the learner class is more restricted or its fit is only approximate, the projection identity need not hold; one should not claim this exact risk decrease without verifying the relevant least-squares property.

Common Mistakes and Practical Meaning

• Using the original responses as the new target. That would fit \(h\) to \(y_i\), ignoring the current predictor. The correct targets are \(r_i=y_i-F(x_i)\); the new learner is added to \(F\).
• Putting \(1/n\) into the individual-loss derivative. The derivative of \(\frac12(y_i-z)^2\) is \(-(y_i-z)\). The factor \(1/n\) appears in the derivative of the averaged empirical risk with respect to the fitted-value coordinate.
• Assuming every learner class makes the same update. A constant learner can adjust only the overall level. A tree can assign different residual averages to different regions of the input space.
• Claiming that any fitted learner must decrease training risk. The exact projection result requires a least-squares projection onto a linear subspace, and its strict decrease requires a nonzero fit and \(0<\alpha<2\).

Least-squares pseudo-residuals make the gradient step tangible: they identify which observations are underpredicted or overpredicted and by how much. Fitting a tree to them creates a structured correction, while the additive update preserves the contributions of earlier learners. This is the basic mechanism behind regression boosting with squared-error loss.