Learning-Rate Shrinkage And Its Effect

Why Scale Each Tree’s Contribution?

A boosted model builds a prediction in stages. At each stage, a new tree supplies an adjustment to the current predictions. If every adjustment is applied in full, the model can change quickly; if each adjustment is scaled down, its path develops more gradually. The scale factor is the learning rate, also called the shrinkage parameter.

Shrinkage changes the model’s additive prediction, not the tree’s structure by itself. A small learning rate often means that more tree contributions are needed to reach a similar fitted value. It may also change performance on new observations: gradual updates can be useful, but there is no rule that a smaller rate always improves accuracy. The appropriate comparison is based on predictive performance, not on the learning rate alone.

Definition: Let $F_0(x)$ be the initial prediction for input $x$, and let $h_m(x)$ be the prediction adjustment supplied by the tree at iteration $m$. With learning rate $\nu$, where $0<\nu\leq 1$, the additive model after $M$ iterations is $$ F_M(x)=F_0(x)+\nu\sum_{m=1}^{M}h_m(x). $$ Equivalently, $F_m(x)=F_{m-1}(x)+\nu h_m(x)$. The value $\nu=1$ applies each tree contribution in full; a value strictly between zero and one shrinks each contribution. If $\nu=0$, no tree contribution is added.

The tree outputs can be positive or negative and can differ across observations. Thus shrinkage does not necessarily move every prediction upward or toward a particular target value; it scales the size of each tree’s change. At a fixed input $x$, the difference between the shrunken prediction and the initial prediction is exactly $\nu$ times the sum of the tree outputs at $x$.

Worked Example: Calculating a Prediction After Three Shrunken Updates

A hypothetical model starts with prediction $F_0(x)=12$ for a policy profile. Three successive trees return $h_1(x)=4$, $h_2(x)=-2$, and $h_3(x)=3$. Calculate the final prediction for learning rates $\nu=1$ and $\nu=0.25$, and compare the total changes from the initial prediction.

Method. Add the tree outputs, multiply their sum by the learning rate, and add the result to the initial prediction. This uses the additive-model formula and avoids treating the learning rate as a multiplier on $F_0(x)$.

The sum of the tree outputs is $$ 4+(-2)+3=5. $$ With $\nu=1$, $$ F_3(x)=12+1(5)=17. $$ The total change is $17-12=5$.

With $\nu=0.25$, the individual scaled contributions are $$ 0.25(4)=1,\qquad 0.25(-2)=-0.5,\qquad 0.25(3)=0.75. $$ Their sum is $1-0.5+0.75=1.25$, so $$ F_3(x)=12+1.25=13.25. $$ The total change is $13.25-12=1.25=0.25(5)$.

Check. The shrunken total change is one quarter of the unshrunken change, as required. The initial prediction remains 12 in both calculations.

Why the Scaling Formula Holds

At each iteration, the model uses the current prediction plus a scaled tree adjustment. Repeated substitution gives the cumulative formula. This short derivation explains why the learning rate applies to each tree contribution and not just to the last one.

Start with the first update.
$F_1(x)=F_0(x)+\nu h_1(x)$.

Substitute into the second update.
$F_2(x)=F_1(x)+\nu h_2(x)=F_0(x)+\nu h_1(x)+\nu h_2(x)$.

Continue by induction.
If $F_{m-1}(x)=F_0(x)+\nu\sum_{j=1}^{m-1}h_j(x)$, then $F_m(x)=F_{m-1}(x)+\nu h_m(x)=F_0(x)+\nu\sum_{j=1}^{m}h_j(x)$. Thus the formula holds for every positive integer $M$.

This identity also gives a direct comparison at a fixed iteration count. If the trees are held fixed, the change from $F_0(x)$ is linear in $\nu$. In actual boosting fits, however, changing the learning rate can change the later trees because each new tree is fitted using the model’s current state. Therefore, two separately fitted models need not have tree outputs that match term by term.

Connection to a Squared-Error Step Size

For squared-error regression, a tree can be viewed as a direction in which to adjust the current predictions. The residual for observation $i$ is $r_i=y_i-F(x_i)$, where $y_i$ is its observed response and $F(x_i)$ is its current prediction. If the tree direction gives value $h_i=h(x_i)$, a full step of size $\rho$ changes the prediction to $F(x_i)+\rho h_i$. The new residual is $r_i-\rho h_i$.

For a fixed direction $h_i$, the sum of squared errors as a function of the step size is $$ Q(\rho)=\sum_{i=1}^{n}(r_i-\rho h_i)^2. $$ When $\sum_i h_i^2>0$, minimizing this quadratic over unrestricted real $\rho$ gives the least-squares step size below. The condition matters: if every $h_i=0$, then the direction changes no predictions and $Q(\rho)$ is constant, so this formula has a zero denominator and does not identify a unique step.

Derivation: Expand the quadratic and differentiate with respect to $\rho$: $$ Q(\rho)=\sum_{i=1}^{n}r_i^2-2\rho\sum_{i=1}^{n}r_i h_i+\rho^2\sum_{i=1}^{n}h_i^2, \qquad Q'(\rho)=-2\sum_{i=1}^{n}r_i h_i+2\rho\sum_{i=1}^{n}h_i^2. $$ Setting $Q'(\rho)=0$ yields $\rho^*=\dfrac{\sum_i r_i h_i}{\sum_i h_i^2}$. Since $Q''(\rho)=2\sum_i h_i^2>0$, this stationary point is the unique minimum.

In a boosting update, the tree direction may be multiplied by this full step and then shrunk: $F_{\text{new}}(x_i)=F(x_i)+\nu\rho^*h_i$. In some formulations, the fitted tree’s leaf values already include the step-size convention, so the update is written simply as $F_{\text{new}}=F+\nu h$. The essential point is to identify what $h$ represents before applying another factor; otherwise the step size may be counted twice.

Worked Example: Shrinking a Least-Squares Update

For three observations, suppose the current residuals are $r=(3,1,-1)$, and a fitted tree gives direction values $h=(2,1,-1)$. Find the unrestricted least-squares step size, the sum of squared errors after a full step, and the sum of squared errors after applying learning rate $\nu=0.25$ to that full step.

Method. Use the least-squares step-size formula, then calculate each updated residual. This distinguishes the full line-search update from its shrunken version.

First calculate the numerator: $$ \sum_i r_i h_i=3(2)+1(1)+(-1)(-1)=6+1+1=8. $$ Then calculate the denominator: $$ \sum_i h_i^2=2^2+1^2+(-1)^2=4+1+1=6. $$ Therefore, the full step is $$ \rho^*=\frac{8}{6}=\frac{4}{3}. $$ The initial sum of squared residuals is $3^2+1^2+(-1)^2=11$.

For the full step, the updated residuals are $$ 3-\frac{4}{3}(2)=\frac{1}{3},\qquad 1-\frac{4}{3}(1)=-\frac{1}{3},\qquad -1-\frac{4}{3}(-1)=\frac{1}{3}. $$ Thus the resulting sum of squared errors is $$ \left(\frac{1}{3}\right)^2+\left(-\frac{1}{3}\right)^2+\left(\frac{1}{3}\right)^2=\frac{1}{3}. $$ For $\nu=0.25$, the effective step size is $0.25(4/3)=1/3$. The updated residuals are $$ 3-\frac{1}{3}(2)=\frac{7}{3},\qquad 1-\frac{1}{3}(1)=\frac{2}{3},\qquad -1-\frac{1}{3}(-1)=-\frac{2}{3}. $$ The shrunken update’s sum of squared errors is $$ \left(\frac{7}{3}\right)^2+\left(\frac{2}{3}\right)^2+\left(-\frac{2}{3}\right)^2 =\frac{49+4+4}{9}=\frac{19}{3}. $$

Check. The full step gives the minimum along this direction, so its error $1/3$ is below the initial error $11$. The shrunken step moves only partway along the direction and has error $19/3$, also below 11 but above the full-step minimum.

Iteration Count and Predictive Accuracy

At a fixed sequence of tree outputs, smaller $\nu$ scales down the cumulative change at every iteration. It can take more iterations to make a change of a given size. But model fitting is sequential: later trees depend on the current predictions, and validation performance depends on both the fitted sequence and the data being evaluated. It is therefore not valid to infer that one learning rate must be more accurate merely because its individual updates are smaller.

The following hypothetical example isolates the arithmetic of the trade-off. It uses one validation observation and a fixed sequence of candidate tree outputs. The sequence is illustrative; it is not a claim that a fitted boosting algorithm must produce these same outputs under different learning rates.

Worked Example: Smaller Updates and a Different Best Iteration

A hypothetical validation response is $y=10$, and the initial prediction is $F_0=8$. Consider the fixed tree-output sequence $h_1=1,\ h_2=1,\ h_3=1,\ h_4=1,\ h_5=-2$. Compare validation absolute error after each iteration for $\nu=1$ and $\nu=0.5$.

Method. Add $\nu h_m$ at each stage, then take the absolute difference between the prediction and 10. For a single validation observation, this absolute error is also the root mean squared error.

With $\nu=1$, the successive predictions after iterations 0 through 5 are $8,9,10,11,12,10$. The corresponding absolute errors are $2,1,0,1,2,0$. For example, at iteration 2, $F_2=8+1+1=10$, giving error $|10-10|=0$. At iteration 5, the final negative tree changes the prediction to $12-2=10$, also giving error 0.

With $\nu=0.5$, each of the first four contributions adds $0.5$, and the fifth adds $-1$. The predictions after iterations 0 through 5 are $8,8.5,9,9.5,10,9$. The corresponding absolute errors are $2,1.5,1,0.5,0,1$. The prediction at iteration 4, for instance, is $8+4(0.5)=10$, so its error is 0. At iteration 5, $F_5=10-1=9$, so its error is 1.

Check. In this fixed sequence, the smaller learning rate reaches zero validation error at iteration 4 rather than iteration 2. The negative fifth contribution changes the errors again, showing why the iteration count matters as well as the learning rate. This example demonstrates the arithmetic, not a universal accuracy ranking between learning rates.

In practice, accuracy should be evaluated using an appropriate validation measure for the prediction task. For squared-error regression, RMSE is one common choice; for other tasks, a different metric may be suitable. Comparisons should be made on data not used to fit the trees. The learning rate and the number of iterations work together: a rate that produces gradual changes may need more iterations, while an excessively long sequence can eventually worsen predictive performance.

Common Mistakes and Exam Traps

Scaling the initial prediction. The update is $F_0+\nu\sum_m h_m$, not $\nu(F_0+\sum_m h_m)$. The learning rate scales tree contributions, not the initial value.
Using the learning rate only on the last tree. Each contribution is scaled. For instance, $F_2=F_0+\nu h_1+\nu h_2$.
Claiming a smaller rate must improve accuracy. It gives smaller updates for a fixed tree sequence, but accuracy depends on the whole fitted sequence and the evaluation data.
Confusing the tree direction with a full step. If a direction has already been multiplied by a line-search step size, check whether shrinkage is applied to that full update or is already built into the tree-output convention.
Assuming a constrained minimum always exists. For the squared-error quadratic, if the allowed step-size interval is closed, the constrained minimizer is the allowed point closest to the unrestricted minimizer. If the interval is not closed, a minimum need not exist. For example, if the unrestricted minimizer is $\rho^*=2$ and the permitted interval is $(0,1)$, the quadratic decreases throughout that interval; its infimum is approached as $\rho$ tends to 1, but 1 is excluded, so there is no minimizing value in the interval.

Key takeaway: Shrinkage replaces each tree adjustment $h_m(x)$ by $\nu h_m(x)$, so the additive change after $M$ iterations is $\nu\sum_{m=1}^{M}h_m(x)$. Smaller updates can require additional iterations, but neither the best learning rate nor the best iteration count can be determined from that algebra alone; predictive performance must be assessed on appropriate held-out data.

Check Your Understanding

For each problem, calculate the requested quantity before checking the answer.

A model begins at $F_0(x)=7$. Two trees return $h_1(x)=3$ and $h_2(x)=-1$. Find $F_2(x)$ for $\nu=0.2$. (Answer: $7.4$)
For residuals $r=(2,-1)$ and direction values $h=(1,1)$, calculate the unrestricted least-squares step size and the sum of squared errors after that full step. (Answer: $\rho^*=0.5$; sum of squared errors $=4.5$)
Using the residuals and direction in Question 2, calculate the sum of squared errors after applying learning rate $\nu=0.5$ to the full step. (Answer: $4.625$)
A fixed tree sequence has outputs $h_1=2$ and $h_2=-1$ at an input, with $F_0=5$. Find the prediction after two iterations for $\nu=0.25$, and state the total change from $F_0$. (Answer: prediction $5.25$; total change $0.25$)
A squared-error line search has unrestricted minimizer $\rho^*=1.4$. The permitted interval is the open interval $(0,1)$. Does a constrained minimizing value exist? (Answer: No; the quadratic decreases toward the excluded endpoint 1, so its infimum is not attained.)

What You'll Learn