Absolute-Error Boosting And Conditional Medians

Why Absolute Error Changes a Boosting Update

In regression boosting, the loss function determines what the model tries to predict. Squared error rewards predictions near the conditional mean and gives large residuals proportionally large gradients. Absolute error has a different target: it favors medians, and its gradient depends on whether a prediction is above or below an observation, not on how far away it is. This distinction matters when a hypothetical claim-severity response is skewed or contains an unusually large value.

Let the training observations be $(x_i,y_i)$, for $i=1,\ldots,n$, and let $F(x)$ be the current prediction. Under absolute-error loss, the empirical objective is the sum of absolute residuals. The initial constant prediction minimizes that objective before any features are used.

Definition: The absolute-error loss for observation $i$ is $L(y_i,F(x_i))=|y_i-F(x_i)|$. A median minimizes the sum of absolute deviations from the observed responses. In boosting, the initial constant prediction is a median of the training responses; an optimized correction for a tree leaf is a median of the current residuals in that leaf.

Why the Median Minimizes Absolute Error

Sort the responses as $y_{(1)}\le\cdots\le y_{(n)}$, and consider the objective $Q(c)=\sum_{i=1}^n|y_i-c|$ for a constant prediction $c$. Moving $c$ slightly to the right increases the distance to every observation below $c$ and decreases the distance to every observation above $c$. Away from observed values, the slope is therefore the number below $c$ minus the number above $c$.

Q'(c)=\#\{i:y_i<c\}-\#\{i:y_i>c\}.

If more than half the observations are below $c$, the slope is positive and moving left reduces the objective. If more than half are above $c$, the slope is negative and moving right reduces it. A minimizing value must leave no more than half the observations strictly on either side: that is precisely the median condition. For an even sample, every value between the two middle ordered observations minimizes the objective; their midpoint is one valid median.

Worked Example: Initializing With an Asymmetric Set of Responses

A hypothetical portfolio team records five claim severities, in hundreds of dollars: $2,3,3,7,20$. Find the absolute-error-optimal initial constant and compare its loss with the mean-based constant.

Method. Under absolute-error loss, initialize at a median. The ordered responses are already shown, so the middle value is $3$. The initial prediction is therefore $3$ hundred dollars, or $300.

At this prediction, the absolute deviations are $1,0,0,4,17$. Their sum is $22$ response units, equivalent to $2,200.

The mean is $(2+3+3+7+20)/5=35/5=7$, or $700. Its absolute deviations are $5,4,4,0,13$, with sum $26$ units, or $2,600. Thus the median has the lower absolute-error objective: $22<26$.

Check. The large value $20$ pulls the mean above the median, but it does not change which response is in the middle. The calculated losses confirm that $3$ is preferable for this objective. Using the mean as the absolute-error initializer is the mistake this comparison avoids.

Conditional Medians and Boosting Updates

A constant median describes the best absolute-error prediction when no features are used. With features, the corresponding population target at a given feature value $x$ is a conditional median of $Y$ given $X=x$. In other words, a value $m(x)$ is a conditional median when neither side of it has conditional probability greater than one half:

P(Y\le m(x)\mid X=x)\ge \tfrac12 \quad\text{and}\quad P(Y\ge m(x)\mid X=x)\ge \tfrac12.

This follows by applying the median argument to the conditional expected loss $E[|Y-c|\mid X=x]$: a prediction that leaves more than half the conditional probability on one side can be moved toward that side to reduce expected absolute error. When the conditional distribution is asymmetric, its median need not equal its mean.

For boosting, let $F_{m-1}$ be the model before iteration $m$. For a nonzero residual $y_i-F_{m-1}(x_i)$, the negative gradient of absolute-error loss with respect to the prediction is its sign. Thus the pseudo-residual records the direction of correction. At an exactly zero residual, the loss is not differentiable; any value in the subgradient interval $[-1,1]$ is valid, and implementations commonly assign zero.

r_{im}= \begin{cases} +1,&y_i-F_{m-1}(x_i)>0,\\ -1,&y_i-F_{m-1}(x_i)<0,\\ 0,&y_i-F_{m-1}(x_i)=0\quad\text{(a common convention).} \end{cases}

A regression tree is fitted to these pseudo-residuals to determine where the model needs upward or downward corrections. Once its regions are fixed, the correction within each leaf is chosen to reduce the original absolute-error objective. For a leaf $R_{jm}$, the exact unshrunk correction is a median of the current residuals $y_i-F_{m-1}(x_i)$ for observations in that leaf. With learning rate $0<\nu\le1$, the update is:

\gamma_{jm}\in\arg\min_{\gamma}\sum_{x_i\in R_{jm}} |y_i-F_{m-1}(x_i)-\gamma|, \qquad F_m(x)=F_{m-1}(x)+\nu\sum_j\gamma_{jm}I(x\in R_{jm}).

The tree-fitting step and leaf-correction step serve different purposes: the pseudo-residuals guide the tree structure, while the median residual gives the loss-minimizing correction for a fixed leaf. A software implementation may use a particular tree-fitting criterion, but this distinction remains important when interpreting the absolute-error update.

Worked Example: Finding Corrections for Two Fixed Leaves

A hypothetical model currently predicts $1,000 for six claims. Their observed severities, in hundreds of dollars, are $5,9,30,10,10,20$. A candidate tree divides the observations into leaf A, containing $5,9,30$, and leaf B, containing $10,10,20$. Find the initial constant, the exact correction in each leaf, and the total absolute loss before and after a full update.

Initial constant. The ordered responses are $5,9,10,10,20,30$. The two middle values are both $10$, so the median, and initial prediction, is $10$ hundred dollars ($1,000).

Current residuals. Subtract the prediction $10$ from each response. Leaf A has residuals $-5,-1,20$, whose median is $-1$. Leaf B has residuals $0,0,10$, whose median is $0$. Therefore $\gamma_A=-1$ and $\gamma_B=0$.

Loss before updating. Leaf A contributes $|-5|+|-1|+|20|=5+1+20=26$. Leaf B contributes $0+0+10=10$. The total is $36$ hundred-dollar units, or $3,600.

Loss after the full update. The prediction in A becomes $10+(-1)=9$; its new residuals are $5-9=-4$, $9-9=0$, and $30-9=21$. Its loss is $4+0+21=25$. Leaf B remains at prediction $10$, with loss $10$. The updated total is $25+10=35$ units, or $3,500.

Check. The absolute-error objective falls from $36$ to $35$ units. Using the mean residual in leaf A, $(-5-1+20)/3=14/3$, instead of its median would not give the minimizing correction. This example avoids confusing a mean-based leaf value with an absolute-error-optimal one.

Asymmetric Residuals: Direction Is Not Magnitude

The sign-based pseudo-residuals give equal gradient magnitude to a small and a large nonzero residual. For example, residuals $-1$ and $-20$ both contribute $-1$, while $+1$ and $+20$ both contribute $+1$. Consequently, the balance of observations on either side can determine the update even when residual sizes are very uneven. The median correction still reflects the ordered residual values, but it is not pulled toward an extreme value in the way a mean can be.

Worked Example: A Large Positive Residual Does Not Set the Leaf Correction

A current prediction for five hypothetical claims leaves residuals, in hundreds of dollars, of $-3,-1,2,2,40$. Find the exact absolute-error leaf correction, its pseudo-residual sign balance, and the result of applying learning rate $\nu=0.5$.

Exact correction. The ordered residuals are $-3,-1,2,2,40$, so their median is $2$. The full leaf correction is $+2$ units, or $200. At the current prediction, the absolute loss is $3+1+2+2+40=48$ units, or $4,800.

Sign information. The pseudo-residuals are $-1,-1,+1,+1,+1$. There are three positive signs and two negative signs, so the direction of the majority is upward. Each sign has magnitude one; the residual $40$ does not have forty times the gradient contribution of a residual $1$.

Shrinkage. With $\nu=0.5$, the applied correction is $0.5(2)=1$ unit, or $100. Subtracting this update from each old residual gives $-4,-2,1,1,39$. The new absolute loss is $4+2+1+1+39=47$ units, or $4,700.

Check. The shrinkage update reduces the loss from $4,800 to $4,700. The mean residual is $(-3-1+2+2+40)/5=8$, not the absolute-error-optimal correction. At correction $8$, the residuals would be $-11,-9,-6,-6,32$, with total absolute loss $64$ units. This check shows why neither the mean nor the largest residual should be substituted for the median.

Common Traps and Interpretation

Starting with a mean. The mean minimizes summed squared error, whereas a median minimizes summed absolute error. Choose the initializer from the loss being optimized.
Using residual magnitudes as gradient weights. Under absolute loss, nonzero pseudo-residuals have magnitude one. A residual of $40$ and a residual of $2$ point in the same direction with the same gradient magnitude.
Using the pseudo-residual average as the exact leaf correction. A tree is fitted to pseudo-residuals to identify regions, but the exact absolute-loss correction in a fixed leaf is a median of the current residuals there.
Forgetting shrinkage. The median is the unshrunk correction. The model applies $\nu$ times that correction, so the fitted prediction changes by less when $0<\nu<1$.

Absolute-error boosting is useful when a median prediction is the target or when reducing the influence of extreme residual magnitudes is important. It does not make unusual observations irrelevant: they still affect the loss and may affect which value is the median. It changes how their size influences the gradient and the location of the optimal update.

Key takeaway: Absolute-error boosting begins with a median, uses residual signs to guide tree fitting, and uses a median of current residuals for an exact fixed-leaf correction. Asymmetry matters through the balance and ordering of residuals, while shrinkage scales the correction actually applied.

Check Your Understanding

For each problem, identify the absolute-error-optimal quantity before calculating the update.

A hypothetical training sample has responses $4,6,6,9,25$. Find the initial constant and the minimum sum of absolute deviations. (Answer: initial constant $6$; minimum absolute loss $24$)
A fixed leaf has current residuals $-4,-1,2,7,11$. Find its exact unshrunk correction and the absolute loss at that correction. (Answer: correction $2$; absolute loss $23$)
A leaf has four residuals $-3,-1,5,9$. Give the full set of minimizing corrections and one conventional median choice. (Answer: any correction from $-1$ to $5$; the midpoint choice is $2$)
A leaf has residuals $-5,-2,1,4,30$ and uses learning rate $0.4$. Find the applied correction and the new sum of absolute residuals. (Answer: correction $0.4$; new absolute loss $41.6$)
A tree has two fixed leaves. Leaf A responses are $1,3,100$, and leaf B responses are $8,9,10$; suppose the current prediction in both leaves is zero. Find each exact leaf correction. (Answer: leaf A correction $3$; leaf B correction $9$)

What You'll Learn