Absolute-Error Prediction and Conditional Medians

Why Absolute Error Leads to a Median

A prediction can be judged by how far it is from the observed response. Under squared-error loss, large errors receive a particularly strong penalty. Under absolute-error loss, an error of 4 costs twice as much as an error of 2. This different penalty changes which prediction is optimal: the conditional mean is generally replaced by a conditional median.

For a response $Y$ and a prediction $a$, the absolute-error loss is $|Y-a|$. Given covariates $X=x$, the relevant quantity is the expected loss under the conditional distribution of $Y$ at that covariate value. Throughout, assume this conditional distribution has a finite first moment, so its absolute-error risk is finite for every finite prediction.

Definition: A median of a real-valued random variable $Y$ is any number $m$ such that $$ P(Y\leq m)\geq \frac12 \qquad\text{and}\qquad P(Y\geq m)\geq \frac12. $$ If $F$ is the distribution function of $Y$, these conditions are equivalently $F(m-)\leq \frac12\leq F(m)$, where $F(m-)=P(Y<m)$. A conditional median at $X=x$ is a median of the conditional distribution of $Y$ given $X=x$.

The definition permits more than one median. A candidate is a median when at least half the probability lies at or below it and at least half lies at or above it. For a distribution with a continuous, strictly increasing distribution function, the median is typically unique. Atoms or gaps in the distribution can produce several medians.

Why Medians Minimize Absolute-Error Risk

Fix a distribution for $Y$, and write its absolute-error risk at prediction $a$ as $r(a)=E[|Y-a|]$. The next result gives an exact characterization of every minimizer. Its finite-first-moment assumption ensures that $r(a)$ is finite; it also allows the one-sided derivatives below to be justified by dominated convergence.

Theorem: If $E[|Y|]<\infty$, then a value $m$ minimizes $r(a)=E[|Y-a|]$ over $a\in\mathbb{R}$ if and only if $m$ is a median of $Y$.

Proof. The function $r$ is convex because $a\mapsto |y-a|$ is convex for every fixed $y$, and taking expectations preserves the convexity inequality. For $h>0$, the right difference quotient of the loss converges pointwise as follows:

\lim_{h\downarrow 0} \frac{|Y-(a+h)|-|Y-a|}{h} = \begin{cases} 1,&Y\leq a,\\ -1,&Y>a. \end{cases}

The absolute value of each difference quotient is at most 1, by the reverse triangle inequality. Dominated convergence therefore gives the right derivative

r'_+(a)=P(Y\leq a)-P(Y>a)=2F(a)-1.

Similarly, the left derivative (the limit of $[r(a)-r(a-h)]/h$ as $h\downarrow0$) is

r'_-(a)=P(Y<a)-P(Y\geq a)=2F(a-)-1.

For a convex function on the real line, $a$ is a minimizer exactly when its left derivative is at most zero and its right derivative is at least zero. Indeed, convexity makes secant slopes nondecreasing: if both derivative inequalities hold, slopes toward points on either side cannot give a lower function value; conversely, a minimum cannot have a positive left derivative or a negative right derivative. Thus $a$ minimizes $r$ exactly when $F(a-)\leq\frac12\leq F(a)$. These are precisely the median conditions. $\square$

This result applies separately to each conditional distribution. If $F_x$ denotes the conditional distribution function of $Y$ given $X=x$, the conditional risk for prediction $a$ is

r_x(a)=E[|Y-a|\mid X=x].

Whenever the conditional first moment is finite, the minimizers of $r_x(a)$ are exactly the medians of $F_x$. Therefore, an absolute-error prediction rule that minimizes risk at each covariate value chooses a conditional median, not generally a conditional mean.

Worked Example: A Unique Median in a Discrete Distribution

Suppose that at a particular covariate value, $Y$ has the following conditional distribution:

P(Y=1)=0.2,\qquad P(Y=4)=0.5,\qquad P(Y=9)=0.3.

The cumulative probability through 1 is $0.2$, while through 4 it is $0.7$. Thus $F(4-)=0.2\leq 0.5\leq 0.7=F(4)$, so 4 is a median. The distribution places more than half its probability at or below 4 and more than half at or above 4; the other possible values do not satisfy both conditions.

The risk at prediction 4 is

$$ r(4)=0.2|1-4|+0.5|4-4|+0.3|9-4| =0.2(3)+0+0.3(5)=2.1. $$

For comparison, predictions 1 and 9 have risks

r(1)=0.2(0)+0.5(3)+0.3(8)=3.9, \qquad r(9)=0.2(8)+0.5(5)+0.3(0)=4.1.

The calculation confirms that prediction 4 has lower risk than either endpoint value. The theorem establishes that it is optimal among all real-valued predictions, not just these three.

When the Optimal Prediction Is Not Unique

A median need not be one of the observed values. For example, if equal probability is concentrated at two separated points, every value between those points satisfies the median definition. The risk can then be exactly constant throughout an interval.

More generally, the set of all minimizers is a nonempty, closed, bounded interval under the finite-first-moment assumption. To see this, $r$ is continuous because $|r(a)-r(b)|\leq |a-b|$. Also,

r(a)=E[|Y-a|]\geq |a|-E[|Y|],

so $r(a)$ tends to infinity as $|a|$ tends to infinity. A continuous function with this property attains its minimum, and its minimizer set is closed and bounded. Finally, convexity implies that if two points minimize $r$, every point between them does too. By the theorem, this minimizer set is exactly the set of medians.

Worked Example: An Interval of Medians

Let $Y$ equal 2 with probability $0.5$ and 8 with probability $0.5$. For any $a$ between 2 and 8, including the endpoints, each outcome is at least as far on one side of $a$ as the other. Directly,

r(a)=0.5|a-2|+0.5|8-a| =0.5(a-2)+0.5(8-a)=3,\qquad 2\leq a\leq 8.

The median conditions give the same conclusion. For $2\leq a\leq 8$, at least half the probability is at or below $a$, and at least half is at or above $a$. Thus every such $a$ is a median and has risk 3.

Outside this interval the risk is larger. If $a<2$, then $r(a)=0.5(2-a)+0.5(8-a)=5-a>3$. If $a>8$, then $r(a)=0.5(a-2)+0.5(a-8)=a-5>3$. The optimal prediction is therefore the entire interval $[2,8]$, not just one selected endpoint.

A Risk-Difference Formula

The following identity gives another way to compare predictions. It is useful for understanding why risk can stay flat across a gap in the distribution.

Result: For any $a<b$ and any real-valued $Y$ with finite first moment, $$ r(b)-r(a)=\int_a^b \bigl(2F(t)-1\bigr)\,dt. $$

Derivation. For each fixed value $y$, moving the prediction from $a$ to $b$ changes the loss by

|y-b|-|y-a| =\int_a^b \left(\mathbf{1}_{\{y<t\}}-\mathbf{1}_{\{y>t\}}\right)\,dt.

The equality follows by checking whether $y$ lies to the left, inside, or to the right of $[a,b]$; the value of the integrand at the single point $t=y$ does not affect the integral. The integrand is bounded, so expectation and integration can be interchanged on this finite interval. Taking expectations gives an integrand $P(Y<t)-P(Y>t)=F(t-)+F(t)-1$. A distribution has at most countably many atoms, so $F(t-)=F(t)$ for almost every $t$ with respect to Lebesgue measure. The integral is therefore $\int_a^b(2F(t)-1)\,dt$, as claimed. $\square$

Worked Example: Checking Risk Across a Median Interval

Return to the distribution with probability $0.5$ at 2 and probability $0.5$ at 8. For every $t$ strictly between 2 and 8, $F(t)=0.5$. Thus, for any $2\leq a<b\leq8$, the risk-difference formula gives

r(b)-r(a)=\int_a^b(2(0.5)-1)\,dt =\int_a^b 0\,dt=0.

This verifies that risk is constant across the median interval. For a prediction below 2, the cumulative probability is 0, so $2F(t)-1=-1$; moving the prediction toward 2 decreases risk. Above 8, $F(t)=1$, so $2F(t)-1=1$; moving farther right increases risk. Both behaviors agree with the direct risk calculation.

Conditional Medians in Prediction

The prediction problem is usually conditional: a rule observes $X$ and then predicts $Y$. For a fixed observed value $x$, the relevant risk is $r_x(a)=E[|Y-a|\mid X=x]$, and its minimizers are the medians of the conditional response distribution. A rule that selects a conditional median for each $x$ is therefore pointwise optimal for absolute-error loss, provided the resulting rule is measurable.

Worked Example: Choosing Predictions by Covariate Value

Let $X$ take values $A$ and $B$, each with probability $0.5$. Suppose the conditional distributions are

\begin{aligned} Y\mid X=A &: \quad P(Y=0)=0.25,\quad P(Y=2)=0.50,\quad P(Y=6)=0.25,\\ Y\mid X=B &: \quad P(Y=1)=0.40,\quad P(Y=5)=0.60. \end{aligned}

When $X=A$, the cumulative probability reaches $0.75$ at 2, and the probability below 2 is $0.25$. Thus 2 is a conditional median. Its conditional risk is

$$ r_A(2)=0.25|0-2|+0.50|2-2|+0.25|6-2| =0.25(2)+0+0.25(4)=1.5. $$

When $X=B$, the cumulative probability through 1 is $0.40$, and through 5 it is 1. Thus 5 is a conditional median. Its conditional risk is

$$ r_B(5)=0.40|1-5|+0.60|5-5| =0.40(4)+0=1.6. $$

The predictor $g(A)=2,\ g(B)=5$ has overall expected absolute error $0.5(1.5)+0.5(1.6)=1.55$. In contrast, using the constant prediction 2 gives conditional risk 1.5 at $A$, but $0.40|1-2|+0.60|5-2|=2.2$ at $B$, for overall risk $0.5(1.5)+0.5(2.2)=1.85$. Conditioning on $X$ allows the prediction to adapt to the response distribution at each covariate value.

Common Mistake

Wrong reasoning: “The mean is the standard prediction, so it must also minimize absolute error.” The mean minimizes squared-error risk under the usual finite-second-moment condition; it does not generally minimize absolute-error risk. Absolute-error risk is minimized by a median. Another related mistake is to use a median of the marginal distribution of $Y$ when the prediction rule has access to $X$. The marginal median need not minimize risk separately at each $x$, as the conditional example shows.

Correction: Match the summary to the loss, then condition on the information available to the prediction rule. Under absolute-error loss, identify the median or medians of the conditional distribution of $Y$ given $X=x$. If there is an interval of such medians, every value in that interval has the same minimum conditional risk.

Key takeaway: Conditional medians, including all values in a nonunique median interval, minimize conditional expected absolute error. The result follows from the one-sided slopes of absolute-error risk and applies separately at each covariate value.

Check Your Understanding

Use the median conditions and absolute-error risks to answer the following questions.

If $P(Y=0)=0.3$, $P(Y=3)=0.4$, and $P(Y=10)=0.3$, identify the median and calculate its absolute-error risk.
If equal probability is placed at 1 and 7, what is the full set of medians, and what is the risk at any prediction in that set?
State the condition on $F(m-)$ and $F(m)$ that characterizes a median.
Why does a prediction rule with covariates use conditional medians rather than necessarily using a median of the marginal response distribution?
For $a<b$, what does the risk-difference formula imply if $F(t)=1/2$ throughout the interval from $a$ to $b$?