Robust Residual Scale With the Median Absolute Deviation

Why Estimate Residual Scale Robustly?

Huber regression limits the influence of large residuals by reducing their weights, but deciding which residuals are large requires a scale for comparison. A scale estimate based on squared deviations can itself be pulled upward by an extreme residual. The median absolute deviation, usually abbreviated MAD, offers a resistant alternative: it measures a typical distance from the residual center using medians rather than squared deviations.

The MAD is useful here because Huber weights depend on standardized residuals. Dividing each residual by a robust estimate of residual scale puts it on a comparable scale, so a chosen Huber cutoff can be applied consistently. The normal-consistency adjustment converts the raw MAD into an estimate that targets the error standard deviation when the errors are normally distributed.

Definition and Normal-Consistency Adjustment

Suppose a fitted regression gives residuals $r_1,\ldots,r_n$, where $r_i=y_i-\hat{y}_i$ is the observed response minus its fitted value. The residuals and their scale have the same units as the response. First find the median residual, then find the median of the absolute distances from that median.

Definition: The residual median absolute deviation is $$ \operatorname{MAD}(r)=\operatorname{median}_{i}\left|r_i-\operatorname{median}_{j}(r_j)\right|. $$ The normal-consistency adjusted MAD scale estimate is $$ \hat{s}_{\operatorname{MAD}}=\frac{\operatorname{MAD}(r)}{0.67449}. $$ For an even number of values, use the usual sample median: the average of the two middle ordered values.

The constant $0.67449$ is not an arbitrary correction. If $Z\sim N(0,1)$, its absolute value has median $q$ satisfying $P(|Z|\le q)=0.5$. By symmetry, $P(|Z|\le q)=2\Phi(q)-1$, where $\Phi$ is the standard normal cumulative distribution function. Thus $2\Phi(q)-1=0.5$, so $\Phi(q)=0.75$ and $q\approx0.67449$. If errors have a normal distribution with standard deviation $\sigma$, their population MAD is therefore approximately $0.67449\sigma$. Dividing by $0.67449$ calibrates the MAD to estimate $\sigma$.

This adjustment is called normal consistency: under a normal error distribution, the population version of the adjusted MAD equals the standard deviation. It does not mean that a finite-sample estimate from regression residuals is exactly unbiased. Fitted residuals are not generally independent observations of the errors, and finite-sample behavior can depend on the model and sample size. The adjusted MAD is a robust scale estimate, not a universal finite-sample correction.

Worked Example: Calculating a MAD-Based Residual Scale

Suppose a hypothetical fitted model has eight residuals, measured in response units: $-2,-1,0,1,2,3,4,18$. Calculate the residual MAD and the adjusted scale estimate.

Find the residual median. The ordered middle values are 1 and 2, so $\operatorname{median}(r)=(1+2)/2=1.5$.

Find the absolute deviations from 1.5. They are $3.5,2.5,1.5,0.5,0.5,1.5,2.5,16.5$. In order, these are $0.5,0.5,1.5,1.5,2.5,2.5,3.5,16.5$. The median is the average of the fourth and fifth values:

\operatorname{MAD}(r)=\frac{1.5+2.5}{2}=2.0.

Apply the normal-consistency adjustment.

\hat{s}_{\operatorname{MAD}}=\frac{2.0}{0.67449}\approx2.965.

The robust residual-scale estimate is about 2.965 response units. It estimates the spread of the residual distribution; it is not the size of a typical residual from zero, nor does it assert that every residual should lie within 2.965 units.

Using the Scale to Set Huber Weights

In Huber iteratively reweighted least squares (IRLS), residuals are standardized by a scale estimate before comparing them with the Huber cutoff. For residual $r_i$ and scale estimate $\hat{s}$, the standardized residual is $u_i=r_i/\hat{s}$. A common Huber cutoff is $k=1.345$. Residuals with $|u_i|\le k$ receive full weight; larger residuals receive a smaller weight.

Definition: Given a positive residual scale estimate $\hat{s}$ and Huber cutoff $k>0$, the Huber IRLS weight for residual $r_i$ is $$ w_i=\begin{cases} 1, & |r_i|/\hat{s}\le k,\\[4pt] k\hat{s}/|r_i|, & |r_i|/\hat{s}>k. \end{cases} $$ Equivalently, $w_i=\min(1,k/|u_i|)$ when $u_i=r_i/\hat{s}\ne0$; set $w_i=1$ when $u_i=0$.

These are weights for a weighted least-squares step: a case with weight below 1 contributes less to that step than a case with weight 1. The weights do not delete observations. In an iterative fit, residuals and the scale estimate can be recalculated as the fit is updated, and the corresponding weights can change. When using a robust regression model, the residuals in this calculation are measured against the fitted regression function; they are not replaced by distances from an independently chosen center.

The formula follows directly from the Huber score function. For standardized residual $u$, the Huber score is $\psi(u)=u$ when $|u|\le k$, and $\psi(u)=k\,\operatorname{sign}(u)$ when $|u|>k$. The IRLS weight is $\psi(u)/u$ for $u\ne0$. In the first case this ratio is $u/u=1$; in the second it is $k/|u|$. Therefore the weight is 1 up to the cutoff and decreases in inverse proportion to the standardized residual’s magnitude beyond it.

Worked Example: Turning a MAD Scale Into Huber Weights

Use the hypothetical residuals from the previous calculation: $-2,-1,0,1,2,3,4,18$. Their adjusted MAD scale is $\hat{s}=2.965$ response units. With $k=1.345$, calculate the weights for residuals 3, 4, and 18.

The cutoff in the original response units is $k\hat{s}$:

k\hat{s}=1.345(2.965)\approx3.988.

A residual whose absolute value is at most 3.988 receives weight 1. Thus the residual 3 has weight 1. For residual 4, which is just beyond the cutoff,

u=\frac{4}{2.965}\approx1.349,\qquad w=\frac{1.345}{1.349}\approx0.997.

For residual 18,

u=\frac{18}{2.965}\approx6.071,\qquad w=\frac{1.345}{6.071}\approx0.222.

The residual of 18 response units receives substantially less weight in the weighted least-squares step than a residual of 3. It is not discarded: its weight is positive. The values also show why scale matters. Applying the cutoff to raw residuals without dividing by a scale would make the weight rule depend on the arbitrary units in which the response is recorded.

Assumptions and Interpretation

The calculation is straightforward, but its interpretation depends on the setting. The normal-consistency interpretation is appropriate when the underlying errors are approximately normal and share a common scale. More generally, the adjusted MAD remains a robust summary of spread, but it need not estimate a conventional standard deviation under a nonnormal error distribution. It also describes spread around the residual center, not whether the regression model is correctly specified.

For using Huber weights, the residual scale should be positive and should describe the residual variation relevant to the fitted model. A scale of zero makes standardization undefined; it can occur, for example, if the residuals have no variation in a small or exactly fitted data set. In that case, the weight calculation cannot proceed as written, and the model or scale-estimation method needs attention. A single common scale also presumes that one scale is a useful summary across the cases being weighted. If residual spread varies systematically across observations, one global scale may not characterize that pattern well.

The residual median used in the MAD formula centers the deviations when measuring spread. It does not change the regression residuals used in the Huber weight formula. In a regression with an intercept, residuals are often centered near zero, but robustly fitted residuals need not have an exactly zero sample median. If the residual median is noticeably away from zero, treat that as a possible diagnostic signal; do not silently subtract it from every residual before applying the regression’s Huber rule. The regression fit, including its intercept, defines the residuals to be weighted.

Worked Example: How One Extreme Residual Affects the MAD

Consider two hypothetical sets of five residuals, in response units: $(-2,-1,0,1,2)$ and $(-2,-1,0,1,200)$. Compare their MAD-based scales and their ordinary sample standard deviations.

For both sets, the residual median is 0. In the first set, the ordered absolute deviations are $0,1,1,2,2$, so the MAD is 1. In the second set, they are $0,1,1,2,200$, which also have median 1. Therefore both adjusted MAD scales are

\hat{s}_{\operatorname{MAD}}=\frac{1}{0.67449}\approx1.483.

The ordinary sample standard deviation changes substantially. For the first set, the mean is 0 and the sum of squared deviations is 10, giving $s=\sqrt{10/4}\approx1.581$. For the second set, the mean is $198/5=39.6$, and the sum of squared deviations is $40006-5(39.6)^2=32165.2$. Thus $s=\sqrt{32165.2/4}\approx89.67$.

In this example, replacing one residual with a very large value leaves the MAD unchanged, while it greatly increases the sample standard deviation. This illustrates the MAD’s resistance to an isolated extreme value. It does not mean the MAD is unaffected by every form of contamination: changing enough observations, or changing the central pattern of residuals, can change the median and MAD.

Common Mistakes and Practical Limits

Omitting the normal-consistency adjustment. The raw MAD is not on the standard-deviation scale for normal errors: it is about $0.67449\sigma$. Divide by 0.67449 when using the normal-consistent estimate.
Taking absolute values from zero instead of the residual median. The MAD is defined using distances from the median residual. If residuals are not centered exactly at zero, these calculations can differ.
Applying Huber weights to unstandardized residuals. The cutoff applies to $|r_i|/\hat{s}$, not directly to $|r_i|$, unless the residuals have already been put on the scale used for the rule.
Calling a small weight a deletion. A Huber weight below 1 reduces a case’s contribution in a weighted step; it is not the same as removing the case from the data.
Treating a robust scale as a cure for every regression problem. MAD-based scaling helps set residual weights, but it does not by itself verify the model or guarantee that every influential feature of the data has been handled. In particular, downweighting based on residual size is not a substitute for examining the design and the observations.

The practical value of the MAD is that it supplies a scale estimate less sensitive to a few extreme residuals than a squared-deviation estimate. Combined with the Huber rule, it provides a principled way to distinguish ordinary residual variation from unusually large residuals when updating a robust fit. The adjustment, assumptions, and units should be reported clearly so that the resulting weights can be understood.

Check Your Understanding

Use the definitions and procedures in this tutorial to answer the following questions.

For residuals $-3,-1,0,2,4$, find the residual median, the MAD, and the normal-consistency adjusted scale.
Why is the normal-consistency constant approximately 0.67449? Give the probability statement that identifies it.
If $\hat{s}=2$, $k=1.345$, and a residual is 5 response units, calculate its Huber weight.
Explain why the MAD can stay stable when one residual becomes extreme, and give one reason it is not immune to all changes in the data.
Why should residuals be centered at their median when calculating the MAD but not automatically recentered before applying the Huber weight rule?