Huber Location Estimation Under Contamination

Why Clip a Location Score?

The sample mean responds to an observation in direct proportion to its distance from the center. Under a normal model this is efficient, but a single extremely large or small value can move it substantially. Huber location estimation modifies the location score so that ordinary residuals retain their linear contribution while sufficiently large residuals have a bounded contribution.

The clipping threshold controls this compromise. A smaller threshold limits the influence of more observations, but typically gives up more efficiency under a normal model. Throughout this tutorial the scale $s>0$ is treated as fixed and known; the focus is the location parameter $\theta$.

Definition: For a clipping constant $c>0$, Huber’s clipped score is $$ \psi_c(u)= \begin{cases} -c, & u<-c,\\ u, & -c\leq u\leq c,\\ c, & u>c. \end{cases} $$ Equivalently, $\psi_c(u)=\max(-c,\min(u,c))$. Its corresponding loss is $\rho_c(u)=\int_0^u\psi_c(v)\,dv$, which equals $u^2/2$ when $|u|\leq c$ and $c|u|-c^2/2$ when $|u|>c$.

For observations $X_1,\ldots,X_n$, the Huber location estimator minimizes the empirical criterion $\sum_{i=1}^n\rho_c((X_i-\theta)/s)$, or equivalently solves its estimating equation when a solution exists. The scale in the residual standardizes deviations before clipping.

Estimating equation: A Huber location estimate $\hat\theta$ satisfies $$ \sum_{i=1}^n \psi_c\!\left(\frac{X_i-\hat\theta}{s}\right)=0. $$ The equation follows by differentiating the criterion with respect to $\theta$: its derivative is $-s^{-1}\sum_i\psi_c((X_i-\theta)/s)$. Because $\rho_c$ is convex, any minimizer satisfies this equation; conversely, any solution is a minimizer.

Solving the Clipped-Score Equation

Each residual contributes its standardized value if it lies between $-c$ and $c$, and contributes only $-c$ or $c$ if it lies beyond those limits. Thus the equation balances clipped residuals, not raw residuals. A useful way to check a proposed estimate is to identify which observations are clipped, substitute their contributions, and verify that the sum is zero.

Worked Example: Solving a Huber Equation by Hand

Take observations $0,1,2,10,11$, with $s=1$ and $c=1.5$. Check whether $\hat\theta=2.25$ solves the estimating equation.

The residuals $X_i-\hat\theta$ are $-2.25,-1.25,-0.25,7.75,8.75$. Applying $\psi_{1.5}$ gives contributions $-1.5,-1.25,-0.25,1.5,1.5$, respectively. Their sum is $$ -1.5-1.25-0.25+1.5+1.5=0. $$ So $2.25$ is a solution. It is also the unique solution: in a neighborhood of $2.25$, the observations 1 and 2 are in the linear part of the score, so the sum has slope $-2$ as a function of $\theta$. The other three contributions are constant there. The sum therefore crosses zero strictly at $2.25$, and, since the full score sum is nonincreasing in $\theta$, it cannot cross back to zero elsewhere.

For comparison, the sample mean is $24/5=4.8$. Here the two large observations each contribute only $1.5$ to the Huber equation, rather than contributing their full residuals. The estimate consequently stays much closer to the central observations.

Influence Under a Contamination Model

Let $F_0$ be a baseline distribution with location $\mu$, and let $T(F)$ denote the population location defined by the Huber equation. Consider the contamination path $F_\varepsilon=(1-\varepsilon)F_0+\varepsilon G$, where $G$ is a contaminating distribution and $\varepsilon$ is a small nonnegative fraction. Assume $T(F_0)=\mu$, the scale $s$ is fixed, and $E_{F_0}[\psi_c((X-\mu)/s)]=0$.

Definition: The influence function of $T$ at $F_0$, when the derivative exists, is $$ \operatorname{IF}(x;T,F_0)= \left.\frac{d}{d\varepsilon}T\big((1-\varepsilon)F_0+\varepsilon\Delta_x\big)\right|_{\varepsilon=0+}, $$ where $\Delta_x$ is the distribution placing probability one at $x$. It describes the first-order change in the population estimate from an infinitesimal amount of contamination at $x$.

Write $U=(X-\mu)/s$, and suppose $$ A=E_{F_0}[\psi_c'(U)]>0. $$ For the clipped score, $\psi_c'(u)=1$ when $|u|<c$ and $0$ when $|u|>c$. If $F_0$ has no probability mass at the clipping boundaries, then $A=P_{F_0}(|U|<c)$. The positive value of $A$ ensures a nonzero local slope of the population equation with respect to location.

Theorem: Suppose the population equation is differentiable in location at $\mu$, differentiation under the expectation is valid, the contamination-path derivative exists, and $0<A=E_{F_0}[\psi_c'(U)]<\infty$. Then $$ \operatorname{IF}(x;T,F_0)=\frac{s}{A}\psi_c\!\left(\frac{x-\mu}{s}\right). $$

Proof. Define $\Psi(\theta,F)=\int\psi_c((x-\theta)/s)\,dF(x)$. By assumption $\Psi(\mu,F_0)=0$. Its derivative with respect to location at $(\mu,F_0)$ is $$ \frac{\partial\Psi}{\partial\theta}(\mu,F_0) =-\frac{1}{s}E_{F_0}[\psi_c'(U)] =-\frac{A}{s}. $$ For the point-mass path $F_\varepsilon=(1-\varepsilon)F_0+\varepsilon\Delta_x$, holding $\theta=\mu$ fixed gives $$ \left.\frac{\partial}{\partial\varepsilon}\Psi(\mu,F_\varepsilon)\right|_{\varepsilon=0+} =\psi_c\!\left(\frac{x-\mu}{s}\right)-\Psi(\mu,F_0) =\psi_c\!\left(\frac{x-\mu}{s}\right). $$ Differentiate the identity $\Psi(T(F_\varepsilon),F_\varepsilon)=0$ at $\varepsilon=0+$. The chain rule yields $$ -\frac{A}{s}\operatorname{IF}(x;T,F_0) +\psi_c\!\left(\frac{x-\mu}{s}\right)=0. $$ Solving for the influence function gives the stated expression. $\square$

This expression is bounded: $|\operatorname{IF}(x;T,F_0)|\leq sc/A$. In particular, an observation far beyond either clipping point has no greater first-order effect than one just at that point. Under the general contamination distribution $G$, the derivative at $\varepsilon=0+$ is $E_G[\operatorname{IF}(X;T,F_0)]$, provided the same differentiation conditions hold.

Worked Example: First-Order Shift from an Extreme Contaminant

Suppose the baseline is $N(\mu,\sigma^2)$, the scale is $s=\sigma$, and $c=1.345$. Consider contamination at the single point $x=\mu+5\sigma$. Since $(x-\mu)/s=5>c$, the score is clipped at $1.345$.

For this baseline, $A=P(|Z|<1.345)$, where $Z\sim N(0,1)$. Using $\Phi(1.345)\approx0.9107$, we obtain $A=2\Phi(1.345)-1\approx0.8214$. Thus $$ \operatorname{IF}(\mu+5\sigma;T,F_0) =\frac{\sigma(1.345)}{0.8214} \approx1.637\sigma. $$ For contamination fraction $\varepsilon=0.02$, the first-order change is approximately $0.02(1.637\sigma)=0.0327\sigma$. This is a local approximation as $\varepsilon$ tends to zero, not an assertion that the exact shift at a two-percent contamination rate equals this value.

If the contaminating point were moved still farther away, its score would remain $1.345$, so this influence function would not grow. The calculation depends on using the fixed scale $s=\sigma$; an estimator that also estimates scale has additional behavior not included here.

Asymptotic Variance and Normal-Model Efficiency

For independent observations from a baseline distribution, the same score equation gives a first-order representation of the estimator. Assume $\hat\theta$ is a consistent solution, the population score has derivative $-A/s$ at $\mu$, the empirical score satisfies a central limit theorem, and the usual local Taylor expansion of the estimating equation is valid. Also assume $E_{F_0}[\psi_c(U)^2]<\infty$. These conditions hold for the normal model with fixed positive scale and $A>0$.

Theorem: Under the preceding regularity conditions, if $X_1,\ldots,X_n$ are independent and identically distributed from $F_0$, then $$ \sqrt{n}(\hat\theta-\mu) =\frac{s}{A}\frac{1}{\sqrt n}\sum_{i=1}^n \psi_c\!\left(\frac{X_i-\mu}{s}\right)+o_p(1), $$ and hence $$ \sqrt{n}(\hat\theta-\mu)\xrightarrow{d} N\!\left(0,\frac{s^2B}{A^2}\right), \qquad B=E_{F_0}[\psi_c(U)^2]. $$

Derivation. Let $\Psi_n(\theta)=n^{-1}\sum_{i=1}^n\psi_c((X_i-\theta)/s)$. The estimating equation is $\Psi_n(\hat\theta)=0$, while $E[\Psi_n(\mu)]=0$. Taylor expansion around $\mu$, using consistency and the assumed local derivative behavior, gives $$ 0=\Psi_n(\mu)-\frac{A}{s}(\hat\theta-\mu)+o_p(n^{-1/2}). $$ Rearranging, $$ \sqrt n(\hat\theta-\mu) =\frac{s}{A}\frac{1}{\sqrt n}\sum_{i=1}^n\psi_c\!\left(\frac{X_i-\mu}{s}\right)+o_p(1). $$ The summands have mean zero and variance $B$. The central limit theorem therefore gives a limiting variance $s^2B/A^2$, as claimed. $\square$

Under the normal model with $s=\sigma$, $U=Z\sim N(0,1)$ and $A=P(|Z|<c)=2\Phi(c)-1$. The sample mean has variance $\sigma^2/n$. Huber’s estimator has asymptotic variance $(\sigma^2/n)B/A^2$, so its asymptotic relative efficiency compared with the sample mean is $A^2/B$. The factor $\sigma^2$ is essential: the estimating equation is based on residuals divided by $\sigma$, but the resulting location uncertainty is measured in the original units.

Worked Example: Efficiency at the Conventional Normal Tuning

Take $c=1.345$ and $Z\sim N(0,1)$. Then $\Phi(c)\approx0.9107$, so $$ A=2\Phi(c)-1\approx0.8214. $$ To compute $B=E[\psi_c(Z)^2]$, separate the central and tail regions: $$ B=E[Z^2\mathbf 1\{|Z|\leq c\}]+c^2P(|Z|>c). $$ For a standard normal variable, integration by parts gives $\int_c^\infty z^2\phi(z)\,dz=c\phi(c)+1-\Phi(c)$. Therefore $$ B=1-2c\phi(c)-2\{1-\Phi(c)\}+2c^2\{1-\Phi(c)\} =A-2c\phi(c)+2c^2\{1-\Phi(c)\}. $$ Using $\phi(1.345)\approx0.1615$ and evaluating with unrounded normal probabilities gives $B\approx0.7102$. (Rounded intermediate values can produce a slightly different last digit.) Consequently, $$ \frac{A^2}{B}\approx\frac{(0.8214)^2}{0.7102} =\frac{0.6747}{0.7102} \approx0.950. $$ Thus the estimator retains about 95% of the sample mean’s asymptotic efficiency at the normal model. Its asymptotic variance is approximately $(\sigma^2/n)(1/0.950)$, or about $1.053\,\sigma^2/n$.

Interpreting the Tradeoff and Avoiding a Common Error

Clipping does not make an estimator immune to contamination. It bounds the first-order effect of an individual observation, while the factor $1/A$ accounts for the slope of the population estimating equation. If the baseline puts little probability in the unclipped region, then $A$ is small and the location equation is relatively flat; the influence bound $sc/A$ can therefore be large even though the score itself is bounded. Choosing $c$ is a tradeoff between sensitivity to extreme residuals and efficiency when the baseline model is credible.

Common mistake: It is tempting to report the normal asymptotic variance as $B/A^2$ and conclude that this is the variance of $\hat\theta$. That quantity is the variance of the limiting distribution of $\sqrt n(\hat\theta-\mu)$ only when the scale is one. With $s=\sigma$, the derivative of the population score is $-A/\sigma$, so its reciprocal contributes a factor $\sigma$ to the linear representation and $\sigma^2$ to the variance. The correct variance is $\sigma^2B/(nA^2)$.

A second point to keep in view is that the numerical efficiency comparison is relative to the sample mean at the normal model and assumes the scale is fixed at the true $\sigma$. It does not establish the performance of a procedure with an estimated scale, nor does it guarantee efficiency under a nonnormal baseline. Those questions require their own analysis.

Key takeaway: Huber location estimation solves a sum of clipped, standardized residuals equal to zero. Its influence function is $s\psi_c((x-\mu)/s)/A$, while its asymptotic variance is $s^2B/(nA^2)$. Under normality, $c=1.345$ gives approximately 95% relative efficiency while keeping the influence bounded.

Check Your Understanding

Use the definitions and derivations above to answer these questions.

For $s=2$, $c=1.5$, and residual $X_i-\theta=5$, what is its contribution to the Huber estimating equation?
What assumptions let $A$ be written as $P(|U|<c)$, and why is $A>0$ needed for the influence-function formula?
For a point contaminant $x$, explain why moving $x$ farther past the clipping threshold does not increase the Huber influence function.
In the normal model, what is $B$, and how does it enter the asymptotic variance of $\hat\theta$?
Why is the normal-model efficiency at $c=1.345$ about $A^2/B$, rather than $B/A^2$?