Worked Comparison of Mean, Median, and Huber Estimators

Three Location Estimators, Two Distributions

The sample mean, sample median, and Huber estimator all summarize location, but they respond differently to extreme observations. This difference is especially clear when comparing a clean normal distribution with a small point-mass contamination. We will use the same two distributions throughout, so the change in target, influence function, and asymptotic variance can be compared directly.

Let \(Z\sim N(0,1)\), and consider the clean distribution \(F_0=N(0,1)\) and the contaminated distribution \(F=0.9N(0,1)+0.1\Delta_3\), where \(\Delta_3\) puts probability one at 3. Thus, with probability \(0.9\) an observation comes from the standard normal component, and with probability \(0.1\) it equals 3. The contamination proportion and location are fixed; the targets calculated below are properties of these distributions, not estimates from a finite sample.

For the numerical Huber calculations, take \(c=1.345\), a commonly used tuning constant. The Huber score is linear for residuals between \(-c\) and \(c\), and is capped at \(-c\) or \(c\) outside that interval. In all three cases, the estimator is the corresponding sample mean, sample median, or solution of the empirical Huber estimating equation.

Clean Normal Model: Targets and Variances

Under \(F_0\), symmetry gives mean, median, and Huber target zero. The sample mean has asymptotic variance \(\operatorname{Var}_{F_0}(X)=1\). For the median, the density at the target is \(\phi(0)=1/\sqrt{2\pi}\), so its asymptotic variance is \(1/[4\phi(0)^2]=\pi/2\).

For a regular scalar estimating equation \(E_G[\psi(X-\theta)]=0\), write \(A=E_G[\psi'(X-\theta)]\) and \(B=E_G[\psi(X-\theta)^2]\). When the target is unique, \(A\ne0\), the score has finite second moment, and the usual differentiability and empirical central limit conditions hold, the influence function is \(\psi(x-\theta)/A\), and the asymptotic variance is \(B/A^2\). For Huber’s score under a continuous distribution, \(\psi'_c(u)=1\) when \(|u|<c\) and \(0\) when \(|u|>c\); the boundary has probability zero in the normal cases here.

What Changes Under Point-Mass Contamination?

The mean target is \(0.9(0)+0.1(3)=0.3\). The median remains below 3, where the mixture distribution function is \(F(x)=0.9\Phi(x)\). It therefore solves \(0.9\Phi(m)=0.5\), or \(\Phi(m)=5/9\), giving \(m\approx0.1397\). Neither target is the original normal location zero: the target is defined by the actual distribution being sampled.

The Huber target solves \(0.9E[\psi_c(Z-\theta)]+0.1\psi_c(3-\theta)=0\). The solution lies below \(3-c\), so the contamination score equals \(c\). For \(\theta\approx0.1825\), set \(a=\theta-c=-1.1625\) and \(b=\theta+c=1.5275\). The normal-component score expectation is

Substitution gives approximately \(-0.1494\); hence the mixture equation is \(0.9(-0.1494)+0.1(1.345)\approx0\). This verifies the target to the displayed precision. The term \(\psi_c(3-\theta)\) is indeed \(c\), since \(3-0.1825=2.8175>1.345\).

Influence Functions and Asymptotic Variances

An influence function describes the first-order change in a target when a small amount of probability is placed at a value \(x\). Formally, for a functional \(T\), its influence function at \(G\), when the derivative exists, is \(\operatorname{IF}(x;T,G)=\left.\frac{d}{d\varepsilon}T((1-\varepsilon)G+\varepsilon\Delta_x)\right|_{\varepsilon=0}\). For regular estimators, the variance of the influence function under \(G\) is the asymptotic variance of the estimator multiplied by \(n\).

For Huber’s result, define \(G_\varepsilon=(1-\varepsilon)G+\varepsilon\Delta_x\), and let its target be \(\theta_\varepsilon\). Differentiate \(E_{G_\varepsilon}[\psi_c(X-\theta_\varepsilon)]=0\) at \(\varepsilon=0\). The derivative with respect to \(\varepsilon\) from changing the distribution is \(\psi_c(x-\theta)-E_G[\psi_c(X-\theta)]=\psi_c(x-\theta)\), since the target equation has mean score zero. The derivative with respect to \(\theta_\varepsilon\) is \(-A\). Thus \(0=\psi_c(x-\theta)-A\theta'_0\), giving the stated influence function. Its variance under \(G\) is \(B/A^2\). The differentiations are valid when the target is locally differentiable, the score derivative is integrable, and the probability of a residual exactly at a kink does not invalidate the derivative; these conditions hold for the continuous normal component and the point mass strictly in the capped region here. \(\square\)

For the median, its target \(m_\varepsilon\) satisfies \((1-\varepsilon)G(m_\varepsilon)+\varepsilon\mathbf{1}\{x\le m_\varepsilon\}=1/2\) when the contaminating point is not at the target. Differentiating at zero gives \(g(m)m'_0-G(m)+\mathbf{1}\{x\le m\}=0\). Since \(G(m)=1/2\), this yields \(m'_0=[1/2-\mathbf{1}\{x\le m\}]/g(m)\). Under \(G\), the indicator is Bernoulli with probability \(1/2\); the influence function has mean zero and variance \(1/[4g(m)^2]\). The density must be positive and continuous near the median for this local expansion and the usual sample-quantile limit to apply. \(\square\)

Putting the Comparison Together

For the mean, the influence function is \(x-E_G[X]\), and its asymptotic variance is \(\operatorname{Var}_G(X)\). Under the contaminated mixture, this variance is \(0.9(1)+0.1(9)-0.3^2=1.71\). For the median, the contaminated target \(m\approx0.1397\) is away from the point mass, and the local density is \(g(m)=0.9\phi(m)\approx0.3556\). Its asymptotic variance is therefore \(1/[4(0.3556)^2]\approx1.978\). The influence-function magnitudes for the median are \(1/[2g(m)]\approx1.406\), with opposite signs on either side of the target.

The mean is most efficient among these three under the clean normal model, but its influence function is unbounded: an increasingly extreme observation can have an arbitrarily large first-order effect. The median has bounded influence, but under the specified contamination its asymptotic variance is largest. Huber’s estimator has bounded influence, with magnitude at most \(c/A\), and retains variance close to the mean’s under the clean normal model. These comparisons concern this choice of tuning constant and this particular contamination; they are not universal rankings for every distribution.

Common Pitfall

A frequent error is to calculate the Huber sandwich variance using the clean normal component alone, or to use the same mixture weighting in \(A\) and \(B\) without checking what each expectation contains. Under \(F\), the point mass at 3 has a capped score of \(c\), so it contributes \(0.1c^2\) to \(B\). It contributes zero to \(A\), because the score is locally flat there. The correct expressions are \(A=0.9P(a<Z<b)\) and \(B=0.9E[\psi_c(Z-\theta)^2]+0.1c^2\). For the values used here, these give \(B\approx0.8315\) and \(B/A^2\approx1.548\). Omitting the contamination term, or using an inconsistent numerical value for \(B\), changes the variance calculation.

A second conceptual error is to call the contaminated target a biased estimate of zero without specifying the target. Each estimator converges to its functional evaluated at \(F\); the three functionals need not equal the clean component’s location. Whether the shift is undesirable depends on the scientific aim: estimating the mixture’s location and estimating the normal component’s location are different tasks.