Joint M-Estimation Of Location And Scatter

Why Estimate Location and Scatter Together?

A multivariate location estimate describes a data cloud’s center, while a scatter estimate describes its spread and orientation. In a joint procedure, each estimate helps define the other: distances from the proposed center are measured using the proposed scatter matrix, and those distances determine how strongly observations contribute to both equations. This coupling is useful when a few observations are far from the main cloud, but it also means the equations must be interpreted and solved as a system.

For observations \(x_1,\ldots,x_n\) in \(\mathbb{R}^p\), write a candidate location as \(\mu\in\mathbb{R}^p\) and a candidate scatter matrix as a symmetric positive-definite matrix \(S\). The squared Mahalanobis distance of observation \(i\) from the candidate center is \(t_i(\mu,S)=(x_i-\mu)^{\mathsf T}S^{-1}(x_i-\mu)\). It is dimensionless and changes with both \(\mu\) and \(S\).

Typically the weights are nonnegative functions of distance. Small distances may receive weights near one, while large distances receive smaller weights. The two functions need not be the same: a choice that limits the effect of a distant observation on the location equation need not be the same choice that controls its contribution to scatter. The scatter contribution is \(v(t_i)(x_i-\hat{\mu})(x_i-\hat{\mu})^{\mathsf T}\); its size depends on both the weight and the residual vector.

Reading the Equations and Calibrating Scale

If \(\sum_i w(t_i)>0\), the location equation can be rearranged as a weighted mean:

This is an implicit identity, not a one-time formula: the weights depend on distances calculated from \(\hat{\mu}\) and \(\hat{S}\). Similarly, the scatter equation is a weighted second-moment equation around the estimated center. Even if all observations receive positive weights, a solution must still produce a positive-definite matrix; for example, residuals confined to a proper subspace cannot generate full-rank scatter.

The constant \(\kappa\) sets the scale convention. A standard calibration is Fisher consistency at a multivariate standard normal distribution. Let \(Z\sim N(0,I_p)\) and \(Q=Z^{\mathsf T}Z\), so \(Q\) has a chi-square distribution with \(p\) degrees of freedom. Provided the expectation below is finite and positive, choose

To see why, consider the scatter equation at the population center \(0\) and scatter \(I_p\). Rotational symmetry gives \(E[v(Q)ZZ^{\mathsf T}]=cI_p\) for some scalar \(c\). Taking traces yields \(pc=E[v(Q)Q]\), so the expectation of the normalized scatter right-hand side is \(I_p\). The location equation has expectation zero under the same model when \(E[|w(Q)Z_j|]\) is finite for each coordinate \(j\): its integrand changes sign under \(Z\mapsto -Z\), whereas \(Q\) does not. Thus the calibration makes the standard normal location and scatter solve the population equations. It does not guarantee a unique finite-sample solution.

For the classical choice \(v(t)=1\), \(\kappa=E[Q]/p=1\), and the scatter equation becomes the usual empirical second-moment equation around the estimated center. A robust choice of \(v\) changes how the radial distances contribute and generally requires its own calibration. The normalization is part of the estimating equations; changing it changes the scatter scale.

This identity is a useful scale check: a proposed solution that does not satisfy it cannot satisfy the scatter equation. It is necessary, not sufficient, since it checks only a scalar trace rather than every entry of the matrix equation.

Affine Equivariance of the Coupled Equations

The radial formulation has a useful transformation property. If coordinates are changed by a nonsingular linear map and a translation, the equations retain their form, provided location and scatter are transformed accordingly. This is important because a coordinate system should not change the substance of a multivariate location-and-scatter estimate.

Proof: The transformed residual is \(y_i-(a+A\hat{\mu})=A(x_i-\hat{\mu})\). The transformed scatter is positive definite because \(A\hat{S}A^{\mathsf T}\) is positive definite when \(A\) is nonsingular. Its inverse is \(A^{-\mathsf T}\hat{S}^{-1}A^{-1}\), so the transformed squared distance is

Thus the weights are unchanged. The transformed location sum is \(A\sum_i w(t_i)(x_i-\hat{\mu})=0\). The transformed scatter right-hand side is \((n\kappa)^{-1}A\{\sum_i v(t_i)(x_i-\hat{\mu})(x_i-\hat{\mu})^{\mathsf T}\}A^{\mathsf T} =A\hat{S}A^{\mathsf T}\). Both equations hold, proving the result. \(\square\)

Computing the Equations and Using a Robust Start

The equations are commonly approached through reweighting. From a current pair \((\mu^{(0)},S^{(0)})\), calculate all distances and weights, then form weighted location and scatter updates. In a simultaneous update, both use the distances and residuals from that current pair; an alternating method may instead update one parameter and then recalculate distances before updating the other. A fixed point of the chosen update must satisfy the corresponding estimating equations.

A robust starting estimate matters because the equations can have multiple solutions or an iteration can be sensitive to its initial values. An extreme starting center or a poorly conditioned starting scatter can produce misleading distances and weights in early iterations. A robust pilot can make the computation more stable by beginning from a location and scatter less affected by a small number of unusual observations. But the pilot does not guarantee that a solution exists, that it is unique, or that an algorithm will reach the desired solution. Those issues require checking the fitted equations and the numerical behavior, not merely naming the initialization method.

Common Mistake: Treating the Weights as Fixed

A tempting but incorrect argument is: “The location estimate is just the weighted mean, so compute weights once and then the location is solved.” That is true only if the weights are fixed independently of the unknowns. Here \(t_i\) depends on both location and scatter, so changing either estimate changes the weights. The weighted-mean expression is an identity that any solution must satisfy, not a closed-form solution in general.

A related mistake is to regard \(\kappa\) as a harmless multiplier that can be changed without consequence. The scatter equation’s scale is directly affected by \(\kappa\); the trace identity shows that a solution must have \(\sum_i v(t_i)t_i=n\kappa p\). Use the calibration appropriate to the intended reference model and weight function, and distinguish that convention from a different scale normalization chosen for a specific example.