Consistency Corrections And Common Calibration Errors

Why Calibration Requires a Convention

A robust scatter estimate can be well defined and affine equivariant yet still be systematically too small or too large under a reference model such as the multivariate normal distribution. A consistency correction rescales the estimate so that its population value equals the target scatter under that model. The correction is not a universal constant attached to the estimator’s name: it depends on the estimator’s defining equation, its tuning choices, and sometimes the sample-size convention.

This distinction matters especially for MCD, S, and M scatter. For MCD, the trimming fraction determines a truncated reference distribution. For S scatter, the right-hand side of the scale equation determines the resulting scale. For M scatter, the weight function and its normalization determine the population multiplier. In each case, write down the exact convention before calculating a factor.

MCD: Correction for a Truncated Normal Sample

Let \(R^2\sim\chi_p^2\), where \(R\) is the Mahalanobis radius of a standard \(p\)-variate normal vector. Suppose the population MCD retains the central fraction \(\gamma\), with \(0<\gamma<1\). Define \(q_\gamma\) by \(P(R^2\le q_\gamma)=\gamma\). The population subset is the central ellipsoid \(R^2\le q_\gamma\); its location is the normal center by symmetry.

The uncorrected covariance of the retained observations is a scalar multiple \(c_\gamma I_p\). Taking its trace gives \(p c_\gamma=E[R^2\mid R^2\le q_\gamma]\). Therefore \(c_\gamma=E[R^2\mathbf{1}\{R^2\le q_\gamma\}]/(p\gamma)\). The factor \(1/\gamma\) is essential: the conditional covariance divides by the probability of retention.

To verify the formula, first take \(\Sigma=I_p\). The density of \(R^2\) is \(f_p(v)=v^{p/2-1}e^{-v/2}/(2^{p/2}\Gamma(p/2))\) for \(v>0\). Directly from this density, \(v f_p(v)=p f_{p+2}(v)\). Hence \(E[R^2\mathbf{1}\{R^2\le q_\gamma\}]=\int_0^{q_\gamma}v f_p(v)\,dv=pF_{\chi^2_{p+2}}(q_\gamma)\). Dividing by \(p\gamma\) gives the stated multiplier. For general \(\Sigma\), write \(X=\mu+\Sigma^{1/2}Z\), with \(Z\sim N_p(0,I_p)\). The central ellipsoid in \(X\)-coordinates is the affine image of the corresponding ball for \(Z\), so its covariance is \(\Sigma^{1/2}(c_\gamma I_p)\Sigma^{1/2}=c_\gamma\Sigma\). Thus the same scalar correction applies. \(\square\)

S Scatter: Solve the Scale Equation You Actually Use

For an S-scatter definition based on a radial loss \(\rho\), consider the population constraint \(E[\rho(\|X-\mu\|_{S^{-1}})]=b\), where \(\|x\|_{S^{-1}}=(x^\mathsf{T}S^{-1}x)^{1/2}\). Assume the expectation exists and that the positive scale equation below has a unique solution. At a normal model, symmetry gives the center \(\mu\); a spherical scatter solution has the form \(S=s_b^2\Sigma\). Its standardized radius then has the same distribution as \(R/s_b\), with \(R\sim\chi_p\). Accordingly, the scale is determined by \(E[\rho(R/s_b)]=b\). If \(s_b\ne1\), the uncorrected scatter is \(s_b^2\Sigma\), so its correction is \(1/s_b^2\).

Indeed, for \(S=s^2\Sigma\), the squared Mahalanobis distance is \((X-\mu)^\mathsf{T}S^{-1}(X-\mu)=R^2/s^2\), so its radius is \(R/s\). The population constraint is therefore exactly \(E[\rho(R/s)]=b\). By the stated uniqueness, \(s=s_b\), not merely some other positive root. Thus \(S=s_b^2\Sigma\); multiplying by \(s_b^{-2}\) gives \(\Sigma\). \(\square\)

Weighted M Scatter: State the Normalization

“M scatter” covers several estimating equations, so the weight and normalization must be specified before a correction can be computed. For a concrete convention, define a population weighted scatter functional by \(T_w(F)=E[w(Q)(X-\mu)(X-\mu)^\mathsf{T}]/E[w(Q)]\), where \(Q=(X-\mu)^\mathsf{T}T_w(F)^{-1}(X-\mu)\). Assume \(w\) is nonnegative, the denominator is finite and positive, and the relevant moments exist. This convention normalizes by the mean weight. It should not be silently substituted for a differently normalized M-scatter estimating equation.

At a normal model with \(T_w(F)=c_w\Sigma\), the distance \(Q\) has the distribution \(\chi_p^2/c_w\). Rotational symmetry implies that the weighted second-moment matrix is a scalar multiple of \(\Sigma\). Taking traces identifies that scalar, giving \(c_w=E[w(R^2/c_w)R^2]/\{pE[w(R^2/c_w)]\}\). A finite positive value yields correction \(1/c_w\).

This point is important: with a self-standardized distance, the weight is evaluated at \(R^2/c_w\). The correct equation for this convention is \(c_w=E[w(R^2/c_w)R^2]/\{pE[w(R^2/c_w)]\}\), not the expression obtained by evaluating \(w\) at \(R^2\). If that equation has a unique positive solution, its solution is the population multiplier and its reciprocal is the correction. In practice, solve this one-dimensional equation numerically for the specified weight. The simpler closed form using \(w(R^2)\) applies to a weight whose argument is the fixed reference Mahalanobis distance under \(\Sigma\), not to the self-standardized functional just defined.

The general calculation can be checked by tracing both sides. For \(T_w=c\Sigma\), the left trace is \(pc\). Conditional on \(R\), spherical symmetry gives \(E[ZZ^\mathsf{T}\mid R]=R^2I_p/p\). Substitution in the weighted expectation therefore gives trace \(E[w(R^2/c)R^2]\), while its normalizing denominator is \(E[w(R^2/c)]\). Equating traces gives the displayed scalar equation. This derivation also shows why silently changing the weight’s distance convention changes the answer.

Common Calibration Errors

The numerical value of a correction is only meaningful together with the convention that produced it. Several errors recur when software output, formulae, or implementations are compared.

• Using the retained fraction as the correction. In MCD, \(\gamma\) determines the cutoff, but the correction is \(\gamma/F_{\chi^2_{p+2}}(q_\gamma)\), not \(\gamma\) itself.
• Confusing radius with squared radius. An S equation written with \(\rho(R/s)\) uses a radius. An equation written with squared distance uses \(R^2/s^2\). Replacing one argument by the other changes the equation and its calibration.
• Ignoring uniqueness. A scalar scale equation must have a specified solution; to identify it with a particular \(s_b\), assume that positive solution is unique, or explicitly refer to the solution corresponding to the chosen root.
• Combining different M normalizations. A weighted covariance divided by the mean weight, an equation with a fixed right-hand-side multiplier, and a self-standardized M functional need not have the same consistency factor.
• Calling every adjustment a consistency correction. An asymptotic normal-consistency factor is distinct from a finite-sample adjustment. The latter depends on implementation details and sample size; it is not determined by the population calculation alone.

Asymptotic and Finite-Sample Factors Are Different

For MCD, the truncated-normal calculation gives a population or asymptotic factor for a specified trimming fraction. Finite-sample MCD implementations may additionally apply a small-sample factor depending on \(n\), \(p\), \(h\), and the precise covariance convention. Dividing by \(h\) versus \(h-1\), or using a particular finite-sample approximation, changes the reported raw value. Such an adjustment should be identified by its formula or implementation; it cannot be inferred from the asymptotic chi-square calculation.

The same separation applies to S and M scatter. First calculate the population multiplier implied by the estimating equation and tuning constants. Then, only if the procedure specifies one, apply its finite-sample adjustment as a separate factor. If the two factors are \(a\) and \(d\), the final multiplicative adjustment is \(ad\), but their meanings and assumptions remain distinct. This makes reports reproducible and prevents a finite-sample factor from being mistakenly presented as a normal-consistency derivation.