Influence Diagnostics for Robust Covariance Inference

Why Influence Is Not the Same as Outlyingness

A multivariate observation can be far from the main data cloud without materially changing a robust covariance estimate. Conversely, a less extreme observation may affect an estimate or a test conclusion if it helps determine a direction or a boundary. Outlyingness describes a point relative to a reference distribution or scatter estimate; influence describes how much an analysis changes when that point is perturbed or removed.

For covariance inference, the relevant question is not only whether a case looks unusual. It is whether the case changes the estimated scatter enough to alter the statistic used for inference. A useful diagnostic therefore compares the full-sample result with a case-deleted result, while keeping track of the fact that deletion also changes the sample size.

Definition: Let $\hat{\Sigma}_n$ be a robust scatter estimate from $n$ observations, and let $\hat{\Sigma}_{(-i)}$ be the estimate after deleting observation $i$. The case-deletion change for scatter is $\hat{\Sigma}_n-\hat{\Sigma}_{(-i)}$. Its size and direction describe the effect of deleting that case for this particular estimator and data set. Multiplying by $n$ is often useful for comparing with a first-order influence approximation.

The direction matters as well as the magnitude. A case can change a variance in one direction while changing a covariance or another variance in the opposite direction. For a symmetric matrix, one possible size measure is the Frobenius norm, $\|A\|_F=\{\operatorname{tr}(A'A)\}^{1/2}$. A scalar norm summarizes magnitude, but the matrix difference should also be inspected to see which components changed.

Casewise Influence and First-Order Approximation

For a population scatter functional $T(F)$, where $F$ is a distribution, the influence function measures the derivative of the functional under a small point-mass contamination. Write $F_{\varepsilon,x}=(1-\varepsilon)F+\varepsilon\Delta_x$, where $\Delta_x$ puts probability one at $x$. When the derivative exists, the influence function is the first-order change as $\varepsilon$ increases from zero.

Definition: The influence function of $T$ at $F$, evaluated at $x$, is $$ \operatorname{IF}(x;T,F) = \left.\frac{d}{d\varepsilon}T(F_{\varepsilon,x})\right|_{\varepsilon=0}. $$ For a regular asymptotically linear estimator, this function describes the leading contribution of an observation to estimation error. It is a local, first-order diagnostic; it is not an exact finite-sample deletion effect.

The distinction between the two diagnostics is important. Case-deletion changes are directly computed for the observed sample, whereas an influence function describes an infinitesimal change at a specified distribution. Under appropriate regularity and deletion stability, the latter approximates the former after scaling.

Result: Suppose an estimator $T_n$ has an asymptotically linear expansion with mean-zero influence function $\operatorname{IF}$, and suppose the expansion is deletion-stable for the case under consideration: the difference between its full-sample and deleted-sample remainders is $o_p(n^{-1})$. If the average influence contribution from the other $n-1$ cases is $o_p(1)$, then $$ n(T_n-T_{(-i)})=\operatorname{IF}(X_i;T,F)+o_p(1). $$ The approximation is conditional on these regularity assumptions; asymptotic linearity alone, stated only with a full-sample remainder of $o_p(n^{-1/2})$, does not guarantee deletion stability at the finer $n^{-1}$ scale.

To see the calculation, write the full-sample and deleted expansions as $$ T_n=T+\frac{1}{n}\sum_{j=1}^n \operatorname{IF}(X_j;T,F)+r_n, \qquad T_{(-i)}=T+\frac{1}{n-1}\sum_{j\ne i}\operatorname{IF}(X_j;T,F)+r_{n,(-i)}. $$ Let $a=\operatorname{IF}(X_i;T,F)$, and let $\bar a_{(-i)}$ be the average of the other $n-1$ influence contributions. Subtracting gives $$ T_n-T_{(-i)} = \frac{a-\bar a_{(-i)}}{n} + r_n-r_{n,(-i)}. $$ The assumed remainder difference is $o_p(n^{-1})$, and $\bar a_{(-i)}=o_p(1)$. Multiplying by $n$ therefore yields the stated approximation. $\square$

Worked Examples: Scatter Changes and Influence

Worked Example: A Radially Clipped Scatter Estimate

Consider five two-dimensional observations $(2,0),(-2,0),(0,2),(0,-2),(10,0)$. For this illustration, treat the center as known to be zero and define a radially clipped scatter estimate with clipping constant $c=1$: $$ \hat{\Sigma}_n=\frac{1}{n}\sum_{j=1}^n \min\left(1,\frac{1}{\|x_j\|^2}\right)x_jx_j'. $$ Each observation has radius at least $2$, so its weight is $1/\|x_j\|^2$. Thus each weighted outer product is the outer product of its unit direction. The first two observations each contribute $\begin{pmatrix}1&0\\0&0\end{pmatrix}$, the next two each contribute $\begin{pmatrix}0&0\\0&1\end{pmatrix}$, and $(10,0)$ contributes $\begin{pmatrix}1&0\\0&0\end{pmatrix}$. Summing and dividing by five gives $$ \hat{\Sigma}_5= \frac{1}{5} \begin{pmatrix}3&0\\0&2\end{pmatrix} = \begin{pmatrix}0.6&0\\0&0.4\end{pmatrix}. $$

Deleting $(10,0)$ leaves two unit contributions in each coordinate direction. The deleted estimate is $$ \hat{\Sigma}_{(-5)} = \frac{1}{4} \begin{pmatrix}2&0\\0&2\end{pmatrix} = \begin{pmatrix}0.5&0\\0&0.5\end{pmatrix}. $$ Therefore, $\hat{\Sigma}_5-\hat{\Sigma}_{(-5)}=\begin{pmatrix}0.1&0\\0&-0.1\end{pmatrix}$, and $5(\hat{\Sigma}_5-\hat{\Sigma}_{(-5)})=\begin{pmatrix}0.5&0\\0&-0.5\end{pmatrix}$. This estimator is deliberately simple and serves to illustrate a deletion calculation; it is not a claim that this particular formula is a universally appropriate scatter estimator.

Although $(10,0)$ is much farther from the origin than the other cases, its clipped contribution is no larger than that of either ordinary observation on the horizontal axis. In fact, deleting $(2,0)$ also leaves two contributions in each direction and produces the same deleted estimate. In this sample, the most outlying case is not more influential on this estimate than that less extreme horizontal case.

Worked Example: A Deletion Change in a Covariance Statistic

Use the estimates from the preceding calculation to illustrate a covariance discrepancy statistic for the specified matrix $\Sigma_0=I_2$. Define $g(\Sigma)=(\Sigma_{11}-1,\Sigma_{22}-1)'$ and $Q_n=n\,g(\hat{\Sigma}_n)'g(\hat{\Sigma}_n)$. This is an uncalibrated statistic for illustrating influence, not a claim about a reference distribution or a test decision.

For the full sample, $g(\hat{\Sigma}_5)=(-0.4,-0.6)'$, so $$ Q_5=5\{(-0.4)^2+(-0.6)^2\} =5(0.16+0.36)=2.6. $$ After deleting $(10,0)$, $g(\hat{\Sigma}_{(-5)})=(-0.5,-0.5)'$. The actual deleted statistic uses the remaining sample size: $$ Q_{4,(-5)}=4\{(-0.5)^2+(-0.5)^2\}=4(0.5)=2. $$ Thus the actual change is $Q_{4,(-5)}-Q_5=-0.6$.

It is informative to separate the change in the discrepancy from the change in its multiplier. Holding the multiplier fixed at five gives $5(0.5)=2.5$, a change of $2.5-2.6=-0.1$. The actual deleted statistic subtracts the deleted discrepancy $0.5$, because its multiplier is four rather than five. Hence the total change is $-0.1-0.5=-0.6$. Comparing the full statistic with a fixed-multiplier calculation alone would miss this term.

How Deletion Changes a Covariance-Test Statistic

The multiplier issue can be stated generally. Let $\theta$ be a parameter estimated by $\hat{\theta}$, let $g(\theta)$ encode the covariance restriction being assessed, and let $W$ be a fixed symmetric matrix. Define $Q_n=n\,g(\hat{\theta})'Wg(\hat{\theta})$. Assume $g$ is twice continuously differentiable in a neighborhood containing both estimates and the line segment between them, with bounded second derivatives there. Write $J(\theta)$ for its derivative matrix and $d=\hat{\theta}_{(-i)}-\hat{\theta}$.

Result: If $\|d\|=O_p(n^{-1})$, both estimates lie in that neighborhood with probability tending to one, and $J$ and $g(\hat{\theta})$ are $O_p(1)$, the difference holding the multiplier fixed satisfies $$ n\left\{g(\hat{\theta}_{(-i)})'Wg(\hat{\theta}_{(-i)}) -g(\hat{\theta})'Wg(\hat{\theta})\right\} = 2n\,J(\hat{\theta})'Wg(\hat{\theta})\mathbin{\cdot}d +O_p(n^{-1}). $$ For the actual statistic with multiplier $n-1$, the exact change is this fixed-multiplier difference minus $g(\hat{\theta}_{(-i)})'Wg(\hat{\theta}_{(-i)})$. Under a null restriction with $g(\theta_0)=0$, if $\hat{\theta}-\theta_0=O_p(n^{-1/2})$, the deleted estimate has the same order, and $d=O_p(n^{-1})$, this additional term is $O_p(n^{-1})$.

For the derivation, Taylor expansion along the segment between the estimates gives $$ g(\hat{\theta}_{(-i)})=g(\hat{\theta})+J(\hat{\theta})d+R, \qquad \|R\|=O_p(\|d\|^2). $$ Writing $g=g(\hat{\theta})$ and $J=J(\hat{\theta})$, symmetry of $W$ yields $$ g(\hat{\theta}_{(-i)})'Wg(\hat{\theta}_{(-i)})-g'Wg = 2g'WJd+(Jd)'W(Jd)+2g'WR+2(Jd)'WR+R'WR. $$ The last four terms are $O_p(n^{-2})$ under the stated bounds. Multiplication by $n$ gives the fixed-multiplier formula. For the actual statistic, set $A=g(\hat{\theta})'Wg(\hat{\theta})$ and $A_{(-i)}=g(\hat{\theta}_{(-i)})'Wg(\hat{\theta}_{(-i)})$. Direct subtraction gives $$ (n-1)A_{(-i)}-nA = n(A_{(-i)}-A)-A_{(-i)}. $$ This displays the multiplier-change term explicitly. Under the additional null-rate conditions, smoothness gives $g(\hat{\theta}_{(-i)})=O_p(n^{-1/2})$, so $A_{(-i)}=O_p(n^{-1})$. Without those conditions, it must not be absorbed into an $O_p(n^{-1})$ remainder. $\square$

Worked Example: Bounded Influence Can Coexist With Large Distance

Worked Example: Contamination of a Clipped Scatter Functional

Define the population functional $T(F)=E_F[\psi(X)]$, where $$ \psi(x)=\min\left(1,\frac{4}{\|x\|^2}\right)xx' $$ for $x\ne0$, and $\psi(0)=0$. Let $F$ place probability $1/4$ at each of $(\sqrt{2},0),(-\sqrt{2},0),(0,\sqrt{2}),(0,-\sqrt{2})$. Each point has squared radius $2$, so its weight is one. The two horizontal outer products average to $\begin{pmatrix}1&0\\0&0\end{pmatrix}$, and the two vertical outer products average to $\begin{pmatrix}0&0\\0&1\end{pmatrix}$; therefore $T(F)=I_2$.

For a point-mass contamination at $x$, $T(F_{\varepsilon,x})=(1-\varepsilon)T(F)+\varepsilon\psi(x)$. Differentiating at zero gives $\operatorname{IF}(x;T,F)=\psi(x)-T(F)$. At $x=(10,0)$, the weight is $4/100=0.04$, so $\psi(x)=\begin{pmatrix}4&0\\0&0\end{pmatrix}$ and $$ \operatorname{IF}((10,0);T,F) = \begin{pmatrix}3&0\\0&-1\end{pmatrix}. $$ Its Frobenius norm is $\sqrt{3^2+(-1)^2}=\sqrt{10}$, while the point’s squared Euclidean distance from the center is $100$.

The influence is bounded over all $x$: $\|\psi(x)\|_F=\min(\|x\|^2,4)\le 4$, and hence $\|\operatorname{IF}(x;T,F)\|_F\le 4+\|I_2\|_F=4+\sqrt{2}$. In contrast, the unweighted outer-product contribution $xx'$ grows quadratically as $\|x\|$ increases. The calculation shows how a robust functional can limit the contribution of very distant points; it does not make distance and influence interchangeable concepts.

Using Diagnostics Responsibly

A practical influence assessment should examine both the estimated scatter and the inferential quantity. For each case, compare the full-sample and deleted scatter estimates, then recompute the covariance statistic with its actual deleted-sample multiplier. Record the direction and magnitude of the changes, not just whether a case crosses an outlyingness threshold. A case that barely changes the statistic may be unusual but not influential for that analysis; a case that moves the statistic across a decision boundary deserves attention even if its distance is not exceptional.

Deletion diagnostics are sensitivity analyses, not automatic grounds for removing observations. A large change can indicate a fragile conclusion, a small sample, or an observation with important scientific meaning. Investigate the data and the study context before deciding how to proceed. Likewise, a small deletion change does not establish that an estimate is robust to every form of contamination: several cases can jointly alter an estimate in ways that single-case deletions do not reveal.

Common Pitfall

Equating a large distance with large influence. Distance is a measure of position relative to a chosen geometry; influence is a change in an estimator or inferential result. Robust weighting can keep the contribution of a distant point bounded.
Comparing statistics with inconsistent multipliers. Recomputing the discrepancy at the deleted estimate but retaining $n$ measures a fixed-multiplier change, not the actual change in a statistic defined with sample size. The actual deleted statistic uses $n-1$.
Interpreting a first-order approximation as an exact deletion result. Influence functions approximate local changes under regularity conditions. For a small or irregular sample, calculate the deletion estimates and statistics directly.
Using one scalar score as the whole diagnosis. A norm can hide which variance or covariance changed. Inspect the matrix difference and the resulting inferential change as well.

Key takeaway: Outlyingness concerns where an observation lies; influence concerns what changes when it is perturbed or deleted. For covariance inference, assess both robust scatter changes and the actual case-deleted statistic, including its changed sample-size multiplier.

Check Your Understanding

Work through these questions using the definitions and calculations above.

What does the case-deletion change $\hat{\Sigma}_n-\hat{\Sigma}_{(-i)}$ measure, and why might its matrix direction matter?
Which deletion-stability condition is needed to turn an asymptotically linear representation into a first-order casewise approximation at the $n^{-1}$ scale?
For a statistic $Q_n=nA_n$, express the actual deleted change using the fixed-multiplier difference and identify the extra term.
In the clipped-scatter example, why does the observation $(10,0)$ not contribute in proportion to its squared distance?
Give one reason that a large case-deletion change should be investigated rather than automatically used to justify removing the observation.