Influence Diagnostics For The Sample Mean And Covariance

Why Examine the Mean and Covariance?

The sample mean and covariance are natural summaries of multivariate data, but an observation far from the main body of a data set can substantially alter either one. An influence function quantifies the first-order effect of adding an infinitesimal amount of probability at a specified point. For the mean, that effect grows linearly with distance; for covariance, it grows quadratically. These calculations make precise why extreme observations can strongly affect classical location and scatter summaries.

We work with a random vector $X\in\mathbb{R}^p$, with distribution $F$, finite second moments, mean $\mu=E[X]$, and covariance matrix $\Sigma=\operatorname{Cov}(X)$. Vectors are columns, and $a^T$ denotes the transpose of $a$. The covariance matrix is assumed finite; when a standardized distance using $\Sigma^{-1}$ is used, we additionally require $\Sigma$ to be positive definite.

Definition: For a functional $T$ and distribution $F$, the influence function at a point $x$ is the right-hand derivative $$ \operatorname{IF}(x;T,F) = \left.\frac{d}{d\varepsilon} T\big((1-\varepsilon)F+\varepsilon\Delta_x\big) \right|_{\varepsilon=0+}, $$ when this derivative exists. Here $\Delta_x$ is the distribution placing probability one at $x$, and $0\leq\varepsilon\leq1$. The contaminated distribution replaces a fraction $\varepsilon$ of the original probability with a point mass at $x$.

The influence function is a local diagnostic: it describes the derivative at zero contamination, not the exact effect of a large replacement. For a sample, the empirical distribution assigns mass $1/n$ to each observation, so evaluating influence at an observation indicates its first-order contribution. The sample covariance convention with denominator $n-1$ differs from the empirical covariance functional with denominator $n$ by a factor $n/(n-1)$; that factor tends to one and does not change the population influence function.

Deriving the Two Influence Functions

Write $F_{\varepsilon,x}=(1-\varepsilon)F+\varepsilon\Delta_x$ and $d=x-\mu$. The mean under contamination is obtained directly by averaging under the two mixture components. The covariance can be derived from the mixture’s within-component and between-component variation.

Theorem: If $F$ has finite second moments, then the mean functional $T_\mu(F)=E_F[X]$ and covariance functional $T_\Sigma(F)=\operatorname{Cov}_F(X)$ have influence functions $$ \operatorname{IF}(x;T_\mu,F)=x-\mu $$ and $$ \operatorname{IF}(x;T_\Sigma,F) =(x-\mu)(x-\mu)^T-\Sigma. $$

Proof. The contaminated mean is $$ \mu_\varepsilon=E_{F_{\varepsilon,x}}[X] =(1-\varepsilon)\mu+\varepsilon x =\mu+\varepsilon d. $$ Differentiating with respect to $\varepsilon$ at zero gives $\operatorname{IF}(x;T_\mu,F)=d$.

For the covariance, the two mixture components have means $\mu$ and $x$, covariances $\Sigma$ and zero, and mixture mean $\mu_\varepsilon=\mu+\varepsilon d$. The mixture covariance is the weighted sum of the component covariances plus the weighted outer products of each component mean’s deviation from the mixture mean. The component means differ from $\mu_\varepsilon$ by $-\varepsilon d$ and $(1-\varepsilon)d$, respectively. Therefore $$ \begin{aligned} \Sigma_\varepsilon &=(1-\varepsilon)\Sigma +(1-\varepsilon)(-\varepsilon d)(-\varepsilon d)^T +\varepsilon\big((1-\varepsilon)d\big)\big((1-\varepsilon)d\big)^T\\ &=(1-\varepsilon)\Sigma+\varepsilon(1-\varepsilon)dd^T. \end{aligned} $$ Thus $$ \left.\frac{d}{d\varepsilon}\Sigma_\varepsilon\right|_{\varepsilon=0+} =-\Sigma+dd^T, $$ which is the stated covariance influence function. $\square$

Worked Example: Mean and Covariance Influence at a Specified Point

Suppose $X$ has mean $\mu=(1,-1)^T$ and covariance $\Sigma=\begin{pmatrix}4&1\\1&2\end{pmatrix}$. Calculate both influence functions at $x=(4,3)^T$.

First, the displacement from the population mean is $$ d=x-\mu=(4-1,\;3-(-1))^T=(3,4)^T. $$ Hence the mean influence is $$ \operatorname{IF}(x;T_\mu,F)= \begin{pmatrix}3\\4\end{pmatrix}. $$ For covariance, form the outer product and subtract $\Sigma$: $$ dd^T= \begin{pmatrix}3\\4\end{pmatrix} \begin{pmatrix}3&4\end{pmatrix} = \begin{pmatrix}9&12\\12&16\end{pmatrix}, \qquad \operatorname{IF}(x;T_\Sigma,F) = \begin{pmatrix}9&12\\12&16\end{pmatrix} - \begin{pmatrix}4&1\\1&2\end{pmatrix} = \begin{pmatrix}5&11\\11&14\end{pmatrix}. $$ The covariance influence is a matrix, not a single change in variance: its off-diagonal entries describe first-order change in covariance as well as its diagonal entries’ changes in variances.

How the Influence Grows with Distance

The mean influence has Euclidean norm $\|x-\mu\|$, which grows linearly as $x$ moves away from $\mu$. For covariance, write $x=\mu+ru$, where $u$ is a unit vector and $r\geq0$. Then $$ \operatorname{IF}(x;T_\Sigma,F)=r^2uu^T-\Sigma. $$ The first term grows quadratically in $r$. In particular, in the direction $u$, its quadratic form is $$ u^T\operatorname{IF}(x;T_\Sigma,F)u =r^2-u^T\Sigma u. $$ Since $u^T\Sigma u$ is finite, this directional covariance influence grows without bound as $r$ increases.

Result: When the contamination point can be any point in $\mathbb{R}^p$, the gross-error sensitivity—the supremum of an influence-function norm over $x$—is infinite for both the mean and covariance. For the mean use the Euclidean norm; for covariance, any usual matrix norm such as the Frobenius or operator norm has unbounded growth.

To see the covariance conclusion directly, for any unit vector $u$, the operator norm of a matrix $A$ is at least $|u^TAu|$. Taking $x=\mu+ru$ gives $$ \left\|\operatorname{IF}(x;T_\Sigma,F)\right\|_{\mathrm{op}} \geq \left|r^2-u^T\Sigma u\right|, $$ which tends to infinity with $r$. For the mean, $\|\operatorname{IF}(x;T_\mu,F)\|=\|x-\mu\|=r$, also unbounded. This proves the stated result. $\square$

When $\Sigma$ is positive definite, the directional structure can also be expressed in standardized coordinates. Let $z=\Sigma^{-1/2}(x-\mu)$. Transforming the covariance influence by $\Sigma^{-1/2}$ on both sides gives $zz^T-I$. Along the direction $z$, its eigenvalue is $\|z\|^2-1$; in directions orthogonal to $z$, its eigenvalue is $-1$. Thus a point with large Mahalanobis distance produces a large positive influence in its radial direction, even though the influence matrix can also have negative eigenvalues.

Worked Example: Standardized Directional Influence

Let $\mu=(0,0)^T$, $\Sigma=\begin{pmatrix}1&0\\0&4\end{pmatrix}$, and consider $x=(0,6)^T$. The displacement is $d=(0,6)^T$, so the unstandardized covariance influence is $$ dd^T-\Sigma = \begin{pmatrix}0&0\\0&36\end{pmatrix} - \begin{pmatrix}1&0\\0&4\end{pmatrix} = \begin{pmatrix}-1&0\\0&32\end{pmatrix}. $$ The standardized displacement is $z=\Sigma^{-1/2}d=(0,3)^T$, with squared length $9$. Accordingly, the standardized influence is $$ zz^T-I= \begin{pmatrix}0&0\\0&9\end{pmatrix} - \begin{pmatrix}1&0\\0&1\end{pmatrix} = \begin{pmatrix}-1&0\\0&8\end{pmatrix}. $$ Its eigenvalue in the displacement direction is $9-1=8$; in the perpendicular direction it is $-1$. The point is three standard deviations from the mean along the second coordinate, and its radial covariance effect is much larger than that of a point close to the center.

Influence as a Sample Diagnostic

For an empirical distribution, the mean influence identifies observations far from the sample center, while covariance influence also accounts for how the observation’s direction contributes to the scatter matrix. An observation can increase variation strongly in one direction and alter off-diagonal covariances. The following example compares the local influence calculation with an exact finite-contamination calculation.

Worked Example: A Distant Point Added to a Four-Point Data Set

Consider the four equally weighted points $(0,0)^T,(2,0)^T,(0,2)^T,(2,2)^T$. Their empirical mean is $\mu=(1,1)^T$. Each centered coordinate takes values $-1$ and $1$, and the products of the two centered coordinates cancel across the four points. Thus the empirical covariance with divisor $n$ is $\Sigma=I$.

For the point $x=(10,0)^T$, the displacement is $d=(9,-1)^T$. Its mean influence and covariance influence are $$ \operatorname{IF}(x;T_\mu,F_n)= \begin{pmatrix}9\\-1\end{pmatrix}, \qquad \operatorname{IF}(x;T_\Sigma,F_n) = \begin{pmatrix}81&-9\\-9&1\end{pmatrix} - \begin{pmatrix}1&0\\0&1\end{pmatrix} = \begin{pmatrix}80&-9\\-9&0\end{pmatrix}. $$ For an exact check, contaminate the empirical distribution with this point at weight $\varepsilon=0.05$. The new mean is $$ \mu_\varepsilon=\mu+0.05d=(1.45,\;0.95)^T. $$ Using the exact covariance formula derived above, $$ \Sigma_\varepsilon =0.95I+0.05(0.95) \begin{pmatrix}81&-9\\-9&1\end{pmatrix} = \begin{pmatrix}4.7975&-0.4275\\-0.4275&0.9975\end{pmatrix}. $$ The first-order covariance change would be $0.05\operatorname{IF}(x;T_\Sigma,F_n)=\begin{pmatrix}4&-0.45\\-0.45&0\end{pmatrix}$. It is close in scale to, but not identical to, the exact change. The discrepancy is the second-order term $-0.05^2dd^T$. The example shows both the large local effect of a distant point and why an influence function is a derivative rather than an exact finite-contamination formula.

Common Mistakes and Interpretation

Leaving out the centering correction. The covariance influence is not just $(x-\mu)(x-\mu)^T$. The term $-\Sigma$ is essential: even observations near the center change the mixture weights and hence the covariance.
Using the contaminated mean in place of the original center without differentiating. The covariance is centered at $\mu_\varepsilon$ under contamination. The mixture calculation accounts for that shift; omitting it can produce an incorrect derivative.
Assuming an influence matrix must be positive semidefinite. Covariance matrices are positive semidefinite, but their derivatives need not be. The term $-\Sigma$ can make the influence negative in some directions, as in the standardized calculation.
Treating the derivative as an exact change. For small $\varepsilon$, $T(F_{\varepsilon,x})$ is approximated by $T(F)+\varepsilon\operatorname{IF}(x;T,F)$, with a remainder that need not be negligible for a substantial contamination fraction.
Calling an observation influential only because it is far from the origin. The relevant displacement for the mean and covariance is from $\mu$; for standardized interpretation, account for the scale and orientation in $\Sigma$.

These calculations explain the sensitivity of classical mean and covariance summaries: their influence functions are unbounded, so sufficiently distant contamination can have arbitrarily large first-order effect. Influence diagnostics are useful for identifying which observations may drive location or scatter estimates and in which directions. They do not, by themselves, establish that an observation is erroneous; an extreme observation may be valid and scientifically important.

Key takeaway: The mean influence is $x-\mu$, while the covariance influence is $(x-\mu)(x-\mu)^T-\Sigma$. Their linear and quadratic growth, respectively, shows that both classical summaries are locally sensitive to extreme observations.

Check Your Understanding

Use the definitions and derivations above to answer these questions.

For a distribution with mean $\mu$, what is the mean influence function at $x$, and how does its norm grow with the distance from $\mu$?
Why does the covariance influence function contain the term $-\Sigma$?
If $x=\mu+ru$ for a unit vector $u$, write the covariance influence and its quadratic form in direction $u$.
Can a covariance influence matrix have a negative eigenvalue even though every covariance matrix is positive semidefinite? Explain.
What additional term distinguishes the exact covariance change under contamination from its first-order approximation?