Breakdown Calculations For Classical Location And Scatter

Why Breakdown Is a Different Question From Sensitivity

The sample mean and covariance are familiar summaries, but their behavior under a small perturbation does not by itself answer how they respond to an arbitrary replacement. Breakdown asks a more severe question: how few observations can be replaced so that an estimate becomes arbitrarily large or otherwise unusable? For the classical mean and covariance, a single replacement suffices. The finite-sample calculation makes this failure precise.

Throughout, \(x_1,\ldots,x_n\) are vectors in \(\mathbb{R}^p\), with \(n\geq 2\). The sample covariance here uses divisor \(n\), rather than \(n-1\). This convention changes a fixed scale factor, but not whether the estimate is unbounded.

Definition: Given a fixed sample \(x=(x_1,\ldots,x_n)\) and an estimator \(T_n\), its finite-sample replacement breakdown fraction is the smallest \(m/n\) for which replacing \(m\) observations by arbitrary values can make the estimator arbitrarily large in norm. For a location estimator, use the Euclidean norm. For a scatter estimator, unboundedness means that the largest eigenvalue can be made arbitrarily large. Scatter breakdown is also often defined to include collapse of the smallest eigenvalue to zero; the calculations below establish breakdown even under the unboundedness criterion alone.

The sample mean and covariance are

\bar{x}=\frac{1}{n}\sum_{i=1}^n x_i, \qquad S=\frac{1}{n}\sum_{i=1}^n (x_i-\bar{x})(x_i-\bar{x})^{\mathsf T}.

A useful identity for the covariance is

S=\frac{1}{n}\sum_{i=1}^n x_i x_i^{\mathsf T}-\bar{x}\bar{x}^{\mathsf T}.

It follows by expanding each centered outer product and using \(\sum_i x_i=n\bar{x}\). We will use it to track the terms that grow when a replacement point moves far from the fixed observations.

One Replacement Makes the Mean Unbounded

Keep \(x_1,\ldots,x_{n-1}\) fixed and replace the final observation by \(T v\), where \(v\neq 0\) is fixed and \(T\) increases without bound. Writing \(s=\sum_{i=1}^{n-1}x_i\), the resulting mean is

\bar{x}(T)=\frac{s+Tv}{n}.

The triangle inequality gives \(\|\bar{x}(T)\|\geq T\|v\|/n-\|s\|/n\). The right-hand side tends to infinity. With no replacements, the mean of the fixed sample is finite. Thus one replacement is both sufficient and the smallest possible positive number of replacements.

Worked Example: Mean and Variance of Three Observations

Take the scalar observations \(0,2,T\), with \(T\) replacing the third observation. Their mean is

\bar{x}(T)=\frac{0+2+T}{3}=\frac{T+2}{3}.

The variance with divisor \(n=3\) is the average squared observation minus the squared mean:

S(T)=\frac{0^2+2^2+T^2}{3} -\left(\frac{T+2}{3}\right)^2 =\frac{3T^2+12-(T^2+4T+4)}{9} =\frac{2T^2-4T+8}{9}.

At \(T=10\), the mean is \(12/3=4\), and the variance is \((200-40+8)/9=168/9=56/3\). Directly, the squared deviations from \(4\) are \(16,4,36\), whose average is also \(56/3\). As \(T\) tends to infinity, the mean grows linearly and the variance grows quadratically. A single replaced observation makes both estimates unbounded.

One Replacement Makes the Covariance Unbounded

The same replacement also forces the largest eigenvalue of the multivariate sample covariance to diverge. To see the leading term, leave \(x_1,\ldots,x_{n-1}\) fixed and set \(x_n=Tv\), with \(v\neq0\). Applying the covariance identity gives

S(T)=\frac{1}{n}\left(\sum_{i=1}^{n-1}x_i x_i^{\mathsf T} +T^2vv^{\mathsf T}\right) -\frac{1}{n^2}(s+Tv)(s+Tv)^{\mathsf T}, \qquad s=\sum_{i=1}^{n-1}x_i.

The coefficient of \(T^2vv^{\mathsf T}\) is \(1/n-1/n^2=(n-1)/n^2\). All remaining terms are fixed or linear in \(T\). In particular, taking the quadratic form in direction \(v\) yields

v^{\mathsf T}S(T)v =\frac{n-1}{n^2}T^2\|v\|^4+O(T).

Here \(O(T)\) denotes terms whose absolute value is bounded by a constant times \(1+|T|\), with the fixed observations and \(v\) held constant. Since \(S(T)\) is positive semidefinite, its largest eigenvalue is at least its Rayleigh quotient in direction \(v\):

\lambda_{\max}(S(T)) \geq \frac{v^{\mathsf T}S(T)v}{\|v\|^2} =\frac{n-1}{n^2}T^2\|v\|^2+O(T) \longrightarrow\infty.

The quadratic term has a positive coefficient because \(n\geq2\) and \(v\neq0\). This proves that one replacement is sufficient for covariance unboundedness. As with the mean, zero replacements leave a fixed finite estimate, so the minimum number is one.

Theorem: For any fixed finite sample and \(n\geq2\), the replacement breakdown fraction of the sample mean is \(1/n\). The sample covariance also has unboundedness replacement breakdown fraction \(1/n\): replacing one observation by \(Tv\), for any fixed \(v\neq0\), makes its largest eigenvalue tend to infinity as \(|T|\) tends to infinity.

Proof. With no replacements, both estimates are fixed and finite. For one replacement, the mean formula \(\bar{x}(T)=(s+Tv)/n\) has norm tending to infinity. The covariance expansion above has a positive quadratic term in direction \(v\), so its largest eigenvalue tends to infinity. Thus one replacement suffices for each estimator, and no smaller positive number of replacements exists. Dividing the required count, one, by \(n\) gives \(1/n\). \(\square\)

Worked Multivariate Calculations

Worked Example: A Bivariate Covariance Matrix With One Extreme Point

Consider the four observations \((0,0),(1,0),(0,1),(T,0)\). The final point is the replacement. The coordinate means are \((T+1)/4\) and \(1/4\). Using \(S_{jk}=n^{-1}\sum_i x_{ij}x_{ik}-\bar{x}_j\bar{x}_k\), calculate each entry.

For the first coordinate, the average of the squares is \((1+T^2)/4\), so

S_{11}=\frac{1+T^2}{4}-\left(\frac{T+1}{4}\right)^2 =\frac{3T^2-2T+3}{16}.

For the second coordinate, the average squared value is \(1/4\), giving \(S_{22}=1/4-(1/4)^2=3/16\). Every observation has product of coordinates zero, so

S_{12}=S_{21}=0-\frac{T+1}{4}\frac{1}{4} =-\frac{T+1}{16}.

Consequently,

S(T)=\frac{1}{16} \begin{pmatrix} 3T^2-2T+3 & -(T+1)\\ -(T+1) & 3 \end{pmatrix}.

Its determinant is

\det S(T) =\frac{3(3T^2-2T+3)-(T+1)^2}{256} =\frac{T^2-T+1}{32} =\frac{(T-\tfrac12)^2+\tfrac34}{32}>0.

In particular, the covariance is nonsingular for every real \(T\). Also, \(\lambda_{\max}(S(T))\geq S_{11}(T)\), and \(S_{11}(T)\) tends to infinity quadratically. Thus the divergence is not an artifact of a zero determinant: a single point drives the scatter magnitude without bound.

Worked Example: A Bivariate Cloud With a Nondegenerate Fixed Part

Take \((0,0),(1,0),(0,1),(1,1)\) as the fixed observations and replace a fifth point by \((T,0)\). The coordinate means are \((T+2)/5\) and \(2/5\). The average squared first coordinate is \((T^2+2)/5\), and the average squared second coordinate is \(2/5\). Therefore,

S_{11}=\frac{T^2+2}{5}-\left(\frac{T+2}{5}\right)^2 =\frac{4T^2-4T+6}{25}, \qquad S_{22}=\frac{2}{5}-\left(\frac{2}{5}\right)^2 =\frac{6}{25}.

Only the fixed point \((1,1)\) has a nonzero coordinate product, so the average product is \(1/5\). Subtracting the product of the means gives

S_{12}=S_{21} =\frac{1}{5}-\frac{T+2}{5}\frac{2}{5} =\frac{1-2T}{25}.

The variance in the first coordinate alone provides a lower bound on the largest eigenvalue: \(\lambda_{\max}(S(T))\geq S_{11}(T)\). Since \((4T^2-4T+6)/25\) tends to infinity, this one replacement makes the bivariate scatter unbounded. The four fixed points span two dimensions, so the example does not rely on starting with a collinear fixed cloud.

Finite-Sample and Asymptotic Interpretation

For a sample of size \(n\), the finite-sample replacement fraction for each classical estimate is \(1/n\). As \(n\) grows, this fraction tends to zero. Thus their asymptotic replacement breakdown point is zero: no fixed positive fraction is required in the large-sample limit, since even one replacement is enough at each finite \(n\).

The conclusion concerns arbitrary replacements, not small measurement errors. A point may move only a modest distance and have a modest effect; breakdown allows its replacement value to be chosen without any bound. The calculations also distinguish location from scatter: the mean grows at order \(T\), while the covariance has a leading term at order \(T^2\).

This is why high-breakdown procedures for location and scatter are useful when a few observations may be grossly contaminated. The classical estimates remain valuable under appropriate models, but their finite-sample breakdown calculation shows that their reliability cannot be judged from the number of contaminated points alone: the magnitude of those points is unrestricted.

Common Mistake

Wrong reasoning: “Only one of \(n\) observations is replaced, so its effect must be diluted by a factor of \(n\) and cannot dominate.”
Correction: The mean contribution is \(Tv/n\), which is still unbounded as \(T\) grows. In the covariance, the quadratic contribution is \((n-1)T^2vv^{\mathsf T}/n^2\), with a positive coefficient. Dividing by \(n\) does not bound either expression when the replacement value itself is unrestricted.

Key takeaway: For the classical sample mean and covariance, one arbitrarily distant replacement is enough to cause unboundedness. Their finite-sample replacement breakdown fraction is \(1/n\), so their asymptotic breakdown point is zero.

Check Your Understanding

Use the replacement calculations and definitions above to answer the following.

For a sample of size \(n=20\), what is the finite-sample replacement breakdown fraction of the sample mean?
In the covariance expansion for \(x_n=Tv\), what is the coefficient multiplying \(T^2vv^{\mathsf T}\)?
Why does a positive factor of \(1/n\) not prevent the sample mean from becoming unbounded?
In the four-point bivariate example, calculate \(S_{11}\) at \(T=2\).
What is the asymptotic replacement breakdown point of the classical covariance as \(n\) tends to infinity?