Pairwise Robust Covariance Estimation

Why Estimate Covariance Pair by Pair?

A covariance matrix summarizes both the scale of each variable and the direction and strength of pairwise association. Classical covariance is sensitive to extreme observations: one unusual case can have a large effect on products of deviations. A robust alternative can replace the ordinary standard deviations of suitable linear combinations with robust scale estimates. The key identity uses the scales of a sum and a difference.

This construction is attractive because it reduces each off-diagonal entry to two univariate scale calculations. But it also has an important limitation: estimating each pair separately does not ensure that all the resulting entries are mutually compatible as one multivariate scatter matrix.

The Sum-and-Difference Identity

For random variables \(X\) and \(Y\) with finite second moments, the covariance is determined by the difference between the variances of their sum and difference.

Theorem: If \(E[X^2]<\infty\) and \(E[Y^2]<\infty\), then

\operatorname{Cov}(X,Y) = \frac{\operatorname{Var}(X+Y)-\operatorname{Var}(X-Y)}{4}.

Proof. The finite second-moment assumptions ensure that all variances and covariances below exist. By bilinearity of covariance,

\begin{aligned} \operatorname{Var}(X+Y) &=\operatorname{Var}(X)+\operatorname{Var}(Y)+2\operatorname{Cov}(X,Y),\\ \operatorname{Var}(X-Y) &=\operatorname{Var}(X)+\operatorname{Var}(Y)-2\operatorname{Cov}(X,Y). \end{aligned}

Subtracting the second equality from the first gives \(4\operatorname{Cov}(X,Y)\). Dividing by \(4\) proves the identity. \(\square\)

The identity suggests a robust plug-in estimate. For an observed pair of variables, calculate robust scales on the paired sums and differences, square those scales to put them on a variance scale, and take their difference.

Definition: Let \(\hat s_{ij,+}\) be a robust univariate scale estimate calculated from the paired observations of \(X_i+X_j\), and let \(\hat s_{ij,-}\) be the corresponding scale estimate for \(X_i-X_j\). The pairwise robust covariance estimate is

\hat c_{ij}^{\,\mathrm{GK}} = \frac{\hat s_{ij,+}^{\,2}-\hat s_{ij,-}^{\,2}}{4}.

The diagonal entries are commonly set to the squared robust marginal scales, \(\hat c_{ii}=\hat s_i^2\). The resulting symmetric matrix is called a pairwise robust covariance estimate or a Gnanadesikan–Kettenring-type estimate.

Here a robust scale is understood to be location invariant and scale equivariant: shifting the observations does not change the scale, and multiplying them by \(b\) multiplies the scale by \(|b|\). It should also be invariant to a sign change, so the estimates for \(X_i-X_j\) and \(X_j-X_i\) agree. Median absolute deviation and other robust scale procedures can supply the univariate estimates; the choice of scale procedure affects the resulting pairwise estimates.

Worked Example: Recovering Covariance Under a Normal Model

Suppose \(X\) and \(Y\) have a bivariate normal distribution with standard deviations \(2\) and \(3\), and correlation \(0.6\). Assume the scale procedure is calibrated so that its population scale equals the standard deviation under normality.

First, the covariance is \(0.6(2)(3)=3.6\). The variances of the sum and difference are therefore

\begin{aligned} \operatorname{Var}(X+Y)&=2^2+3^2+2(3.6)=20.2,\\ \operatorname{Var}(X-Y)&=2^2+3^2-2(3.6)=5.8. \end{aligned}

Under the stated normal calibration, the corresponding population robust scales are \(\sqrt{20.2}\) and \(\sqrt{5.8}\). Substitution gives

\frac{(\sqrt{20.2})^2-(\sqrt{5.8})^2}{4} = \frac{20.2-5.8}{4} = \frac{14.4}{4} = 3.6.

Thus the pairwise identity recovers the covariance in this model. The normal calibration matters: it makes the robust scale of each normal linear combination equal to that combination’s standard deviation.

What the Robust Estimate Targets

Replacing standard deviations with robust scales does not automatically preserve the classical covariance identity for every distribution. For an arbitrary distribution, a robust scale of \(X+Y\) need not equal \(\sqrt{\operatorname{Var}(X+Y)}\). Consequently, the difference of the two squared robust scales generally defines a robust pairwise association functional, not necessarily the classical covariance.

A precise target is obtained by writing \(S(F_Z)\) for the population value of the chosen univariate scale functional applied to a random variable \(Z\). The pairwise functional is

C_{ij}^{S}(F) = \frac{S^2(F_{X_i+X_j})-S^2(F_{X_i-X_j})}{4}.

If the scale estimates converge in probability to their population scale values, then the sample pairwise estimate converges to this functional. To see this, suppose \(\hat s_{ij,+}\xrightarrow{p}s_{ij,+}\) and \(\hat s_{ij,-}\xrightarrow{p}s_{ij,-}\), with both limits finite. Squaring and subtraction are continuous operations, so the continuous mapping theorem gives

\hat c_{ij}^{\,\mathrm{GK}} \xrightarrow{p} \frac{s_{ij,+}^{\,2}-s_{ij,-}^{\,2}}{4} = C_{ij}^{S}(F).

This consistency statement identifies the target; it does not say that the target equals classical covariance. Equality does hold when the scale functional returns the standard deviation for the relevant linear combinations, as with a correctly calibrated scale under a multivariate normal model. It can also hold under other models that give the chosen scale a common proportional relationship to the standard deviations of all such combinations, with the calibration accounting for that proportionality.

Worked Example: Computing a Pairwise Estimate From Robust Scales

Suppose the robust scale calculation for a pair of variables returns \(\hat s_{ij,+}=2.4\) for their sum and \(\hat s_{ij,-}=1.6\) for their difference. Suppose also that their marginal robust scales are \(1\) and \(2\).

Square the two combination scales before taking the difference:

\hat c_{ij}^{\,\mathrm{GK}} = \frac{2.4^2-1.6^2}{4} = \frac{5.76-2.56}{4} = \frac{3.20}{4} = 0.8.

Using the marginal robust scales as the diagonal entries gives the pairwise matrix

\begin{pmatrix} 1 & 0.8\\ 0.8 & 4 \end{pmatrix}.

Its determinant is \(1(4)-0.8^2=3.36>0\), and its diagonal entries are positive, so this two-variable matrix is positive definite. The corresponding standardized pairwise entry is \(0.8/(1\cdot2)=0.4\). These calculations establish properties of this pair; they do not ensure that adding estimates for other pairs will preserve positive semidefiniteness.

Why the Assembled Matrix Can Fail

A true covariance matrix cannot have arbitrary entries. For a random vector \(X\) with finite second moments and covariance matrix \(\Sigma\), every vector \(a\) satisfies

a^\mathsf{T}\Sigma a = \operatorname{Var}(a^\mathsf{T}X) \geq 0.

Thus \(\Sigma\) is positive semidefinite. In particular, its entries must work together to give nonnegative variance to every linear combination. The sum-and-difference calculation constrains one pair at a time, but does not enforce all these joint constraints. Robust estimates for different pairs can respond differently to the data, so combining them may produce entries that are individually plausible but jointly incompatible.

Theorem: A pairwise matrix formed from robust covariance estimates is not guaranteed to be positive semidefinite merely because each diagonal entry is positive and each two-variable pair is individually admissible.

Worked Example: Individually Admissible Entries, Indefinite Matrix

Consider three variables with marginal robust scales all equal to \(1\). Suppose the pairwise sum-and-difference scale calculations give the following squared scales:

\begin{array}{c|cc|c} \text{Pair} & \hat s_{ij,+}^{\,2} & \hat s_{ij,-}^{\,2} & \hat c_{ij}^{\,\mathrm{GK}}\\ (1,2) & 3.8 & 0.2 & (3.8-0.2)/4=0.9\\ (1,3) & 3.8 & 0.2 & (3.8-0.2)/4=0.9\\ (2,3) & 0.2 & 3.8 & (0.2-3.8)/4=-0.9 \end{array}

Every scale shown is positive. Each two-variable matrix has diagonal entries \(1,1\) and off-diagonal entry either \(0.9\) or \(-0.9\). Its determinant is \(1-0.9^2=0.19>0\), so each pair separately gives a positive definite two-variable matrix.

Putting the three pair estimates together gives

R= \begin{pmatrix} 1 & 0.9 & 0.9\\ 0.9 & 1 & -0.9\\ 0.9 & -0.9 & 1 \end{pmatrix}.

Its determinant is

\det(R) = 1+2(0.9)(0.9)(-0.9)-(0.9^2+0.9^2+(-0.9)^2) = 1-1.458-2.43 = -2.888.

A positive-semidefinite matrix must have a nonnegative determinant, so \(R\) is not positive semidefinite. This example exhibits the compatibility problem: every pair passes its own two-variable check, yet the assembled three-variable matrix fails a necessary global check. It is an algebraic illustration of what pairwise assembly does not guarantee; it does not assert that these particular scale outputs must arise from every common complete data set.

The issue is structural, not a defect in the covariance identity. The identity is exact for classical variances, but robust scale functionals need not preserve the cross-pair constraints that a single multivariate covariance matrix obeys. Pairwise estimation can still be useful when robustness and simplicity are priorities, but any use that requires a valid covariance matrix must account for this possibility. Enforcing positive semidefiniteness changes the assembled entries, so such a step is a substantive modification rather than an automatic consequence of the pairwise formulas.

Common Mistakes

Subtracting scales instead of squared scales. The identity concerns variances, so the estimate is \((\hat s_+^2-\hat s_-^2)/4\), not \((\hat s_+-\hat s_-)/4\). The latter also has the wrong units for covariance.
Calling every robust pairwise functional classical covariance. The equality with covariance requires the relevant scale values to represent standard deviations, as under the stated normal calibration. Otherwise, the robust functional may have a different target.
Assuming valid pairs guarantee a valid matrix. Positive definiteness of every two-variable block does not impose all the constraints on combinations of three or more variables. Check the assembled matrix when positive semidefiniteness is required.
Ignoring how pairs are formed. The sum and difference scales must be calculated from the paired values of the same two variables. The identity is about the distribution of those particular linear combinations.

Key takeaway: Pairwise robust covariance uses the difference of robust variances of sums and differences. It can estimate classical covariance under suitable scale calibration and distributional conditions, but assembling pair estimates alone does not guarantee a positive-semidefinite scatter matrix.

Check Your Understanding

Answer these questions by using the identity and checking which conditions or limitations apply.

Derive the pairwise covariance estimate from the two variance expansions for \(X+Y\) and \(X-Y\).
If the estimated robust scales of a pair’s sum and difference are \(3\) and \(1\), respectively, what is the pairwise robust covariance estimate?
Why does normal calibration allow the sum-and-difference estimate to recover covariance under a bivariate normal model?
Can three positive diagonal entries and three individually positive-definite two-variable blocks guarantee that the assembled three-variable matrix is positive semidefinite? Explain.
What is the target of the pairwise estimate when the selected robust scale does not equal standard deviation for the relevant distributions?