Projection-Pursuit Estimation Of Scatter

From Robust Projection Scales to Scatter

A scatter matrix describes the size and orientation of multivariate variation. One way to estimate it robustly is to examine the data one direction at a time: project the observations onto a line, compute a robust univariate scale, and search for directions with large projected scale. This is the central idea of projection-pursuit estimation of scatter.

A directional scale does not automatically specify a scatter matrix. For a matrix $S$, the scale in direction $u$ would have to be $\sqrt{u^T S u}$, so all squared directional scales would need to fit one quadratic form. In general, robust scales computed from different projections do not fit exactly. A sequential construction instead selects orthogonal directions of successively large projected scale and uses those scales as the matrix’s eigenvalues.

Definition: For observations $X_1,\ldots,X_n\in\mathbb{R}^p$, define the projected median absolute deviation in a unit direction $u$ by $$ s_n(u)=\frac{\operatorname{median}_{1\le i\le n}\left|u^T X_i-\operatorname{median}_{1\le j\le n}u^T X_j\right|}{c_0}, \qquad c_0=\Phi^{-1}(0.75)\approx 0.67449. $$ Here $\Phi$ is the standard normal distribution function. The division by $c_0$ makes $s_n(u)$ consistent for the standard deviation of a normal projection. Assume the relevant medians are uniquely defined and the scales used in the construction are positive.

The median absolute deviation (MAD) is a robust univariate scale: a single arbitrarily extreme observation does not necessarily make it arbitrarily large. The constant $c_0$ is a normal calibration, not a claim that the projected data must be normal. The scale is homogeneous: for any scalar $a$, the scale of the projection in direction $au$ is $|a|s_n(u)$. The construction below searches over unit directions, so this homogeneity fixes the scale of each chosen direction.

Definition: A sequential projection-pursuit scatter estimator is constructed as follows. Choose $v_1$ to maximize $s_n(u)$ over unit vectors $u$. Having chosen $v_1,\ldots,v_{k-1}$, choose $v_k$ to maximize $s_n(u)$ over unit vectors satisfying $u^T v_j=0$ for every $j<k$. Set $\lambda_k=s_n(v_k)^2$, and define $$ S_{\mathrm{PP}}=\sum_{k=1}^{p}\lambda_k v_kv_k^T. $$ Assume each maximizer is unique up to sign. A sign change in any $v_k$ leaves the matrix unchanged.

At each step, the feasible set is the unit sphere intersected with the orthogonal complement of the previously selected directions. It is a closed, bounded set, hence compact; if $s_n$ is continuous, a maximum exists on it. The sequence of maxima is nonincreasing because each successive feasible set is contained in the preceding one. Thus $\lambda_1\ge\cdots\ge\lambda_p$, and the selected $v_k$ are eigenvectors of the constructed matrix with corresponding eigenvalues $\lambda_k$. These statements concern the matrix produced by this sequential procedure; they do not assert that every directional scale is represented exactly by that matrix.

When Directional Scales Come From a Scatter Matrix

The construction has a particularly clear interpretation when the population directional scales follow a quadratic form. For example, if $X$ has an elliptical distribution with scatter matrix $\Sigma$, and the chosen univariate scale is calibrated so that the scale of $u^T X$ is $\sqrt{u^T\Sigma u}$, then maximizing projected scale is equivalent to maximizing a Rayleigh quotient.

Theorem: Let $S$ be a symmetric positive-definite $p\times p$ matrix, and suppose a directional scale satisfies $s(u)^2=u^TSu$ for every unit vector $u$. The sequential projection-pursuit procedure selects an orthonormal eigenbasis of $S$, in order of nonincreasing eigenvalues, and its constructed matrix is $S$.

Proof. By the spectral theorem, write $S=Q\operatorname{diag}(\mu_1,\ldots,\mu_p)Q^T$, where $Q$ is orthogonal and $\mu_1\ge\cdots\ge\mu_p>0$. For any unit vector $u$, write $a=Q^Tu$, also a unit vector. Then $$ u^TSu=\sum_{j=1}^p\mu_j a_j^2\le \mu_1\sum_{j=1}^p a_j^2=\mu_1. $$ Equality is attained by an eigenvector for $\mu_1$. If the largest eigenvalue is simple, its direction is the unique maximizing line. Once the first $k-1$ eigenvectors have been selected, any feasible unit vector lies in their orthogonal complement. Its expansion in the remaining eigenvectors gives $u^TSu\le\mu_k$, with equality at an eigenvector for $\mu_k$. Induction gives the ordered eigenbasis. The procedure assigns $\lambda_k=\mu_k$, and therefore $\sum_k\lambda_kv_kv_k^T=S$. If eigenvalues are repeated, maximizing directions within the tied eigenspace need not be unique, but every orthonormal eigenbasis of that eigenspace gives the same matrix. $\square$

Worked Example: Recovering an Elliptical Scatter Matrix

Suppose calibrated population scales in every direction satisfy $s(u)^2=u^T\Sigma u$, with $$ \Sigma=\begin{pmatrix}4&1\\1&2\end{pmatrix}. $$ Calculate three directional scales and identify the principal directions and eigenvalues.

For the coordinate directions $e_1=(1,0)^T$ and $e_2=(0,1)^T$, $$ s(e_1)^2=e_1^T\Sigma e_1=4,\qquad s(e_2)^2=e_2^T\Sigma e_2=2. $$ Thus $s(e_1)=2$ and $s(e_2)=\sqrt{2}$. For $w=(1,1)^T/\sqrt{2}$, $$ s(w)^2=w^T\Sigma w =\frac{1}{2}(4+1+1+2)=4. $$ The off-diagonal entry can be recovered from these directional quadratic values: $$ \Sigma_{12}=s(w)^2-\frac{s(e_1)^2+s(e_2)^2}{2}=4-\frac{4+2}{2}=1. $$ To find the principal directions, solve $\det(\Sigma-\lambda I)=0$: $$ (4-\lambda)(2-\lambda)-1=\lambda^2-6\lambda+7=0, \qquad \lambda=3\pm\sqrt{2}. $$ The eigenvector for $3+\sqrt{2}$ is the direction of $(1,\sqrt{2}-1)^T$, since the first row of $(\Sigma-\lambda I)v=0$ gives $v_2=(\lambda-4)v_1=(\sqrt{2}-1)v_1$. The other eigenvector is orthogonal to it and corresponds to $3-\sqrt{2}$. Sequential maximization therefore returns these directions and eigenvalues. The example also illustrates why two coordinate scales alone cannot determine a general scatter matrix: the diagonal projection supplies information about the off-diagonal entry.

Orthogonal Equivariance

A useful estimator should respond predictably to changes of coordinates. This construction has a precise invariance under orthogonal transformations: rotating or reflecting all observations rotates or reflects the estimated scatter in the same way. The claim relies on the direction search and its orthogonality constraints, not on normality.

Theorem: Let $Q$ be an orthogonal $p\times p$ matrix, and transform the observations to $X_i'=QX_i$. If the sequential maximizers are unique up to sign at every step, then $$ S_{\mathrm{PP}}(X_1',\ldots,X_n')=Q S_{\mathrm{PP}}(X_1,\ldots,X_n)Q^T. $$

Proof. For any unit vector $u$, the transformed projection is $u^TX_i'=u^TQX_i=(Q^Tu)^TX_i$. Because $Q^T$ preserves lengths, $Q^Tu$ is a unit vector. The projected observations, and hence their MAD, satisfy $s_n'(u)=s_n(Q^Tu)$. Thus maximizing the transformed scale over unit vectors is equivalent to maximizing the original scale, with directions mapped by $v\mapsto Qv$. If $v_1,\ldots,v_{k-1}$ are orthonormal, then $Qv_1,\ldots,Qv_{k-1}$ are orthonormal as well, so the feasible directions at step $k$ correspond under the same map. Induction gives transformed selected directions $Qv_k$ and the same values $\lambda_k$. Consequently, $$ S_{\mathrm{PP}}(X_1',\ldots,X_n') =\sum_{k=1}^p\lambda_k(Qv_k)(Qv_k)^T =Q\left(\sum_{k=1}^p\lambda_kv_kv_k^T\right)Q^T. $$ This proves the result. $\square$

Worked Example: Rotating the Data Rotates the Estimate

Suppose a two-dimensional data set has unique sequential directions $v_1=(1,0)^T$ and $v_2=(0,1)^T$, with scales $3$ and $1$. Its projection-pursuit scatter estimate is $$ S_{\mathrm{PP}}=9v_1v_1^T+v_2v_2^T =\begin{pmatrix}9&0\\0&1\end{pmatrix}. $$ Rotate the data through angle $\theta$, using $Q=\begin{pmatrix}\cos\theta&-\sin\theta\\\sin\theta&\cos\theta\end{pmatrix}$. The transformed directions are $Qv_1$ and $Qv_2$, with their scales still $3$ and $1$. Direct multiplication gives $$ QS_{\mathrm{PP}}Q^T =\begin{pmatrix} 9\cos^2\theta+\sin^2\theta&8\sin\theta\cos\theta\\ 8\sin\theta\cos\theta&9\sin^2\theta+\cos^2\theta \end{pmatrix}. $$ The eigenvalues remain $9$ and $1$; the principal directions rotate with the data. This is orthogonal equivariance in a numerical example.

Why Affine Equivariance Is Different

Orthogonal transformations preserve lengths and angles. A general nonsingular linear transformation $A$ does not: it changes the unit sphere into an ellipsoid and need not preserve perpendicularity. For a transformed direction $u$, its projection is $(A^Tu)^TX_i$, so its scale is $s_n(A^Tu)$. But the algorithm also requires successive directions to be perpendicular in the ordinary Euclidean inner product. After a general transformation, the images of perpendicular directions need not be perpendicular.

Consequently, the procedure as defined is not generally affine equivariant. It may be affine equivariant for special distributions or under additional construction and conditions, but affine equivariance does not follow from using robust univariate scales. This is a different property from the orthogonal equivariance proved above. A scale that is robust in each projection does not by itself guarantee that the combined matrix transforms as $AS_{\mathrm{PP}}A^T$.

Worked Example: A Robust Scale in One Projection

Consider five projected observations $-2,-1,0,1,100$. Their median is $0$, and the absolute deviations from that median are $2,1,0,1,100$. In sorted order these are $0,1,1,2,100$, so their median is $1$. The calibrated MAD is therefore $$ s_n=\frac{1}{0.67449}\approx 1.4826. $$ If the final projected observation is replaced by $10{,}000$, the median remains $0$, and the sorted absolute deviations become $0,1,1,2,10{,}000$. The MAD remains $1$, so the calibrated scale is still approximately $1.4826$. This calculation shows resistance to an extreme value in this particular projection; it does not imply that every multivariate estimate is unaffected by arbitrary contamination.

Common Pitfall and Practical Interpretation

A common mistake is to calculate a robust scale separately along the original coordinate axes and place the squared values on the diagonal of a scatter matrix. That construction ignores cross-direction information and depends on the selected coordinate system. The projection-pursuit procedure instead searches over directions, then combines the resulting orthonormal directions and scales. A related mistake is to conclude that the estimator is affine equivariant because each univariate scale is robust. Robustness of the one-dimensional ingredients and equivariance of the multivariate construction are separate properties.

The sequential search is useful when large projected variation is informative and classical covariance can be unduly affected by extreme observations. Its interpretation is direct: $v_1$ is a direction of largest robust projected scale, and later directions describe the largest remaining scales subject to orthogonality. In non-elliptical settings, these are the principal directions of the constructed estimate, not necessarily directions that reproduce every projection’s robust scale. Computation also requires an actual search over directions; a grid or numerical optimizer approximates the maximizers and can affect the result.

Key takeaway: Projection-pursuit scatter turns robust one-dimensional scales into ordered eigenvalues and orthogonal directions. The construction is orthogonally equivariant under the stated uniqueness conditions, while general affine equivariance requires more than robust scales alone.

Check Your Understanding

Use the definitions and results above to answer the following questions.

Why is the normal calibration constant for the MAD $\Phi^{-1}(0.75)$, rather than the median of the projected observations?
What condition on directional scales makes sequential projection pursuit recover an existing scatter matrix exactly?
Why do the sequentially selected scales form a nonincreasing sequence?
Which property of an orthogonal transformation is used in the proof of orthogonal equivariance?
Why does orthogonal equivariance not establish affine equivariance?