Sparse PCA Under Correlated Noise

Why Correlated Noise Changes the Sparse PCA Problem

Sparse PCA seeks a principal direction with only a small number of nonzero coordinates. The usual spherical-noise model makes the population covariance a multiple of the identity plus a low-rank signal, so the noise does not favor particular directions. With correlated noise, however, the background covariance itself has preferred directions. The leading eigenvector of the observed covariance can therefore reflect both the signal and the noise.

Whitening removes the background covariance at the population level, but it also changes coordinates. A sparse loading in the original variables can become dense after whitening. Thus, “whiten first, then run sparse PCA” is not automatically a method for recovering the original sparse loading. The key distinction is between sparsity in the scientific variables and sparsity in transformed coordinates.

Definition: Consider independent, mean-zero observations $X_1,\ldots,X_n\in\mathbb{R}^p$ with covariance $$ \Sigma=\Sigma_0+\lambda vv^T, $$ where the background covariance $\Sigma_0$ is known and positive definite, $\lambda>0$, and $v$ is a unit vector. A sparse-spike model assumes that $v$ has at most $s$ nonzero coordinates, with $s\ll p$. The support of $v$ is $\operatorname{supp}(v)=\{j:v_j\ne 0\}$. The spike is the rank-one covariance contribution $\lambda vv^T$; the sparse target is the loading $v$ in the original coordinates.

This model is also generated by $X_i=\sqrt{\lambda}\,z_i v+\varepsilon_i$, when $z_i$ has mean zero and variance one, $\varepsilon_i$ has covariance $\Sigma_0$, and the two are independent. The observations need not be Gaussian for this covariance statement. Gaussianity will be specified when deriving a likelihood below.

Ordinary PCA and Whitening

Ordinary PCA uses eigenvectors of $\Sigma$. If $\Sigma_0$ is not spherical, its leading eigenvectors need not align with $v$, even at the population level. Whitening instead applies an invertible matrix $A$ satisfying $A\Sigma_0A^T=I$. The transformed covariance is $$ A\Sigma A^T=I+\lambda(Av)(Av)^T. $$ Its signal direction is $Av$, not generally $v$. If $A$ mixes coordinates, the sparsity pattern can change.

Theorem: If $A\Sigma_0A^T=I$, then the covariance of $AX_i$ is $I+\lambda(Av)(Av)^T$. Its unique signal eigendirection is $Av$, with eigenvalue $1+\lambda\|Av\|_2^2$, while all orthogonal directions have eigenvalue $1$. Consequently, sparse PCA on whitened observations targets the transformed loading $Av$. To interpret it in the original coordinates, the transformation must be undone, and that inverse mapping need not preserve sparsity.

Proof. By covariance transformation, $\operatorname{Cov}(AX_i)=A\Sigma A^T$. Substituting the model and the whitening identity gives $A\Sigma A^T=I+\lambda(Av)(Av)^T$. Write $w=Av$, which is nonzero because $A$ is invertible and $v\ne0$. Then $(I+\lambda ww^T)w=(1+\lambda w^Tw)w$. For any $u$ orthogonal to $w$, $w^Tu=0$, so $(I+\lambda ww^T)u=u$. Since $\lambda\|w\|_2^2>0$, the eigenvalue along $w$ is strictly larger than the eigenvalue on its orthogonal complement. The signal eigendirection is therefore $Av$. $\square$

Worked Example: Whitening Turns a One-Coordinate Spike into a Dense One

Let $\Sigma_0=\begin{pmatrix}1&0.6\\0.6&1\end{pmatrix}$, $v=(1,0)^T$, and $\lambda=1$. The loading $v$ is one-sparse. A Cholesky factorization is $\Sigma_0=LL^T$, where $L=\begin{pmatrix}1&0\\0.6&0.8\end{pmatrix}$, since $0.6^2+0.8^2=1$. Thus $A=L^{-1}=\begin{pmatrix}1&0\\-0.75&1.25\end{pmatrix}$ whitens the noise: $A\Sigma_0A^T=I$.

The transformed loading is $Av=(1,-0.75)^T$. Both coordinates are nonzero, so this whitening has changed a one-sparse loading into a two-coordinate loading. Its squared norm is $1+0.75^2=1.5625$, also equal to $v^T\Sigma_0^{-1}v=1/(1-0.6^2)=1.5625$. The whitened covariance is therefore $I+(Av)(Av)^T$; its signal eigenvalue is $1+1.5625=2.5625$, with eigenvector proportional to $(1,-0.75)^T$, not $(1,0)^T$.

This change is not an unavoidable property of every whitening convention: different whitening matrices can differ by an orthogonal rotation. But none of those choices makes sparsity invariant in general. Diagonal rescaling and coordinate permutation preserve the locations of zero entries; a general linear transformation does not. A whitening method should therefore be judged against the coordinate system in which the sparse scientific interpretation is defined.

Worked Example: Ordinary PCA Can Rotate Away from the Sparse Loading

Use the same $\Sigma_0$, $v$, and $\lambda$ as above. The observed covariance is $\Sigma=\begin{pmatrix}2&0.6\\0.6&1\end{pmatrix}$. Its eigenvalues are \[ \frac{3+\sqrt{1+4(0.6)^2}}{2} =\frac{3+\sqrt{2.44}}{2}\approx2.281, \qquad \frac{3-\sqrt{2.44}}{2}\approx0.719. \]

For the leading eigenvalue $\rho\approx2.281$, the first row of $(\Sigma-\rho I)u=0$ gives $(2-\rho)u_1+0.6u_2=0$. Hence $u_2/u_1=(\rho-2)/0.6\approx0.468$. The leading eigenvector has two nonzero coordinates, despite the true loading being $v=(1,0)^T$. This is not sampling error: it occurs in the population covariance because the off-diagonal background covariance couples the coordinates.

The calculation demonstrates a limitation of unadjusted PCA, not a claim that ordinary PCA always fails. If the background covariance is proportional to the identity, the population signal direction remains $v$.

Modeling the Noise in the Original Coordinates

One alternative is to keep the original coordinates and explicitly account for $\Sigma_0$. If the background is known, the population covariance contrast $\Sigma-\Sigma_0$ isolates the spike. In data, replacing $\Sigma$ with a sample covariance introduces estimation error; the noise covariance must still be known accurately or estimated well enough for the contrast to be useful.

Theorem: If $\Sigma_0$ is known and $\Sigma=\Sigma_0+\lambda vv^T$ with $\lambda>0$ and $\|v\|_2=1$, then the covariance contrast $\Sigma-\Sigma_0$ has leading eigenvector $v$, up to sign, and its only nonzero eigenvalue is $\lambda$.

Proof. Subtraction gives $\Sigma-\Sigma_0=\lambda vv^T$. Multiplying by $v$ yields $(\Sigma-\Sigma_0)v=\lambda v(v^Tv)=\lambda v$. If $u^Tv=0$, then $(\Sigma-\Sigma_0)u=\lambda v(v^Tu)=0$. The matrix therefore acts as multiplication by $\lambda$ on the span of $v$ and as zero on its orthogonal complement. Since $\lambda>0$, $v$ is its unique leading eigendirection, with sign as the only ambiguity. $\square$

Worked Example: Recovering the Support from a Population Covariance Contrast

Let $p=3$, $\Sigma_0=I$, $\lambda=2$, and $v=(1,0,1)^T/\sqrt{2}$. The loading has unit norm and support $\{1,3\}$. Its spike contribution is \[ 2vv^T= \begin{pmatrix} 1&0&1\\ 0&0&0\\ 1&0&1 \end{pmatrix}. \] Thus $\Sigma=I+2vv^T$, and subtracting the known background gives $\Sigma-\Sigma_0=\begin{pmatrix}1&0&1\\0&0&0\\1&0&1\end{pmatrix}$. Multiplication by $v$ returns $2v$, while multiplication by $e_2=(0,1,0)^T$ returns zero. The leading eigenvector is $v$ up to sign, and its nonzero coordinates identify the original support.

This is a population calculation. In finite samples, a raw contrast $S-\Sigma_0$ will generally have noise in every entry, so exact zero entries and exact support recovery do not follow from the identity alone.

Under Gaussian sampling, direct likelihood modeling gives a principled way to incorporate a known background while imposing sparsity on $v$ in its original coordinates. Let $S=n^{-1}\sum_{i=1}^n X_iX_i^T$ for mean-zero observations, and suppose $X_i$ are independent $N(0,\Sigma_0+\lambda vv^T)$, with known $\Sigma_0$ and fixed $\lambda>0$. The Gaussian log-likelihood, apart from a constant, is \[ \ell(v)=-\frac{n}{2}\left\{\log\det(\Sigma_0+\lambda vv^T)+ \operatorname{tr}\left[S(\Sigma_0+\lambda vv^T)^{-1}\right]\right\}. \]

Derived likelihood objective: For unit-norm $v$, maximizing the Gaussian likelihood over loadings with at most $s$ nonzero coordinates is equivalent to maximizing \[ -\log(1+\lambda q(v))+ \frac{\lambda\,v^T\Sigma_0^{-1}S\Sigma_0^{-1}v}{1+\lambda q(v)}, \qquad q(v)=v^T\Sigma_0^{-1}v, \] over $\|v\|_2=1$ and $\|\!v\!\|_0\le s$. Here $\|\!v\!\|_0$ denotes the number of nonzero coordinates.

To derive this objective, the matrix determinant lemma gives $\det(\Sigma_0+\lambda vv^T)=\det(\Sigma_0)(1+\lambda q(v))$. The Sherman–Morrison identity gives \[ (\Sigma_0+\lambda vv^T)^{-1} =\Sigma_0^{-1} -\frac{\lambda\Sigma_0^{-1}vv^T\Sigma_0^{-1}}{1+\lambda q(v)}. \] Taking the trace after multiplication by $S$, the first term is $\operatorname{tr}(S\Sigma_0^{-1})$, which does not depend on $v$. The second contributes $-\lambda v^T\Sigma_0^{-1}S\Sigma_0^{-1}v/(1+\lambda q(v))$. Likewise, $\log\det(\Sigma_0)$ and the trace term independent of $v$ can be dropped. Multiplication by the positive constant $2/n$ leaves the stated objective. The support constraint is on $v$, not on a whitened version of it.

Choosing an Approach and Avoiding a Support Mistake

The methods differ in what they make sparse and how they use the noise model. Whitening converts the background to identity and can simplify spectral analysis, but a sparsity constraint after transformation describes sparsity in transformed coordinates. Covariance subtraction targets the original spike directly when the background is known, while likelihood modeling can incorporate the same background and the sampling distribution in a finite-sample objective. If $\Sigma_0$ is estimated rather than known, its estimation error propagates into whitening, covariance subtraction, and likelihood fitting; treating an estimated background as exact can overstate the evidence for a recovered support.

Approach	What is made simple?	What sparsity means	Main caution
Ordinary PCA	Uses the observed covariance directly	Not enforced by itself	Correlated noise can rotate the leading direction
Whitening, then sparse PCA	Background covariance becomes identity	Sparsity of $Av$	$Av$ may be dense even when $v$ is sparse
Covariance contrast	Subtracts a known background at population level	Sparsity of the loading in original coordinates	Sample and background-estimation errors remain
Direct likelihood	Models the full covariance and sampling distribution	Sparsity constraint on original $v$	Optimization can be nonconvex and depends on model assumptions

A common mistake is to whiten, find a sparse-looking vector in the transformed coordinates, and report its nonzero entries as the support of the original loading. The reasoning fails because the transformed loading is $Av$, whose support need not equal the support of $v$. For example, the two-dimensional calculation above maps $(1,0)^T$ to $(1,-0.75)^T$. The correction is to state which coordinate system carries the sparsity assumption, use the corresponding constraint in the estimation method, and map estimates back with care. Inverting the whitening transformation recovers coordinates algebraically, but does not by itself ensure that an estimated support is accurate.

Key takeaway: Correlated noise is not merely a change in eigenvalue scale: it can change eigenvectors and mix coordinates. Whitening is useful for removing background covariance, but original-coordinate sparse recovery requires either an appropriate transformation-aware constraint or direct modeling of the noise in the original coordinates.

Check Your Understanding

Use the model and distinctions developed above to answer these questions.

For $\Sigma=\Sigma_0+\lambda vv^T$, what is the signal direction after applying a whitening matrix $A$, and why might it have a different support from $v$?
In the two-dimensional worked example, calculate the ratio of the coordinates of the leading population eigenvector of $\Sigma$. What causes the second coordinate to be nonzero?
Why does subtracting a known $\Sigma_0$ recover the rank-one spike at the population level, and what prevents the same argument from guaranteeing exact sample support recovery?
In the Gaussian likelihood objective, where should the sparsity constraint be imposed if the scientific target is the original loading?
Give one reason whitening may still be useful, and one reason it is not automatically a sparse-recovery solution.

Sparse PCA Under Correlated Noise

What You'll Learn