Spiked Covariance Models

Why Spiked Covariance Models Separate Signal From Noise

In high-dimensional principal component analysis (PCA), a population covariance matrix can have a small number of directions with more variation than the rest. A spiked covariance model represents this structure by combining isotropic background variation with a low-rank signal component. The model is useful because it makes the population eigenvalues and eigendirections explicit, while keeping the number of signal directions small relative to the dimension.

Here, “spike” refers to an eigenvalue of the population covariance that is larger than the common noise eigenvalue. It does not refer to an observed sample eigenvalue. A sample covariance matrix is random, and its eigenvalues need not equal the population eigenvalues. The present focus is the population model itself, not the asymptotic behavior of those sample eigenvalues.

The Rank-One Model

Let \(X\) be a random vector in \(\mathbb{R}^p\) with finite second moments, mean \(\mu=E[X]\), and positive definite covariance matrix \(\Sigma=\operatorname{Cov}(X)\). In the rank-one model, a unit vector \(u\in\mathbb{R}^p\) gives the signal direction, and the covariance is

Here \(I_p\) is the \(p\)-dimensional identity matrix, \(\sigma^2\) is the isotropic noise variance, and \(\theta\) is the excess variance in direction \(u\). The population eigenvalue in that direction is \(\sigma^2+\theta\). All directions orthogonal to \(u\) have eigenvalue \(\sigma^2\). When \(\sigma^2=1\), it is also common to call \(\alpha=1+\theta\) the spike eigenvalue; \(\theta\) and \(\alpha\) are different quantities.

The form of the model can also be seen from a simple random-effects construction. Suppose \(Z\) is a scalar with mean zero and variance one, and \(\varepsilon\) is a random vector with mean zero and covariance \(\sigma^2 I_p\), independent of \(Z\). Set \(X=\mu+\sqrt{\theta}\,uZ+\varepsilon\). Independence makes the cross-covariance terms zero, so

This construction explains the interpretation: one shared scalar source of variation moves the vector along \(u\), while the noise contributes equally in every direction. The covariance model itself does not require this particular distributional construction; finite second moments and the stated covariance are enough.

Proof. Since \(u^\mathsf{T}u=1\),

For any \(v\) orthogonal to \(u\), \(u^\mathsf{T}v=0\), and therefore

The vector \(u\) together with an orthonormal basis for its \((p-1)\)-dimensional orthogonal complement spans \(\mathbb{R}^p\). These eigenvectors account for all \(p\) eigenvalues, completing the result. \(\square\)

Finite-Rank Spikes

A model with \(r\) signal directions replaces the single direction \(u\) by a matrix \(U=(u_1,\ldots,u_r)\) whose columns are orthonormal. With positive excess strengths \(\theta_1,\ldots,\theta_r\), write

The orthonormality condition says \(u_j^\mathsf{T}u_k=0\) for \(j\ne k\) and \(u_j^\mathsf{T}u_j=1\). It makes the columns of \(U\) the population signal directions. The covariance has eigenvalue \(\sigma^2+\theta_j\) in direction \(u_j\), for each \(j\), and eigenvalue \(\sigma^2\) on the orthogonal complement of the column space of \(U\). If \(r<p\), the latter eigenvalue has multiplicity \(p-r\). In high-dimensional settings, the number \(r\) is often held fixed as \(p\) and the sample size grow; the spikes are then a low-rank departure from the noise covariance.

Proof. Let \(e_j\) be the \(j\)th coordinate vector in \(\mathbb{R}^r\). Since \(U^\mathsf{T}u_j=e_j\) and \(De_j=\theta_j e_j\),

If \(v\) is orthogonal to every column of \(U\), then \(U^\mathsf{T}v=0\), so

The \(r\) orthonormal columns of \(U\) and an orthonormal basis for their orthogonal complement together form an orthonormal basis of \(\mathbb{R}^p\). The displayed eigenvalues therefore describe the entire covariance spectrum. \(\square\)

When Factor Loadings Are Not Orthogonal

A low-rank covariance contribution need not be presented using orthonormal columns. For example, a factor model may write \(\Sigma=\sigma^2 I_p+BB^\mathsf{T}\) for a loading matrix \(B\). In that expression, the columns of \(B\) need not themselves be covariance eigenvectors. The population spikes are determined by the eigenvalues and eigenvectors of the full covariance matrix, not by simply reading off the loading columns.

What the Model Says—and What It Does Not

In the finite-rank model, the population spectrum has \(r\) signal eigenvalues \(\sigma^2+\theta_j\) and a noise eigenvalue \(\sigma^2\) repeated on the remaining directions. The spike strengths are population quantities. Estimating them from data requires studying a sample covariance matrix, whose eigenvalues depend on the realized observations as well as the population model. In a high-dimensional asymptotic regime, the sample spectrum can behave differently from the population spectrum; the population calculation alone does not establish where a sample eigenvalue will lie.

The model is useful for expressing a clear structural assumption: most directions share a common background variance, while a small number of directions have additional variance. It provides a baseline for studying high-dimensional PCA without assuming that the population covariance is itself diagonal in the observed coordinate system. The signal vectors can be any orthonormal directions, including directions formed from many coordinates.

Common Mistakes

• Calling the excess strength the spike eigenvalue. In \(\Sigma=\sigma^2 I_p+\theta uu^\mathsf{T}\), the eigenvalue is \(\sigma^2+\theta\), not \(\theta\). When \(\sigma^2=1\), the eigenvalue is \(1+\theta\).
• Assigning a spike to every coordinate. The signal directions are the eigenvectors \(u_j\), which can combine several coordinates. The entries of \(u_j\) describe the direction; they are not separate eigenvalues.
• Treating sample and population eigenvalues as identical. The model specifies \(\Sigma\). A sample covariance matrix computed from finitely many observations is random, so its eigenvalues are not generally exactly the eigenvalues of \(\Sigma\).
• Assuming arbitrary loading columns are eigenvectors. That conclusion follows in the stated finite-rank form because \(U^\mathsf{T}U=I_r\). Without orthonormality, calculate the eigensystem of the covariance contribution instead.