Multiple and Repeated Population Spikes

Why Multiplicity Changes What PCA Can Identify

In high-dimensional principal component analysis, a population covariance matrix may contain several directions of elevated variance. When the corresponding population eigenvalues are distinct and sufficiently large, their sample counterparts can appear as separated eigenvalue outliers, and their eigenvectors can carry information about the associated population directions. But repeated or nearly repeated spikes change the question: does the data distinguish each direction, or only the larger subspace they span?

We study this distinction in the Gaussian spiked covariance model. Let \(p,n\to\infty\) with \(p/n\to\gamma\), where \(0<\gamma<\infty\), and let the observations be independent \(N_p(0,\Sigma)\) vectors. Suppose the population covariance has a fixed number \(r\) of spikes above the unit noise level:

where \(v_1,\ldots,v_r\) are orthonormal population directions. The sample covariance is \(S=n^{-1}\sum_{i=1}^n X_iX_i^\top\). A spike is supercritical when \(\alpha_j>1+\sqrt{\gamma}\). Such a spike produces a sample eigenvalue whose limit is strictly above the Marchenko–Pastur upper edge \((1+\sqrt{\gamma})^2\). The conclusions below concern upper spikes; lower population spikes require the corresponding lower-edge analysis.

The Outlier Map and Its Multiplicity

For a supercritical spike, the outlier equation established in the resolvent analysis gives the limiting sample eigenvalue as a function of its population value:

The threshold matters: this formula describes a separated outlier only above the threshold. A subcritical spike does not create an eigenvalue separated from the upper bulk edge. At the threshold, the displayed map meets that edge.

To see why the outlier locations preserve the ordering of distinct supercritical spikes, differentiate the map:

For \(\alpha>1+\sqrt{\gamma}\), we have \(\alpha-1>\sqrt{\gamma}\), so \(\lambda'(\alpha)>0\). Thus \(\lambda\) is strictly increasing on the supercritical range: if \(\alpha_a<\alpha_b\) are both supercritical, then \(\lambda(\alpha_a)<\lambda(\alpha_b)\). The outlier equation applies once for each spike direction, so a value repeated \(k\) times gives \(k\) sample eigenvalues with the same deterministic limit. Their finite-sample values need not be exactly equal. The theorem asserts convergence to a common location, not an exact finite-sample tie. \(\square\)

Separated Spikes and Directional Alignment

When a supercritical spike is simple and separated from the other population spikes, its sample eigenvector can identify its population direction to a nonzero degree. In the Gaussian identity-noise model, if \(\hat v\) is the sample eigenvector associated with a simple spike \(\alpha\), its limiting squared alignment with the corresponding unit population eigenvector \(v\) is

This expression is positive above the threshold and tends to zero as the spike approaches the threshold from above. It is generally less than one at fixed \(\gamma>0\): a separated outlier is evidence about its population direction, not a guarantee that the sample eigenvector equals that direction. For a simple spike, the sign of an eigenvector is arbitrary, which is why alignment is expressed through the squared inner product.

Repeated Spikes: The Eigenspace Is the Identifiable Object

Suppose a supercritical value \(\alpha\) has multiplicity \(k>1\), and let \(V\) be a \(p\times k\) matrix whose orthonormal columns span its population eigenspace. Its contribution to the covariance is \((\alpha-1)VV^\top\). Replacing \(V\) by \(VR\), for any \(k\times k\) orthogonal matrix \(R\), leaves the covariance unchanged:

Therefore the covariance model cannot distinguish one chosen orthonormal basis of the repeated eigenspace from another. In particular, the labels assigned to individual directions within that space are not identifiable from the distribution. This is not a failure of a particular eigenvector algorithm: all those choices specify exactly the same population covariance.

There is a corresponding symmetry in the data. A rotation acting within the repeated population eigenspace leaves \(\Sigma\) unchanged, and hence leaves the Gaussian data distribution unchanged. A sample eigenvector basis inside the resulting group of outliers can rotate with the data; it should not be interpreted as consistently recovering a particular, pre-labelled population direction. The meaningful population target is the shared \(k\)-dimensional eigenspace. Its sample counterpart carries information about that space, but at a fixed positive aspect ratio it is not in general an exact copy of it.

Nearly Colliding Spikes

Two distinct, fixed supercritical values have distinct limiting outlier locations. But that statement is not a uniform guarantee of easy separation when their gap shrinks as the dimension grows. The map’s local slope describes how population gaps translate into limiting outlier gaps:

provided both values remain in the supercritical range. The approximation follows from differentiability of \(\lambda\). In particular, the slope approaches zero as \(\alpha\) approaches the threshold from above, so spikes close to that threshold have especially compressed limiting locations. If a spike gap itself tends to zero, the limiting map alone does not determine whether the two sample eigenvalues will be distinguishable at a given sample size; that requires comparing the gap with their finite-sample fluctuations.

The same issue affects directions. A simple algebraic illustration is a symmetric two-dimensional matrix with diagonal entries \(\lambda_1,\lambda_2\) and a small off-diagonal perturbation \(\varepsilon\). If its leading eigenvector makes angle \(\theta\) with the first coordinate axis, then

When the gap \(|\lambda_1-\lambda_2|\) is large relative to \(|\varepsilon|\), the angle is small. When the gap is comparable to or smaller than the perturbation, the angle need not be small. This calculation explains why a nearly colliding pair of sample eigenvalues can have unstable individual eigenvectors even if the two-dimensional space they span is more stable. It is an illustration of eigenvalue-gap sensitivity, not a claim about the exact distribution or scale of sample fluctuations.

Common Mistake: Treating Every Sample Eigenvector as a Population Direction

A common mistake is to count sample eigenvectors above the bulk edge and assign each one a unique population direction, even when the corresponding population spikes are equal or nearly equal. That reasoning confuses the number of outlier eigenvalues with identifiability of a particular basis. A repeated spike produces as many outlier eigenvalues as its multiplicity, but the covariance is invariant to rotations within the repeated eigenspace. With near-collisions, distinct limiting locations do not by themselves ensure stable finite-sample eigenvectors.

The correction is to distinguish three cases. For a simple, separated supercritical spike, the associated sample eigenvector has nonzero asymptotic alignment with its population direction. For a repeated supercritical spike, treat the group of eigenvectors as an estimate of a shared subspace, not as labelled directions. For a shrinking gap, assess whether individual directions are resolvable only after accounting for finite-sample variability; an eigenvector’s apparent ordering is not sufficient evidence.