High-Dimensional Asymptotic Regimes for PCA

Why the Growth Regime Matters

PCA is computed from a sample covariance matrix, but its interpretation depends on how that matrix behaves as the sample size increases. If the number of variables stays fixed, each covariance entry can settle toward its population value. If the number of variables grows along with the sample, the analysis must also account for the increasing number of entries and the changing rank of the sample covariance. The phrase high-dimensional PCA therefore does not describe one asymptotic setting: it describes several, with different consequences.

Let \(X_1,\ldots,X_n\) be independent observations in \(\mathbb{R}^p\), each with mean \(\mu\) and covariance matrix \(\Sigma\). Use the centered sample covariance \(S=(n-1)^{-1}\sum_{i=1}^n (X_i-\bar X)(X_i-\bar X)^\top\), where \(\bar X=n^{-1}\sum_{i=1}^n X_i\). The key quantity for classifying the main regimes is the aspect ratio \(p/n\). This tutorial focuses on what that ratio means for classical covariance-based PCA approximations, not on the detailed eigenvalue limits of particular high-dimensional models.

Three Main Asymptotic Regimes

These are descriptions of sequences, not labels that can be assigned from one data set alone. For example, \(p=100\) and \(n=1{,}000\) has ratio \(0.1\), but by itself it does not tell us whether future problems keep \(p\) fixed, make it grow proportionally, or let it grow much faster than \(n\). The intended sequence matters.

What Fixed-Dimension PCA Approximations Rely On

In fixed dimension, entrywise consistency of \(S\) gives convergence of the whole covariance matrix in operator norm. This is a particularly useful bridge from covariance estimation to PCA: small operator-norm error controls errors in eigenvalues and, when a population eigenvalue is separated from the others, in its eigenvector direction.

Here is the reasoning. Each entry of \(S\) is a sample covariance, and the law of large numbers gives convergence in probability to the corresponding entry of \(\Sigma\). Finite \(p\) then turns entrywise convergence into operator-norm convergence: for a \(p\times p\) matrix \(A\), \(\|A\|_{\mathrm{op}}\leq\|A\|_{\mathrm{F}}\leq p\max_{j,k}|A_{jk}|\). The required sample moments exist because \(E\|X_i\|^2<\infty\).

To see why a gap matters, write \(E_n=S-\Sigma\), and choose the sign of \(\hat v_1\) so that its component along \(v_1\) is nonnegative. The eigenvalue perturbation bound gives \(|\hat\lambda_1-\lambda_1|\leq\|E_n\|_{\mathrm{op}}\). Projecting \(S\hat v_1=\hat\lambda_1\hat v_1\) onto the orthogonal complement of \(v_1\) yields \((\hat\lambda_1 I-\Sigma_{\perp})P_{\perp}\hat v_1=P_{\perp}E_n\hat v_1\), where \(\Sigma_{\perp}\) is the restriction of \(\Sigma\) to that complement. On the event \(\|E_n\|_{\mathrm{op}}<\delta\), every eigenvalue of \(\hat\lambda_1 I-\Sigma_{\perp}\) is at least \(\delta-\|E_n\|_{\mathrm{op}}\). Therefore \(\|P_{\perp}\hat v_1\|\leq\|E_n\|_{\mathrm{op}}/(\delta-\|E_n\|_{\mathrm{op}})\). This tends to zero in probability. For unit vectors with the chosen sign, that implies \(\|\hat v_1-v_1\|\to0\) in probability.

The same fixed-dimension framework supports classical eigenvalue and eigenvector approximations that keep \(p\) fixed while \(n\) grows. Such results are not automatically uniform in \(p\). A small error in each covariance entry does not, by itself, guarantee a small error for the whole matrix when the number of entries is increasing.

Proportional Growth: Errors Are Not Automatically Negligible

When \(p/n\to\gamma>0\), the dimension is no longer a fixed constant in the covariance-convergence argument. The number of covariance entries increases, and sampling variation across directions can remain important even as \(n\to\infty\). As a result, the fixed-dimensional justification for treating the sample covariance as a small operator-norm perturbation of \(\Sigma\) cannot simply be carried over. In common isotropic noise models, the sample spectrum has non-negligible sampling variation at this scale; a fixed-dimensional approximation should not be used to dismiss it.

There is an especially transparent rank consequence when \(\gamma>1\). Since \(p>n-1\) eventually, a centered sample covariance has at least \(p-(n-1)\) zero eigenvalues. Thus the sample covariance cannot be a small operator-norm perturbation of a full-rank population covariance such as \(I_p\): on a sample-null direction \(u\), \(Su=0\) but \(I_pu=u\), so \(\|S-I_p\|_{\mathrm{op}}\geq1\). This example establishes a concrete failure of covariance operator-norm consistency in part of the proportional regime. When \(0<\gamma\leq1\), the same rank argument does not apply, but that does not restore fixed-dimensional approximations; high-dimensional spectral behavior must still be justified under the model and scaling at hand.

Ultra-High Dimension: Rank and Scale Become Central

When \(p/n\to\infty\), centering limits the sample covariance rank to at most \(n-1\), while the ambient dimension grows much faster. Most directions in \(\mathbb{R}^p\) therefore lie outside the span of the centered observations. The covariance matrix is singular, and its nonzero eigenvalues summarize variation in the observed sample span rather than providing a full-rank estimate across all \(p\) coordinates. In this setting, statements based on inverting the sample covariance require particular care.

The trace gives a useful scale check. Under a population covariance \(I_p\), \(E[\operatorname{tr}(S)]=p\), so the expected average of the \(n-1\) nonzero eigenvalues is \(p/(n-1)\) when the centered sample has rank \(n-1\). This average grows without bound if \(p/n\to\infty\). It is not an approximation to the population eigenvalue 1 in each of those directions; it reflects the concentration of the total sample variance into a much smaller sample span.

Two Useful Derivations

The rank and trace claims can be established directly, without an asymptotic approximation. Let \(Z\) be the \(n\times p\) matrix whose \(i\)th row is \(X_i-\bar X\). Each column of \(Z\) sums to zero, so its columns lie in the \((n-1)\)-dimensional subspace orthogonal to the all-ones vector. Therefore \(\operatorname{rank}(Z)\leq n-1\). Since \(S=Z^\top Z/(n-1)\), \(\operatorname{rank}(S)=\operatorname{rank}(Z)\leq n-1\). In particular, if \(p>n-1\), \(S\) is singular.

For the trace calculation, use \(\operatorname{tr}(S)=(n-1)^{-1}\sum_{i=1}^n\|X_i-\bar X\|^2\). The identity \(\sum_i\|X_i-\bar X\|^2=\sum_i\|X_i-\mu\|^2-n\|\bar X-\mu\|^2\) follows by expanding each squared norm and using \(\sum_i(X_i-\bar X)=0\). Taking expectations gives \(n\operatorname{tr}(\Sigma)-n\operatorname{tr}(\Sigma)/n=(n-1)\operatorname{tr}(\Sigma)\), since \(E\|X_i-\mu\|^2=\operatorname{tr}(\Sigma)\) and \(E\|\bar X-\mu\|^2=\operatorname{tr}(\Sigma)/n\). Dividing by \(n-1\) yields \(E[\operatorname{tr}(S)]=\operatorname{tr}(\Sigma)\). For \(\Sigma=I_p\), this is \(p\), as used above. \(\square\)

Common Mistake and Practical Interpretation

Common mistake: “The sample size is large, so the sample covariance and its principal components must be close to their population counterparts.” This reasoning omits the dimension. It is justified by the fixed-dimension convergence argument only when its assumptions and dimension scaling apply. If \(p\) grows with \(n\), the number of directions being estimated also grows; if \(p>n-1\), the centered sample covariance is singular regardless of how large \(n\) is.

A sound PCA analysis therefore states the asymptotic regime before invoking a classical approximation. In fixed dimension, covariance consistency and a nonzero eigengap support stable principal directions. In proportional growth, sampling variation in the spectrum cannot generally be treated as negligible simply because the sample size increases. In ultra-high dimension, rank deficiency and the concentration of sample variance in the observed span are immediate structural facts. These distinctions guide both statistical interpretation and computation: a low-rank sample representation can be useful, but it should not be mistaken for a full-rank estimate of population covariance structure.