Why Weak Factors Need a Subspace Error Analysis
In a factor model, the leading principal components are intended to recover a low-dimensional signal space, not necessarily a particular choice of axes within that space. This distinction matters when factors are repeated or close in strength: individual eigenvectors can rotate within the signal space even when the space itself is estimated accurately. The relevant error is therefore the angle between the true factor subspace and the estimated principal subspace.
When factor eigenvalues are large relative to the noise, a moderate covariance estimation error may be small compared with the signal gap. This tutorial focuses on a different regime: the number of variables \(p\) grows, the factor eigenvalues \(d_n\) also grow but satisfy \(d_n=o(p)\), and the sample size \(n\) must be large enough relative to both. The key quantity is not \(d_n\) alone, but the size of the covariance error relative to the eigengap.
The projection matrices \(P_U=UU'\) and \(P_V=VV'\) describe the subspaces without depending on which orthonormal bases were chosen. For equal-dimensional subspaces, \(\|P_U-P_V\|_{\mathrm{op}}=\|\sin\Theta\|_{\mathrm{op}}\). Thus a small principal-angle error means that vectors in the estimated subspace lie close to the true subspace.
A Perturbation Bound for the Leading Subspace
Let \(\Sigma\) be a symmetric population covariance matrix with ordered eigenvalues \(\lambda_1\geq\cdots\geq\lambda_p\). Let \(U\) span its leading \(r\) eigenvectors, and define the eigengap \(g=\lambda_r-\lambda_{r+1}>0\). Suppose the sample covariance is \(S=\Sigma+E\), where \(E\) is symmetric, and let \(V\) span the leading \(r\) eigenvectors of \(S\).
Proof. Write \(U_\perp\) for an orthonormal basis of the orthogonal complement of \(U\), and let \(\widehat\Lambda\) be the diagonal matrix of the leading \(r\) eigenvalues of \(S\). Since \(SV=V\widehat\Lambda\), projection onto the complement of \(U\) gives \[ \Lambda_\perp W-W\widehat\Lambda=-U_\perp'EV,\qquad W=U_\perp'V, \] where \(\Lambda_\perp\) is the restriction of \(\Sigma\) to the complement of \(U\). The eigenvalues of \(\Lambda_\perp\) are at most \(\lambda_{r+1}\). Weyl’s eigenvalue inequality gives \(\lambda_r(S)\geq\lambda_r-\delta\), so every eigenvalue of \(\widehat\Lambda\) is at least \(\lambda_r-\delta\). The two spectra are separated by at least \(g-\delta>0\).
For these separated symmetric matrices, the equation \(AW-WB=-C\) has the integral representation \(W=\int_0^\infty e^{tA}Ce^{-tB}\,dt\), when the spectrum of \(A\) lies below that of \(B\). Here the integrand’s operator norm is at most \(e^{-t(g-\delta)}\|C\|_{\mathrm{op}}\). Integrating this bound yields \(\|W\|_{\mathrm{op}}\leq\|C\|_{\mathrm{op}}/(g-\delta)\). Taking \(C=U_\perp'EV\) gives \(\|C\|_{\mathrm{op}}\leq\delta\), and \(\|W\|_{\mathrm{op}}=\|\sin\Theta(U,V)\|_{\mathrm{op}}\). This proves the bound. If \(\delta<g/2\), then \(g-\delta>g/2\), giving the stated simpler inequality. \(\square\)
This result isolates the central issue: a covariance estimate can be accurate in absolute terms yet insufficient for recovery if its error is not small compared with the eigengap. The bound is a sufficient perturbation guarantee, not a claim that every violation of its condition makes recovery impossible.
Gaussian Factor Model and a Sample-Size Condition
For a transparent calculation, consider independent observations \[ x_i=Uf_i+e_i,\qquad i=1,\ldots,n, \] where \(U'U=I_r\), \(f_i\sim N(0,D_n)\), \(e_i\sim N(0,\sigma^2 I_p)\), and the factors and noise are mutually independent. Assume \(r\) is fixed, \(\sigma^2>0\) is fixed, and all eigenvalues of \(D_n\) lie between \(c d_n\) and \(C d_n\) for fixed positive constants \(c,C\). The population covariance is \[ \Sigma=UD_nU'+\sigma^2 I_p. \] Its leading \(r\)-dimensional eigenspace is the span of \(U\), and the gap between the \(r\)th and \((r+1)\)st population eigenvalues is \(g_n=\lambda_{\min}(D_n)\), of order \(d_n\).
Use the uncentered sample second-moment matrix \(S=n^{-1}\sum_{i=1}^n x_ix_i'\). The observations have mean zero, so this choice estimates \(\Sigma\) directly. Decomposing the error gives \[ S-\Sigma =U\left(\frac{FF'}{n}-D_n\right)U' +\left(\frac{EE'}{n}-\sigma^2I_p\right) +\frac{UFE'+EF'U'}{n}, \] where \(F\) and \(E\) are the matrices with columns \(f_i\) and \(e_i\).
Derivation. The factor matrix has fixed row dimension \(r\). Entrywise Gaussian variance calculations therefore give \(\|FF'/n-D_n\|_{\mathrm{op}}=O_p(d_n/\sqrt n)\). For a \(p\times n\) matrix \(E\) of independent \(N(0,\sigma^2)\) entries, Gaussian singular-value concentration gives \(\|EE'/n-\sigma^2I_p\|_{\mathrm{op}}=O_p(\sqrt{p/n}+p/n)\). This bound follows by applying the chi-squared tail bound to \(\|E'x\|^2/n\) on a fixed-radius net of the unit sphere in \(\mathbb R^p\); the net has at most exponentially many points in \(p\), yielding deviations of order \(\sqrt{p/n}+p/n\).
For the cross term, condition on \(F\). Each of the \(p\) rows of \(EF'/n\) is Gaussian with covariance \(\sigma^2 FF'/n^2\). Since \(r\) is fixed and \(\|FF'/n\|_{\mathrm{op}}=O_p(d_n)\), its operator norm is \(O_p(\sqrt{p d_n/n})\). Adding the three bounds proves the covariance-error statement.
To see the rate, divide each error term by the eigengap, which is of order \(d_n\). The resulting terms are \(n^{-1/2}\), \(\sqrt{p/(n d_n)}\), \(\sqrt{p/n}/d_n\), and \(p/(n d_n)\). Under \(d_n\to\infty\) and \(n d_n/p\to\infty\), the last two are no larger in order than terms already tending to zero: \(\sqrt{p/n}/d_n=\sqrt{p/(n d_n)}/\sqrt{d_n}\), and \(p/(n d_n)\to0\). The full error divided by the gap thus tends to zero in probability. The Davis–Kahan bound applies with probability tending to one and gives the displayed rate.
The condition \(d_n=o(p)\) describes weak growth relative to dimension; by itself, it does not ensure recovery. In this isotropic-noise Gaussian model, \(n d_n/p\to\infty\) is a transparent sufficient condition: increasing signal strength can compensate for a sample size that is small relative to \(p\), while a weaker signal requires more observations. This is a sufficient result under the stated model, not a universal threshold for all factor or noise structures.
Worked Examples
Worked Example: A Weak but Recoverable Growing Signal
Suppose \(p\to\infty\), there is one factor, \(d_n=p^{0.4}\), and \(n=p^{0.8}\). Then \(d_n/p=p^{-0.6}\to0\), so the factor is weak relative to the number of variables, while \(d_n\to\infty\). Check the sample-size condition: \[ \frac{n d_n}{p}=\frac{p^{0.8}p^{0.4}}{p}=p^{0.2}\longrightarrow\infty. \]
The rate bound is \[ n^{-1/2}+\sqrt{\frac{p}{n d_n}} =p^{-0.4}+\sqrt{p^{-0.2}} =p^{-0.4}+p^{-0.1}. \] Both terms tend to zero, and the second is larger for large \(p\). Thus the theorem guarantees consistency, with error \(O_p(p^{-0.1})\). The signal is sublinear in \(p\), but the sample size is sufficient relative to the signal and dimension.
Worked Example: Checking a Finite-Scale Error Bound
Take \(p=10{,}000\), \(n=1{,}000{,}000\), one factor with \(d_n=100\), and \(\sigma^2=1\). The population gap is \(g_n=100\), and \(d_n/p=0.01\). The orders in the covariance-error bound are \[ \frac{d_n}{\sqrt n}=\frac{100}{1000}=0.1,\qquad \sqrt{\frac{p d_n}{n}}=\sqrt{1}=1, \] \[ \sqrt{\frac{p}{n}}=\sqrt{0.01}=0.1,\qquad \frac{p}{n}=0.01. \]
Their sum is \(1.21\) times an unspecified constant implicit in \(O_p\); it is not an exact realized error. Dividing the displayed order terms by the gap gives \(0.001\), \(0.01\), \(0.001\), and \(0.0001\), respectively. The leading rate terms are therefore \(n^{-1/2}=0.001\) and \(\sqrt{p/(n d_n)}=0.01\). The asymptotic calculation predicts a small angle at these scales, but it does not provide a numerical confidence bound without explicit constants or an observed covariance matrix.
Worked Example: Why Dimension Can Outrun Sample Information
Let \(d_n=p^{0.25}\) and \(n=p^{0.75}\). The signal remains weak relative to dimension because \(d_n/p=p^{-0.75}\to0\), and it grows because \(d_n\to\infty\). But \[ \frac{n d_n}{p}=\frac{p^{0.75}p^{0.25}}{p}=1, \qquad \sqrt{\frac{p}{n d_n}}=1. \] The sufficient consistency condition fails, and the bound does not tend to zero. This does not prove that recovery is impossible; it says this theorem supplies no consistency guarantee in this regime.
If instead \(n=p\), then \(n d_n/p=p^{0.25}\to\infty\), and the rate becomes \[ p^{-1/2}+\sqrt{\frac{p}{p\,p^{0.25}}} =p^{-1/2}+p^{-0.125}\longrightarrow0. \] The comparison shows how a larger sample can restore the theorem’s guarantee without changing the weak-signal scaling.
Interpretation and a Common Pitfall
The principal-angle criterion is useful because it evaluates the scientific object of interest: whether the estimated factor span captures the population span. If the factors have nearly equal strengths, the individual estimated axes may change substantially under small perturbations, while their joint subspace remains close. Reporting only separate eigenvector comparisons can therefore obscure successful recovery of the factor space.
- Pitfall: comparing covariance error with the dimension rather than the gap. A small error relative to \(p\) is not enough. The perturbation result compares \(\|S-\Sigma\|_{\mathrm{op}}\) with \(g_n\), which here is of order \(d_n\).
- Pitfall: treating \(d_n=o(p)\) as a recovery condition. This describes the signal’s scale relative to dimension; it says nothing by itself about whether enough observations are available.
- Pitfall: calling a sufficient condition a necessary threshold. The condition \(n d_n/p\to\infty\) is sufficient under the Gaussian isotropic-noise assumptions used here. Different noise structure or additional information can change the analysis.
Check Your Understanding
Use the definitions and sufficient bound above to answer the following.
- Why does principal-angle error compare subspaces rather than require a fixed orientation for each estimated eigenvector?
- In the Gaussian factor model, what is the population eigengap between the factor space and its orthogonal complement?
- If \(d_n\to\infty\) and \(d_n=o(p)\), why is the condition \(n d_n/p\to\infty\) still needed for the stated consistency guarantee?
- For \(d_n=p^{0.3}\) and \(n=p^{0.9}\), does \(n d_n/p\) diverge? What does the sufficient result imply?
- Does failure of the sufficient condition prove that PCA cannot recover the subspace? Explain.