Likelihood-Ratio Tests for the Number of Components

Why Test Whether Trailing Eigenvalues Are Equal?

In population principal component analysis, a proposed retained dimension \(k\) can be assessed by asking whether the remaining \(p-k\) covariance eigenvalues are all equal. Under this hypothesis, the directions outside the leading \(k\)-dimensional subspace have a common variance: they do not exhibit further covariance structure that distinguishes one trailing direction from another. The question is statistical, not merely descriptive: observed sample eigenvalues vary even when their population counterparts are equal.

The likelihood-ratio test compares an unrestricted Gaussian covariance model with one in which the trailing population eigenvalues are constrained to be equal. Its statistic pools the trailing sample eigenvalues and measures how much fit is lost by doing so. The resulting test concerns equal trailing variances; it does not assert that those variances are zero.

Write the ordered eigenvalues of the sample covariance matrix as \(s_1\ge \cdots\ge s_p>0\), where \(S=n^{-1}\sum_{j=1}^n(X_j-\bar X)(X_j-\bar X)^\mathsf{T}\). We use the maximum-likelihood divisor \(n\). The usual finite-sample calculation below assumes \(n>p\), so that \(S\) is positive definite with probability one under the Gaussian model. The common trailing estimate under the null will be the arithmetic mean of \(s_{k+1},\ldots,s_p\).

Deriving the Likelihood-Ratio Statistic

The likelihood ratio is the largest likelihood under the null divided by the largest likelihood under the unrestricted alternative. After maximizing over the unknown mean, the likelihood for a covariance matrix is proportional to \(|\Sigma|^{-n/2}\exp\{-n\,\operatorname{tr}(S\Sigma^{-1})/2\}\). Under the unrestricted alternative, the maximum-likelihood estimate is \(S\).

Under the null, the covariance eigenvectors are not fixed in advance. For any fixed set of covariance eigenvalues, aligning the eigenvectors of \(\Sigma\) with those of \(S\) in the same order minimizes \(\operatorname{tr}(S\Sigma^{-1})\). This follows from the trace rearrangement inequality. With that alignment, the first \(k\) covariance eigenvalues can be estimated as \(s_1,\ldots,s_k\). The common trailing value \(\tau\) is estimated by maximizing the likelihood over the remaining block.

To verify the constrained estimate, let \(u\) be the common trailing covariance eigenvalue. The terms of the negative twice log-likelihood that depend on \(u\), apart from constants, are \(n\{r\log u+(\sum_{i=k+1}^p s_i)/u\}\). Differentiating with respect to \(u\) gives \(n\{r/u-(\sum_{i=k+1}^p s_i)/u^2\}\), which vanishes at \(u=\bar s_{\mathrm{tail}}\). The second derivative there is \(nr/\bar s_{\mathrm{tail}}^{\,2}>0\), so this is the minimum of the negative log-likelihood. Also \(s_k\ge s_{k+1}\ge \bar s_{\mathrm{tail}}\), so the estimate respects the required ordering.

At this estimate, the trace term is \(\sum_{i=1}^k s_i/s_i+\sum_{i=k+1}^p s_i/\bar s_{\mathrm{tail}}=k+r=p\). It is also \(p\) at the unrestricted estimate \(S\). The exponential likelihood factors therefore cancel, leaving the determinant ratio displayed above. By the arithmetic-geometric mean inequality, \(\bar s_{\mathrm{tail}}^{\,r}\ge\prod_{i=k+1}^p s_i\); hence \(0<\Lambda\le1\) and \(T\ge0\), as a likelihood-ratio statistic must.

Asymptotic Reference Distribution

For this test, the asymptotic degrees of freedom are not simply the number of pairwise equalities among the trailing eigenvalues. The equality constraint leaves the trailing eigenspace free to rotate, and that freedom affects the dimension of the covariance restriction. Under the regularity conditions below, the null distribution of \(T\) is asymptotically chi-square.

Here is a derivation. Work in population eigenvector coordinates, so that the trailing population block is \(\tau I_r\). The Gaussian covariance central limit theorem implies that the trailing \(r\times r\) block of \(\sqrt n(S-\Sigma)/\tau\) converges to a symmetric Gaussian matrix \(A\) whose diagonal entries are independent \(N(0,2)\), whose entries above the diagonal are independent \(N(0,1)\), and whose distinct upper-triangular entries are independent. The fixed gap between the leading eigenvalues and \(\tau\) separates the trailing cluster. Consequently, the trailing sample eigenvalues have first-order deviations \(n^{-1/2}\tau\) times the eigenvalues of \(A\). In particular, after subtracting their average, their squared deviations satisfy

This first-order statement follows by expressing the sample covariance in leading and trailing blocks. The off-diagonal blocks are \(O_p(n^{-1/2})\); because the population blocks are separated by a fixed eigenvalue gap, their effect on the trailing eigenvalues is second order, \(O_p(n^{-1})\). The first-order trailing eigenvalue changes are therefore the eigenvalues of the trailing block fluctuation.

It remains to expand the logarithms in \(T\). Put \(\delta_i=s_i-\bar s_{\mathrm{tail}}\) for the trailing indices. Then \(\sum\delta_i=0\), and each \(\delta_i=O_p(n^{-1/2})\). A Taylor expansion around \(\bar s_{\mathrm{tail}}\), which converges in probability to \(\tau>0\), gives \(\sum\log(s_i)=r\log(\bar s_{\mathrm{tail}})-\sum\delta_i^2/(2\bar s_{\mathrm{tail}}^2)+o_p(n^{-1})\). The cubic remainders are \(O_p(n^{-3/2})\) because \(r\) is fixed. Thus \(T=n\sum\delta_i^2/(2\bar s_{\mathrm{tail}}^2)+o_p(1)\).

Finally, the traceless symmetric matrices form a space of dimension \(r(r+1)/2-1\). In an orthonormal coordinate system for the Frobenius inner product, the Gaussian matrix \(A\) has independent standard-normal coordinates after the scaling implicit in \(\|A\|_F^2/2\). Projecting out its trace removes one coordinate, so \(\|A-\operatorname{tr}(A)I_r/r\|_F^2/2\) has a chi-square distribution with \(r(r+1)/2-1\) degrees of freedom. Combining this fact with the expansion proves the theorem. \(\square\)

The degrees of freedom reflect the dimension of the restriction on the trailing symmetric covariance block: it has \(r(r+1)/2\) free entries, and imposing that it be a scalar multiple of the identity leaves one free scalar. Their difference is \(r(r+1)/2-1\). This explains why counting only the apparent equality among ordered eigenvalue values can give the wrong reference distribution.

Interpreting the Test for PCA Dimension

For a proposed \(k\), a small p-value is evidence against the model in which all directions after the first \(k\) have a common population variance. This provides one inferential check on a retained dimension. It is not a direct test that the covariance has rank \(k\): the null common value \(\tau\) is positive, and the trailing directions retain variance. Nor does failure to reject prove equality; it means the data did not provide sufficient evidence against the specified null at the chosen significance level.

A practical analysis may examine more than one proposed \(k\), but those tests should not be treated as independent conclusions or as an automatic rule for selecting dimension. The statistic is calibrated for a specified null and Gaussian sampling assumptions. Its chi-square reference is asymptotic, so small samples or violations of normality can make that approximation unreliable.

Common Mistakes

• Using the wrong tail summary. The null maximum-likelihood estimate pools trailing eigenvalues by their arithmetic mean. The likelihood ratio compares their product with the corresponding power of that mean.
• Using \(r-1\) degrees of freedom. This overlooks the covariance structure of the trailing block and its free eigenspace orientation. The asymptotic degrees of freedom are \(r(r+1)/2-1\).
• Calling this a test of zero residual variance. The null specifies a common positive trailing eigenvalue, not a zero eigenvalue or a rank-\(k\) covariance matrix.
• Treating non-rejection as proof. A non-significant result is not proof that trailing population eigenvalues are equal; it may also reflect limited information or an inaccurate approximation.