Simultaneous Inference for Eigenvalues and Variance Shares

Why Simultaneous Inference Is Different

A principal components analysis often reports several eigenvalues and several explained-variance proportions together. If each estimate is paired with a separate 95% confidence interval, those intervals do not generally provide 95% coverage for the whole collection. Moreover, the estimates are dependent: a change in one eigenvalue can affect both other eigenvalues and the total variance used in every variance share.

This tutorial constructs joint asymptotic confidence statements for a specified collection of simple population eigenvalues and variance shares. The central object is their joint covariance. It enters both the joint calibration of maximum-statistic intervals and the shape of a Wald confidence ellipsoid.

For the sample covariance matrix, use \(S=n^{-1}\sum_{i=1}^n(X_i-\bar X)(X_i-\bar X)'\), with \(\bar X=n^{-1}\sum_iX_i\). Let \(\hat\lambda_j\) and \(\hat u_j\) be its ordered eigenvalues and corresponding unit eigenvectors. The sample share is \(\hat\rho_j=\hat\lambda_j/\operatorname{tr}(S)\). Assume the eigenvalues used in the analysis are distinct, so their first-order perturbations have the usual scalar form.

Joint First-Order Variation

Put \(Y=X-\mu\). The influence function for a simple eigenvalue is \(\psi_{\lambda_j}(X)=(u_j'Y)^2-\lambda_j\). For a variance share, the denominator also varies. Applying the quotient rule gives \(\psi_{\rho_j}(X)=\{\psi_{\lambda_j}(X)-\rho_j(\|Y\|^2-T)\}/T\). These formulas show directly why treating the estimates as independent is usually unjustified: each share includes the fluctuation in total variance, and several eigenvalue influence functions are computed from the same observation.

Choose \(q\) coordinates for \(\theta\), and stack their influence functions into a vector \(\psi(X)\). If \(E\|Y\|^4<\infty\), the multivariate central limit theorem and first-order eigenvalue and quotient expansions give \(\sqrt n(\hat\theta-\theta)\xrightarrow{d}N_q(0,V)\), where \(V=E[\psi(X)\psi(X)']\). This covariance includes every cross-covariance among the selected eigenvalues and shares.

Here is the justification. The first-order eigenvalue expansion for a simple eigenvalue gives \(\hat\lambda_j-\lambda_j=n^{-1}\sum_i\psi_{\lambda_j}(X_i)+o_p(n^{-1/2})\). The corresponding trace expansion is \(\operatorname{tr}(S)-T=n^{-1}\sum_i(\|X_i-\mu\|^2-T)+o_p(n^{-1/2})\). The quotient map \((a,b)\mapsto a/b\) is differentiable at \((\lambda_j,T)\), since \(T>0\). Its derivative gives the displayed share influence function. Stacking these expansions yields \(\sqrt n(\hat\theta-\theta)=n^{-1/2}\sum_i\psi(X_i)+o_p(1)\). The fourth-moment condition ensures finite covariance for this vector, so the multivariate central limit theorem gives the stated limit. For the plug-in estimate, the sample eigenvalues, eigenvectors and shares are consistent under the stated spectral gaps; the empirical covariance of the substituted influence vectors converges to \(V\) by the law of large numbers and the finite fourth moment. Thus \(\hat V\xrightarrow{p}V\). \(\square\)

In practice, define \(\hat\psi_i\) by substituting \(Y_i=X_i-\bar X\), \(\hat u_j\), \(\hat\lambda_j\), \(\hat\rho_j\), and \(\operatorname{tr}(S)\) into the influence formulas. Then set \(\hat V=n^{-1}\sum_i(\hat\psi_i-\bar{\hat\psi})(\hat\psi_i-\bar{\hat\psi})'\). This estimates the covariance of the root-\(n\) limit; the estimated covariance of \(\hat\theta\) itself is \(\hat V/n\).

Two Ways to Make Statements Simultaneous

One approach forms a maximum of standardized errors. Let \(\hat s_a^2=\hat V_{aa}\), for \(a=1,\ldots,q\), and let \(\hat R\) be the correlation matrix corresponding to \(\hat V\). For a chosen error probability \(\alpha\), let \(\hat c_{1-\alpha}\) be the \(1-\alpha\) quantile of \(\max_a|Z_a|\) for \(Z\sim N_q(0,\hat R)\). Subject to the regularity conditions that the relevant marginal variances are positive and the limiting quantile is unique, intervals \(\hat\theta_a\pm\hat c_{1-\alpha}\hat s_a/\sqrt n\) have joint asymptotic coverage \(1-\alpha\). Unlike separate marginal intervals, their critical value reflects the dependence encoded in \(\hat R\).

To verify the coverage, the event that every interval covers is exactly the event \(\max_a|\sqrt n(\hat\theta_a-\theta_a)/\hat s_a|\leq\hat c_{1-\alpha}\). By the joint limit and \(\hat s_a\xrightarrow{p}\sqrt{V_{aa}}\), the vector inside the maximum converges in distribution to a mean-zero normal vector with correlation matrix \(R\). Consistency of \(\hat R\) and uniqueness of the quantile imply \(\hat c_{1-\alpha}\xrightarrow{p}c_{1-\alpha}\). The limiting maximum has a continuous distribution at that quantile, so convergence in distribution gives the claimed limiting probability. \(\square\)

A second approach uses a joint Wald ellipsoid. If \(V\) is nonsingular, the quadratic form \(n(\hat\theta-\theta)'\hat V^{-1}(\hat\theta-\theta)\) converges in distribution to a chi-square random variable with \(q\) degrees of freedom. The ellipsoid at the chi-square quantile has asymptotic joint coverage \(1-\alpha\); its coordinate projections give simultaneous intervals. This method uses the full covariance matrix in the ellipsoid's orientation, while the maximum method calibrates the largest standardized coordinate directly.

Worked Examples

Singularity, Interpretation, and a Common Pitfall

If all \(p\) variance shares are included, they sum to one. Their joint covariance matrix is therefore singular: the deviations in the shares cannot vary freely in all \(p\) directions. More generally, redundant coordinates or exact constraints can make a selected joint covariance singular. Do not invert a singular matrix as though it were nonsingular. One may omit a redundant share, use a maximum-statistic method with positive marginal variances, or use a Wald construction on the nonredundant subspace with the appropriate rank and generalized inverse.

A common mistake is to report several marginal 95% intervals and describe them as a 95% joint confidence statement. The individual coverage statements do not imply that all intervals cover simultaneously with probability 0.95. Another mistake is to calculate standard errors separately and set all cross-covariances to zero. As the share example shows, the common random denominator can induce substantial dependence. The correction is to form the joint influence vector first, estimate its full covariance, and calibrate the complete collection together.