Sign-Invariant Inference for a Single Eigenvector

Why Eigenvector Inference Must Ignore Sign

A principal component is a direction, not an oriented arrow. If $v_1$ is a unit eigenvector of a covariance matrix, then $-v_1$ is an equally valid eigenvector for the same eigenvalue. Consequently, an inferential statement that accepts one sign but rejects the other would depend on an arbitrary convention rather than on the statistical problem.

This tutorial constructs a confidence set for a single, well-separated population eigenvector. It uses angular coordinates around the sample direction and a Wald statistic whose value is unchanged by eigenvector sign choices. The result is an asymptotic confidence set: its coverage approaches the requested level as the sample size grows under the assumptions stated below.

Definition: A direction represented by a unit vector $v$ is the equivalence class $[v]=\{v,-v\}$. Its rank-one projector is $P=vv^T$. Since $(-v)(-v)^T=vv^T$, the projector represents the direction without choosing a sign. For two unit vectors $u$ and $v$, their projective angle is $\theta(u,v)=\arccos(|u^Tv|)$, which lies in $[0,\pi/2]$.

The absolute value in the angle is essential: vectors pointing in opposite orientations represent the same direction. The angle also has a projector expression, $ \sin^2\theta(u,v)=1-(u^Tv)^2 $. Thus the angle is zero precisely when $uu^T=vv^T$. Inference for a principal direction can therefore be reported either as a set of directions or, equivalently, as a set of rank-one projectors.

Model and Tangent Coordinates

Suppose $X_1,\ldots,X_n$ are independent observations from a $p$-variate normal distribution $N_p(\mu,\Sigma)$, where $\Sigma$ is positive definite. Write the ordered eigenvalues and corresponding orthonormal eigenvectors of $\Sigma$ as $\lambda_1>\lambda_2>\cdots>\lambda_p>0$ and $v_1,\ldots,v_p$. The strict inequalities make each population eigenvector direction identifiable. Let $\hat\lambda_1>\cdots>\hat\lambda_p$ and $\hat v_1,\ldots,\hat v_p$ denote the ordered eigenvalues and unit eigenvectors of the sample covariance matrix $S$. The choice of sign for each $\hat v_j$ is arbitrary.

For a candidate unit vector $u$, choose $s\in\{-1,1\}$ so that $s\hat v_1^Tu\geq 0$, and define its tangent coordinates relative to the sample eigenbasis by $t_j(u)=\sqrt n\,\hat v_j^T(su)$, for $j=2,\ldots,p$. If the inner product is zero, either sign may be chosen. These coordinates measure the candidate’s displacement away from the sample direction in the sample eigenvector directions orthogonal to $\hat v_1$.

Under the stated normal model, the first-order eigenvector perturbation result gives the asymptotic coordinate variance $\lambda_1\lambda_j/(\lambda_1-\lambda_j)^2$ in direction $v_j$. The sample version is $\hat\tau_j^2=\hat\lambda_1\hat\lambda_j/(\hat\lambda_1-\hat\lambda_j)^2$. These are variances for the coordinates scaled by $\sqrt n$, not for unscaled eigenvector errors.

Wald statistic: For a candidate unit vector $u$, define $$ W_n(u)=\sum_{j=2}^{p}\frac{t_j(u)^2}{\hat\tau_j^2} =\sum_{j=2}^{p}\frac{n\{\hat v_j^T(su)\}^2(\hat\lambda_1-\hat\lambda_j)^2}{\hat\lambda_1\hat\lambda_j}. $$ For confidence level $1-\alpha$, the sign-invariant Wald confidence set is the collection of directions $[u]$ for which $W_n(u)\leq \chi^2_{p-1,\,1-\alpha}$. Here $\chi^2_{p-1,\,1-\alpha}$ is the $1-\alpha$ quantile of a chi-square distribution with $p-1$ degrees of freedom.

The statistic uses only the components orthogonal to the estimated leading direction. Its weights account for the different sampling variances of those components. Directions near an eigenvalue tie have larger perturbation variance, reflected by the small denominator $(\hat\lambda_1-\hat\lambda_j)^2$ in $\hat\tau_j^2$. The confidence set is generally an anisotropic neighborhood rather than a circular cone.

Why the Wald Set Has Asymptotic Coverage

The key asymptotic fact is the joint limit of the leading sample eigenvector’s tangent coordinates. Choose the sign of $\hat v_1$ to align it with $v_1$, and choose the signs of the other sample eigenvectors to align them with their population counterparts. Under the fixed-dimensional normal model and the strict eigenvalue ordering,

\sqrt n\,v_j^T\hat v_1 \xrightarrow{d} N\!\left(0,\frac{\lambda_1\lambda_j}{(\lambda_1-\lambda_j)^2}\right), \qquad j=2,\ldots,p,

and the limiting coordinates are independent. To see the source of this limit, first-order perturbation gives the $v_j$ component of the leading eigenvector error as $v_j^T(S-\Sigma)v_1/(\lambda_1-\lambda_j)$. For normal observations, the covariance between the corresponding sample cross-products in distinct eigendirections is zero, and their variances are $\lambda_1\lambda_j$. The multivariate central limit theorem therefore yields the displayed joint normal limit. The eigenvalue gaps ensure the perturbation expansion is valid.

Theorem: Under the normal model above, with fixed $p$, distinct positive population eigenvalues, and $n$ tending to infinity, the Wald confidence set has asymptotic coverage $1-\alpha$ for the population direction $[v_1]$: $$ P\!\left\{W_n(v_1)\leq \chi^2_{p-1,\,1-\alpha}\right\}\longrightarrow 1-\alpha. $$

Proof. Use the sign conventions in the preceding paragraph temporarily to describe the limit. Orthogonality of the estimated eigenvectors gives, for $j\geq2$, $\hat v_j^Tv_1=-\hat v_1^T v_j+o_p(n^{-1/2})$. Indeed, expand $v_1$ in the estimated orthonormal eigenbasis; its component along $\hat v_1$ is $1+O_p(n^{-1})$, and the remaining components are $O_p(n^{-1/2})$, so the first-order cross-components have opposite signs. Consequently, the vector with coordinates $\sqrt n\,\hat v_j^Tv_1$, $j=2,\ldots,p$, converges to a centered normal vector with diagonal covariance entries $\tau_j^2=\lambda_1\lambda_j/(\lambda_1-\lambda_j)^2$. The sign alignment $s$ in the definition can reverse all these coordinates, which does not change their joint squared, variance-standardized length.

Because $\hat\lambda_j\xrightarrow{p}\lambda_j$, each $\hat\tau_j^2\xrightarrow{p}\tau_j^2$. Slutsky’s theorem then implies that $W_n(v_1)$ converges in distribution to the sum of squares of $p-1$ independent standard normal variables, namely $\chi^2_{p-1}$. This distribution is continuous at its $1-\alpha$ quantile, so the probability of falling below that quantile converges to $1-\alpha$, as claimed. $\square$

Worked Examples

Worked Example: A Two-Dimensional Angular Confidence Set

Suppose $p=2$, $n=1000$, and the observed sample eigenvalues are $\hat\lambda_1=9$ and $\hat\lambda_2=4$. For a 95% confidence set, $\chi^2_{1,0.95}\approx3.841$. There is just one tangent coordinate, and its estimated variance is \[ \hat\tau_2^2=\frac{9\cdot4}{(9-4)^2}=\frac{36}{25}=1.44. \]

If $\theta=\arccos(|\hat v_1^Tu|)$ is the angle between a candidate direction and the sample direction, then $|\hat v_2^Tu|=\sin\theta$. The candidate is included exactly when \[ \frac{1000\sin^2\theta}{1.44}\leq3.841, \qquad \sin^2\theta\leq\frac{3.841(1.44)}{1000}=0.005531. \] Thus $\theta\leq\arcsin(\sqrt{0.005531})\approx0.0744$ radians, or about $4.27^\circ$. The set is a cone around the sample direction; it includes both orientations of every vector in that cone.

Worked Example: An Anisotropic Set in Three Dimensions

Let the observed eigenvalues be $9,4,1$, let $n=500$, and consider a candidate whose coordinates in the sample eigenbasis are $u=(\sqrt{0.9976},0.04,0.02)^T$. This is a unit vector because $0.9976+0.04^2+0.02^2=1$. Its first coordinate is positive, so the aligning sign is $s=1$.

The estimated tangent variances are \[ \hat\tau_2^2=\frac{9\cdot4}{(9-4)^2}=1.44, \qquad \hat\tau_3^2=\frac{9\cdot1}{(9-1)^2}=\frac{9}{64}=0.140625. \] The tangent coordinates are $t_2=\sqrt{500}(0.04)$ and $t_3=\sqrt{500}(0.02)$. Hence \[ W_n(u)=500\left(\frac{0.04^2}{1.44}+\frac{0.02^2}{0.140625}\right) =500(0.0011111+0.0028444)\approx1.978. \]

Since the 95% cutoff for two degrees of freedom is $5.991$, this candidate is included. A displacement of the same size in the third eigendirection costs more in the statistic than one in the second: the estimated variance $0.140625$ is smaller than $1.44$. The confidence set therefore adjusts for direction-specific uncertainty.

Worked Example: Checking Sign Invariance Numerically

Suppose the tangent coordinates for a candidate are $t=(0.5,0.1)^T$, with estimated variances $1.44$ and $0.140625$. Then \[ W=\frac{0.5^2}{1.44}+\frac{0.1^2}{0.140625} =0.17361+0.07111\approx0.24472. \]

Now replace $\hat v_1$ by $-\hat v_1$. The aligning sign changes from $s$ to $-s$, so the aligned candidate $su$ reverses to $-su$. Both tangent coordinates consequently reverse, from $(0.5,0.1)$ to $(-0.5,-0.1)$. The estimated eigenvalues and the diagonal variance estimates are unchanged, so the quadratic form remains $0.24472$. If instead the sign of $\hat v_2$ is reversed, only the first coordinate changes sign; its square, and therefore the statistic, is unchanged. These are changes of representation, not changes in the candidate direction or evidence.

Interpretation and Common Pitfall

The confidence set can be reported directly as a set of unit-vector directions, with $u$ and $-u$ identified, or mapped to projectors $uu^T$. The projector form is particularly useful when results are compared across software or analyses that choose eigenvector signs differently. The angular description gives an intuitive summary, while the weighted Wald statistic retains the distinct uncertainty in each tangent direction.

The guarantee is asymptotic and depends on a simple, separated leading eigenvalue, fixed dimension, and the stated normal sampling model. If the leading eigenvalue is repeated, there is no unique population leading direction to cover; the relevant target is a leading eigenspace. If the eigengap is small, the formula indicates greater uncertainty, but finite-sample coverage may still be poor. These are not reasons to reinterpret a sign change as a different result: sign invariance is exact for the statistic, while its coverage claim remains subject to its assumptions.

Mistake: Treating $v_1$ and $-v_1$ as different population parameters. Correction: They define the same direction and the same rank-one projector.
Mistake: Saying that reversing $\hat v_1$ leaves the aligned vector $su$ unchanged. Correction: It reverses $s$, and therefore reverses $su$ and every tangent coordinate. The quadratic form is unchanged because the coordinates are squared.
Mistake: Using an unweighted angle cutoff as if uncertainty were equal in every tangent direction. Correction: The Wald set weights each coordinate by its estimated variance; an unweighted angular summary does not generally reproduce that set.
Mistake: Reporting the confidence set as if it guaranteed exact coverage at finite $n$. Correction: The chi-square calibration here gives asymptotic, not generally exact, coverage.

Check Your Understanding

Use the definitions and the Wald construction to answer the following.

Why do $v$ and $-v$ represent the same principal direction, and what matrix represents that direction without a sign choice?
In the Wald statistic, why are there $p-1$ tangent coordinates rather than $p$?
What happens to the aligned candidate $su$ and its tangent coordinates when the sign of $\hat v_1$ is reversed? Why does the statistic not change?
For a two-dimensional problem, explain how the Wald inequality becomes a bound on the projective angle.
Name two assumptions needed for the stated asymptotic coverage result, and explain what target changes when the leading population eigenvalue is repeated.