Why a Principal Component Can Point Either Way
A principal component is often described by a direction, but a direction has two possible orientations. If a unit eigenvector points along that direction, its negative points along the same line in the opposite orientation. The numerical signs of the loadings and scores therefore depend on which orientation is reported; the underlying PCA geometry does not.
This matters when reading software output, comparing results from two analyses, or explaining a loading pattern. One output might report a component with positive loadings and scores where another reports negative loadings and scores. That difference alone does not indicate a different fitted direction or a different representation of the observations.
Here \(x_i\) is the observation after applying the centering and, if appropriate, scaling used in the PCA. The entries of \(q_j\) are its loadings in the eigenvector-coefficient convention. When PCA is based on a covariance matrix, \(x_i\) is centered in the original units. When it is based on a correlation matrix, the variables are centered and standardized first. The sign argument works in either case, provided the same transformed data are used throughout.
The Sign Change, Step by Step
Let \(S\) be the covariance matrix used for PCA and let \(q_j\) be a unit eigenvector satisfying \(S q_j=\lambda_j q_j\). Multiplying this equation by \(-1\) shows that \(-q_j\) is also an eigenvector for the same eigenvalue. The variance associated with that component is unchanged.
For any centered observation \(x_i\), the score under the reversed eigenvector is
Thus the score changes sign. But the component’s projected vector is the score multiplied by the direction vector. Both factors change sign, so their product does not:
The same reasoning applies to a reconstruction using several components. If \(Q\) contains unit eigenvectors as columns and \(t_i=Q^\mathsf{T}x_i\) contains the corresponding scores, the reconstruction using those components is \(Qt_i\). Reversing the sign of any column reverses its score as well, leaving that column’s contribution to the reconstruction unchanged.
Worked Example: Reversing One Eigenvector
A two-variable covariance matrix is \(S=\begin{pmatrix}5&1\\1&5\end{pmatrix}\). Use the leading eigenvector to calculate the score and projection of the centered observation \(x=(4,2)^\mathsf{T}\), then repeat with the eigenvector’s sign reversed. Name the method and identify the error it avoids.
Method. Find a unit eigenvector, calculate its dot product with the observation, and multiply the score by the eigenvector to obtain the projection.
For \(q_1=(1,1)^\mathsf{T}/\sqrt{2}\), the eigenvector equation is
So \(q_1\) is an eigenvector with eigenvalue 6. Its length is \(\sqrt{(1/\sqrt{2})^2+(1/\sqrt{2})^2}=1\), as required for the score and projection calculations. The score of \(x\) is
The projection onto this component is
Now use \(q'_1=-q_1=(-1,-1)^\mathsf{T}/\sqrt{2}\). Its score is \(t'_{i1}=-3\sqrt{2}\), and its projection is
The score changes from \(3\sqrt{2}\) to \(-3\sqrt{2}\), while the projected observation remains \((3,3)^\mathsf{T}\). Common mistake avoided: treating a negative score as evidence that the observation’s projection has moved to the opposite side of the axis, without also reversing the eigenvector.
Variance Is Also Unchanged
Suppose \(S\) is the covariance matrix for the centered variables used in the PCA, \(q_j\) is a unit eigenvector, and \(t_{ij}=q_j^\mathsf{T}x_i\) is the score. The variance of the score is
The second equality uses the eigenvector equation, and the final equality uses \(q_j^\mathsf{T}q_j=1\). With \(-q_j\), the calculation gives \((-q_j)^\mathsf{T}S(-q_j)=q_j^\mathsf{T}S q_j=\lambda_j\). Equivalently, negating all scores changes neither their squared deviations from their mean nor their variance. The eigenvalue therefore remains the component variance.
This result concerns reversing the sign of an eigenvector, not changing the eigenvalue or choosing a different component. For a given eigenvector, the score signs reverse together, so comparisons among observations along that component are retained: if one observation has a larger score than another before reversal, it has a smaller score after reversal. The ordering reverses only because the chosen orientation has reversed.
Worked Example: Scores and Variances for a Centered Data Set
Three centered observations are \(x_1=(2,0)^\mathsf{T}\), \(x_2=(0,2)^\mathsf{T}\), and \(x_3=(-2,-2)^\mathsf{T}\). Calculate their scores on both principal components, reverse both eigenvector signs, and check the component variances. Name the method and the common mistake it avoids.
Method. Compute the sample covariance matrix, find its unit eigenvectors, take dot products for the scores, and use the sample score variance with denominator \(n-1\).
The observations sum to zero, so they are centered. Their sample covariance matrix is
The unit eigenvectors are \(q_1=(1,1)^\mathsf{T}/\sqrt{2}\) and \(q_2=(1,-1)^\mathsf{T}/\sqrt{2}\), with eigenvalues 6 and 2. For example, \(S q_1=(6,6)^\mathsf{T}/\sqrt{2}=6q_1\), and \(S q_2=(2,-2)^\mathsf{T}/\sqrt{2}=2q_2\).
Taking dot products gives the scores:
For the first component, the sample mean score is \((\sqrt{2}+\sqrt{2}-2\sqrt{2})/3=0\), and the sample variance is \((2+2+8)/2=6\). For the second component, the mean is \((\sqrt{2}-\sqrt{2}+0)/3=0\), and the sample variance is \((2+2+0)/2=2\). These match the respective eigenvalues.
Replacing \(q_1\) with \(-q_1\) changes the first score column to \((-\sqrt{2},-\sqrt{2},2\sqrt{2})\). Replacing \(q_2\) with \(-q_2\) changes the second to \((-\sqrt{2},\sqrt{2},0)\). The squared scores, and hence the two sample variances, are unchanged. Common mistake avoided: changing a score’s sign but then claiming its variance changes sign. Variance is based on squared deviations, not signed scores.
Reconstruction with More Than One Component
When several components are retained, each contributes its own score-times-direction vector. Reversing a subset of component signs leaves each contribution intact, as long as the corresponding scores are reversed too. This remains true for a truncated reconstruction: it applies to every retained component, whether or not all components are used.
The following example uses two orthonormal component directions. It checks a full reconstruction and shows that a one-component approximation is invariant as well.
Worked Example: Two-Component Reconstruction Under Sign Changes
A two-variable PCA has unit eigenvectors \(q_1=(3/5,4/5)^\mathsf{T}\) and \(q_2=(-4/5,3/5)^\mathsf{T}\). For the centered observation \(x=(5,0)^\mathsf{T}\), compute both scores and reconstruct \(x\). Then reverse both signs and check the reconstruction. Name the method and the common mistake it avoids.
Method. Project onto each orthonormal component by a dot product, then add the component projections \(t_{ij}q_j\).
The directions are unit length because \((3/5)^2+(4/5)^2=1\) and \((-4/5)^2+(3/5)^2=1\). They are orthogonal because \((3/5)(-4/5)+(4/5)(3/5)=0\). The two scores are
The first component contributes \(3q_1=(9/5,12/5)^\mathsf{T}\). The second contributes \(-4q_2=(16/5,-12/5)^\mathsf{T}\). Adding the contributions gives
After reversing both directions, the scores become \(-3\) and \(4\). The contributions are \((-3)(-q_1)=3q_1\) and \(4(-q_2)=-4q_2\), exactly the vectors calculated above. If only the first component is retained, its reconstruction is \((9/5,12/5)^\mathsf{T}\) before and after the sign reversal. Common mistake avoided: changing the sign of a component direction in a reconstruction formula while leaving its score unchanged.
Interpreting Loadings and Comparing Output
Reversing an eigenvector reverses every loading in that eigenvector, not just one entry. Thus a loading pattern such as \((0.6,0.8)\) becomes \((-0.6,-0.8)\). The relative signs within the component are unchanged: both variables still load with the same sign relative to one another. A pattern such as \((0.6,-0.8)\) likewise becomes \((-0.6,0.8)\), preserving the fact that the two loadings have opposite signs.
A sign convention can make printed results easier to compare, for example requiring the loading with the largest absolute value to be positive. Such a convention is a reporting choice, not a mathematical property of PCA. If there is a tie for largest absolute loading, that rule alone does not choose a unique orientation. Regardless of the convention, reversing all entries of one eigenvector and the corresponding score column leaves the PCA representation unchanged.
- Scores: every score on the reversed component changes sign.
- Loadings: every entry of that component’s eigenvector changes sign.
- Eigenvalue and score variance: unchanged.
- Direction as an axis and projected observations: unchanged.
- Reconstruction: unchanged when scores and directions are changed together.
When comparing two sets of PCA output, first check whether their corresponding eigenvectors differ only by an overall minus sign. If so, compare the loading patterns and scores after reversing one set consistently. Do not infer that a component has changed its substantive meaning solely from the sign of its scores or loadings.
Check Your Understanding
For each problem, track the eigenvector and its corresponding scores together. Answers are provided for checking.
- A unit component vector is \(q=(3/5,4/5)^\mathsf{T}\), and a centered observation is \(x=(10,5)^\mathsf{T}\). Find the score and projection using \(q\), then give them using \(-q\). (Answer: score \(10\) and projection \((6,8)^\mathsf{T}\); with \(-q\), score \(-10\) and the same projection \((6,8)^\mathsf{T}\).)
- A component has scores \((2,-1,3)\) across three observations. If its eigenvector is reversed, state the new score vector and explain whether the sample variance changes. (Answer: \((-2,1,-3)\); the sample variance is unchanged because the mean and each deviation change sign, leaving squared deviations unchanged.)
- For \(S=\begin{pmatrix}7&2\\2&7\end{pmatrix}\), use \(q=(1,1)^\mathsf{T}/\sqrt{2}\) and \(x=(3,-1)^\mathsf{T}\). Find the eigenvalue, score, and projection; then state the reversed-sign result. (Answer: eigenvalue \(9\), score \(\sqrt{2}\), projection \((1,1)^\mathsf{T}\); with \(-q\), score \(-\sqrt{2}\) and projection \((1,1)^\mathsf{T}\).)
- Two orthonormal component vectors are \(q_1=(1,0)^\mathsf{T}\) and \(q_2=(0,1)^\mathsf{T}\), and \(x=(2,-3)^\mathsf{T}\). Find the two scores and full reconstruction if only \(q_2\) is reversed. (Answer: scores are \(2\) and \(3\) after reversal; contributions are \((2,0)^\mathsf{T}\) and \((0,-3)^\mathsf{T}\), giving \((2,-3)^\mathsf{T}\).)