Three PCA Quantities That Answer Different Questions
In principal components analysis, an eigenvector, a loading, and a score are related but not interchangeable. Eigenvector coefficients describe the direction of a component in variable coordinates. Variable loadings summarize how that direction is scaled by the component variance. Observation scores give the coordinates of individual observations along the component directions.
This tutorial uses covariance-based PCA on centered quantitative variables. The distinction matters in applications such as analyzing two hypothetical measures of insurance claim experience: a direction in the two-variable space is not itself an observation, and neither is it the same as the scaled relationship between a variable and a component.
The loading convention must be stated because some sources use the word “loading” for the eigenvector coefficients themselves. Here, “eigenvector coefficient” means \(q_{jk}\), and “covariance-scale loading” means \(\sqrt{\lambda_k}q_{jk}\). A third quantity, the correlation between variable \(j\) and component \(k\), is not generally the same as either one.
The score is calculated from the centered observation and the eigenvector coefficients, not from the covariance-scale loadings. All quantities refer to the same fitted PCA, but they answer different questions:
| Quantity | Calculation | Interpretation |
|---|---|---|
| Eigenvector coefficient | Entry \(q_{jk}\) of \(q_k\) | Variable \(j\)'s coefficient in the direction of component \(k\) |
| Covariance-scale loading | \(\sqrt{\lambda_k}q_{jk}\) | Variable \(j\)'s entry in the scaled PCA factorization of \(S\) |
| Observation score | \(x_i'q_k\) | Observation \(i\)'s coordinate along component \(k\) |
How the Quantities Are Connected
Collect the unit eigenvectors as columns of \(Q\), and their eigenvalues on the diagonal of \(\Lambda\). The eigenvalue decomposition is \(S=Q\Lambda Q'\). The matrix of covariance-scale loadings is \(L=Q\Lambda^{1/2}\), so it satisfies \(S=LL'\). This scaled factorization explains why the loadings incorporate the component variances while the eigenvector coefficients do not.
For a centered observation \(x_i\), its vector of component scores is \(z_i=Q'x_i\). Thus, the coefficient vector for component \(k\), rather than its loading column, projects the observation onto that component. The sample covariance matrix of all component scores is \(\Lambda\): the components have variances equal to their eigenvalues and have zero sample covariance with one another.
A loading can also be related to a variable-score correlation. Let \(s_j=\sqrt{S_{jj}}\), the sample standard deviation of variable \(j\). The sample covariance between variable \(j\) and component \(k\) is \(\lambda_k q_{jk}\). Since the score has standard deviation \(\sqrt{\lambda_k}\), their correlation is
Consequently, in covariance-based PCA a covariance-scale loading is not automatically a correlation: it must be divided by the standard deviation of its variable. In particular, loadings and correlations can have different numerical values when the variables are not measured on a unit-standard-deviation scale.
Why the Score Covariance and Loading Factorization Hold
These relationships follow directly from the definitions and the eigenvector equations. For centered data, \(S=X'X/(n-1)\), and the score matrix is \(Z=XQ\). Therefore,
because the eigenvectors are orthonormal, so \(Q'Q=I\). This establishes that the score columns have sample variances \(\lambda_k\) and zero sample covariances for distinct components. For the loading factorization, \(L=Q\Lambda^{1/2}\) gives
The distinction is therefore structural: scores use the directions \(Q\) to transform observations, while covariance-scale loadings scale those directions by the square roots of the eigenvalues to factor the covariance matrix.
Worked Examples
Worked Example: Computing Coefficients, Loadings, and Scores
A hypothetical portfolio study records two quantitative measures for each policy group. The variables have sample means 10 and 20, and their covariance matrix is \(S=\begin{bmatrix}4&2\\2&4\end{bmatrix}\). For one group, the raw observations are \((12,20)\). Find both components' eigenvector coefficients, covariance-scale loadings, and scores for this group.
Step 1: Find the eigenvalues. The characteristic equation is \(\det(S-\lambda I)=(4-\lambda)^2-4=0\). Thus \((4-\lambda)^2=4\), giving eigenvalues \(\lambda_1=6\) and \(\lambda_2=2\).
Step 2: Find unit eigenvectors. For \(\lambda_1=6\), the equations imply equal coordinates, so \(q_1=(1,1)'/\sqrt{2}\). For \(\lambda_2=2\), the coordinates have opposite signs, so \(q_2=(1,-1)'/\sqrt{2}\). Each vector has length 1, and their dot product is 0.
Step 3: Calculate the loadings. For component 1, \(\sqrt{6}q_1=(\sqrt{3},\sqrt{3})'\), approximately \((1.732,1.732)'\). For component 2, \(\sqrt{2}q_2=(1,-1)'\). The coefficients remain the entries of the unit eigenvectors; the loadings are those coefficients scaled by the square root of the corresponding eigenvalue.
Step 4: Center the observation. Subtract the means: \(x=(12-10,20-20)'=(2,0)'\).
Step 5: Calculate the scores using the eigenvector coefficients. The first score is \(x'q_1=(2,0)(1,1)'/\sqrt{2}=2/\sqrt{2}=\sqrt{2}\). The second is \(x'q_2=(2,0)(1,-1)'/\sqrt{2}=2/\sqrt{2}=\sqrt{2}\). Both scores are approximately 1.414.
Check and method. The score variances over the full data set are the eigenvalues 6 and 2, as required by the score covariance result. The method avoids the common mistake of using the scaled loadings as projection coefficients.
Worked Example: Seeing the Effect of Eigenvalue Scaling
A hypothetical two-variable covariance matrix is \(S=\begin{bmatrix}9&0\\0&1\end{bmatrix}\). An observation has centered values \(x=(2,-1)'\). Calculate the coefficients, loadings, and scores, and explain why the loading values cannot be substituted for the coefficients when finding scores.
Step 1: Identify the eigenpairs. The diagonal entries give eigenvalues 9 and 1, with unit eigenvectors \(q_1=(1,0)'\) and \(q_2=(0,1)'\).
Step 2: Calculate the loadings. The first loading column is \(\sqrt{9}q_1=(3,0)'\); the second is \(\sqrt{1}q_2=(0,1)'\). Thus, \(L=\begin{bmatrix}3&0\\0&1\end{bmatrix}\), whereas \(Q=\begin{bmatrix}1&0\\0&1\end{bmatrix}\).
Step 3: Calculate the scores. The scores are \(x'q_1=(2,-1)(1,0)'=2\) and \(x'q_2=(2,-1)(0,1)'=-1\). The score vector is \((2,-1)'\).
Check and method. The score calculation uses the two unit eigenvectors, and the score variances equal 9 and 1. Multiplying by the loading columns instead would produce \((6,-1)'\), which is not the score vector. The error introduces eigenvalue scaling into the projection.
Worked Example: Converting a Loading to a Correlation
For a hypothetical two-variable analysis, the leading eigenpair is \(\lambda_1=3\), \(q_1=(1,1)'/\sqrt{2}\), and the sample standard deviation of the first variable is \(s_1=\sqrt{2}\). Find its covariance-scale loading and its correlation with the first component score.
Step 1: Calculate the loading. The first coefficient is \(q_{11}=1/\sqrt{2}\), so \(l_{11}=\sqrt{3}/\sqrt{2}=\sqrt{3/2}\), approximately 1.225.
Step 2: Convert to a correlation. Divide the loading by the variable's standard deviation: \(l_{11}/s_1=\sqrt{3/2}/\sqrt{2}=\sqrt{3}/2\), approximately 0.866.
Check and method. The loading is approximately 1.225, while the correlation is approximately 0.866. They differ because the variable's standard deviation is not 1. The method avoids interpreting a covariance-scale loading as a correlation without checking the variable scale.
Common Mistakes and Practical Interpretation
In a two-variable PCA, each eigenvector coefficient indicates the direction and relative weight assigned to one variable in a component. A score indicates where one centered observation falls along that direction. A covariance-scale loading includes the component's variance scale. These interpretations should be kept separate when explaining a numerical PCA result.
A common exam trap is to compute scores by multiplying centered observations by the loadings. For the first worked example, that mistake gives the first-component value \(2\sqrt{3}\), approximately 3.464, by taking \((2,0)(\sqrt{3},\sqrt{3})'\). The correct score is \(2/\sqrt{2}=\sqrt{2}\), approximately 1.414, because the projection uses \(q_1\), not \(\sqrt{\lambda_1}q_1\). The loading is scaled for the covariance factorization; it is not the score coefficient vector.
Before reporting a result, state the convention for loadings. Then label the quantities explicitly: give \(q_k\) for the eigenvector coefficients, \(L=Q\Lambda^{1/2}\) for covariance-scale loadings, and \(Z=XQ\) for centered observation scores. This prevents ambiguous statements such as “the loading is 0.7” when it is unclear whether 0.7 is an eigenvector entry, a scaled loading, or a correlation.
Check Your Understanding
For each problem, distinguish the requested PCA quantity before calculating.
- A component has eigenvalue 25 and unit eigenvector \(q=(0.6,0.8)'\). Find its covariance-scale loading vector. (Answer: \((3,4)'\))
- A centered observation is \(x=(1,-2)'\), and its component eigenvector is \(q=(0.6,0.8)'\). Find its score. (Answer: \(-1\))
- A covariance matrix is \(S=\begin{bmatrix}5&1\\1&5\end{bmatrix}\). Its eigenvalues are 6 and 4, with eigenvectors \(q_1=(1,1)'/\sqrt{2}\) and \(q_2=(1,-1)'/\sqrt{2}\). Give the covariance-scale loading columns. (Answer: \((\sqrt{3},\sqrt{3})'\) and \((\sqrt{2},-\sqrt{2})'\))
- A two-variable covariance matrix is \(S=\begin{bmatrix}16&0\\0&4\end{bmatrix}\). Give the eigenvector coefficient matrix, eigenvalues in descending order, and covariance-scale loading matrix. (Answer: \(Q=I\), eigenvalues \(16,4\), and \(L=\begin{bmatrix}4&0\\0&2\end{bmatrix}\))
- A variable has standard deviation 3 and covariance-scale loading 1.5 for a component. Find the variable-component score correlation. (Answer: \(0.5\))