Three PCA Quantities, Three Different Roles
A principal component analysis (PCA) produces several quantities that may all be shown beside variable names or observations in software output. They are related, but they answer different questions. Coefficients describe a direction in variable space; scores give each observation’s coordinate along that direction; and loadings describe how variables relate to component scores, under a convention that must be stated.
The distinction matters when interpreting a component or using its scores as features. For example, a coefficient of 0.8 is not an observation’s score, and it is not automatically a correlation of 0.8 between a variable and a component.
The term loading is not used identically by every source or software package. A frequently used alternative quantity is the covariance-scaled loading \(\sqrt{\lambda_j}v_{kj}\). We will name that quantity explicitly rather than treating the word “loading” as unambiguous.
Coefficients and Scores
The coefficient vector \(v_j\) specifies how to combine the centered variables. For observation \(i\), its score is the weighted sum \(t_{ij}=\sum_{k=1}^p x_{ik}v_{kj}\). A large positive score means the observation lies far in the positive direction of the coefficient vector; a negative score means it lies in the opposite direction. Scores are in the scale induced by the input variables, while coefficients are dimensionless direction coordinates.
For a symmetric covariance matrix, eigenvectors associated with distinct eigenvalues are orthogonal. Each coefficient vector is normalized to have length one. If an eigenvalue is repeated, vectors within its eigenspace need not be orthogonal merely because they are different eigenvectors; an orthonormal basis can be chosen for that eigenspace. This distinction is important when describing the geometry of PCA directions.
Worked Example: Coefficients, Scores, and Loadings From a Centered Matrix
A hypothetical data set has four observations and two centered variables. Let \(r=\sqrt{3}/2\), and suppose the centered data matrix is \(\begin{pmatrix}1&1/2+r\\-1&-1/2+r\\1&1/2-r\\-1&-1/2-r\end{pmatrix}\). Calculate the covariance matrix, the component coefficients, all observation scores, and the correlation loadings.
Method and covariance matrix. Denote the columns by \(X_1\) and \(X_2\). Their sums of squares are \(\sum_i x_{i1}^2=4\) and \(\sum_i x_{i2}^2=4(1/4+r^2)=4\), since \(r^2=3/4\). Their cross-product is \(\sum_i x_{i1}x_{i2}=2\): the first and third rows contribute \(1/2+r\) and \(1/2-r\), and the second and fourth rows contribute \(1/2-r\) and \(1/2+r\) after multiplying by their first-column entries. Thus the sample covariance matrix is
Coefficients. The unit eigenvectors are \(v_1=(1,1)^\mathsf{T}/\sqrt{2}\) and \(v_2=(1,-1)^\mathsf{T}/\sqrt{2}\). Multiplication by \(S\) gives \(Sv_1=2v_1\) and \(Sv_2=(2/3)v_2\), so the eigenvalues are \(\lambda_1=2\) and \(\lambda_2=2/3\). The entries of these vectors—not the eigenvalues—are the component coefficients.
Scores. Multiply each row by each coefficient vector. For example, observation 1 has \(t_{11}=[1+(1/2+r)]/\sqrt{2}=(3/2+r)/\sqrt{2}\) and \(t_{12}=[1-(1/2+r)]/\sqrt{2}=(1/2-r)/\sqrt{2}\). The same row-by-vector calculation gives the full score table below. Values are rounded to three decimal places only at this final step.
| Observation | PC 1 score | PC 2 score |
|---|---|---|
| 1 | 1.673 | -0.259 |
| 2 | -0.448 | -0.966 |
| 3 | 0.448 | 0.966 |
| 4 | -1.673 | 0.259 |
Correlation loadings. Each variable has sample standard deviation \(\sqrt{4/3}\). The correlation loading is \(\sqrt{\lambda_j}v_{kj}/\sqrt{S_{kk}}\), as justified below. Therefore PC 1 has loadings \(0.866\) for both variables, and PC 2 has loadings \(0.500\) for \(X_1\) and \(-0.500\) for \(X_2\). For instance, the PC 1 loading for \(X_1\) is \(\sqrt{2}(1/\sqrt{2})/\sqrt{4/3}=0.866\).
Check. The squared PC 1 scores sum to 6, and the squared PC 2 scores sum to 2. Dividing each by \(n-1=3\) gives sample score variances 2 and \(2/3\), exactly the two eigenvalues. The score columns also each sum to zero, as they should when centered observations are projected onto a fixed direction.
Common mistake avoided: Do not call \(1/\sqrt{2}\) a score. It is a coefficient. A score is calculated separately for each row of data by taking its dot product with the coefficient vector.
Why the Loading Formula Works
The correlation-loading formula follows from the eigenvector equation and the variance of a score. It applies to centered variables using the sample covariance matrix \(S\), a unit eigenvector \(v_j\), and a positive eigenvalue \(\lambda_j\). A zero-variance variable has no defined correlation loading.
For a random observation represented by the centered variable vector \(X\), the component score is \(T_j=X^\mathsf{T}v_j\).
\(\operatorname{Var}(T_j)=v_j^\mathsf{T}Sv_j=v_j^\mathsf{T}(\lambda_jv_j)=\lambda_j\), since \(v_j^\mathsf{T}v_j=1\).
\(\operatorname{Cov}(X_k,T_j)=(Sv_j)_k=\lambda_jv_{kj}\), by the definition of \(S\) and the eigenvector equation.
If \(S_{kk}>0\) and \(\lambda_j>0\), then \(\operatorname{Corr}(X_k,T_j)=\lambda_jv_{kj}/(\sqrt{S_{kk}}\sqrt{\lambda_j})=\sqrt{\lambda_j}v_{kj}/\sqrt{S_{kk}}\).
This derivation also explains why covariance-scaled loadings and correlation loadings are not always the same. The covariance-scaled quantity \(\sqrt{\lambda_j}v_{kj}\) equals the correlation loading only when variable \(k\) has unit standard deviation. In correlation-based PCA, the input variables have been standardized to unit variance, so the two quantities agree. In covariance-based PCA with variables on different scales, they generally do not.
Reading Loadings Under Different Conventions
The coefficients answer: which direction in variable space defines the component? The score answers: where does this observation fall along that direction? A correlation loading answers: how strongly and in which direction is this variable linearly associated with the component score? Those interpretations depend on using the appropriate quantity.
| Quantity | Formula under the stated convention | Interpretation |
|---|---|---|
| Coefficient | \(v_{kj}\) | Weight of variable \(k\) in the unit direction for component \(j\) |
| Observation score | \(t_{ij}=x_i^\mathsf{T}v_j\) | Coordinate of observation \(i\) along component \(j\) |
| Covariance-scaled loading | \(\sqrt{\lambda_j}v_{kj}\) | A variance-scaled coefficient; it is not generally a correlation |
| Correlation loading | \(\sqrt{\lambda_j}v_{kj}/\sqrt{S_{kk}}\) | Correlation of variable \(k\) with component score \(j\) |
Worked Example: A Score Is Not a Loading
A fitted PCA uses a unit component coefficient vector \(v=(2/3,1/3,2/3)^\mathsf{T}\). A centered observation has coordinates \(x=(2,-1,1)^\mathsf{T}\). Calculate its component score and explain why none of the coefficients is itself that score.
Method. A score is a projection, so multiply matching entries and add: \(t=x^\mathsf{T}v=2(2/3)+(-1)(1/3)+1(2/3)=4/3-1/3+2/3=5/3\). The score is therefore \(5/3\), or approximately \(1.667\) to three decimal places.
Check. The coefficient vector has squared length \(4/9+1/9+4/9=1\), as required. The score combines all three centered measurements; the individual coefficients \(2/3,1/3,2/3\) only specify their weights. This calculation is appropriate whether the centered observation is in the data used to fit PCA or is being projected onto an already fitted component.
Common mistake avoided: Reporting \(2/3\) as the observation’s component value confuses a variable’s weight with the observation’s coordinate. The dot product, including the observed centered values, is required.
Worked Example: Correlation Loadings in Covariance-Based PCA
Suppose a three-variable covariance matrix has eigenvalues \(9,4,1\), with unit eigenvectors \(v_1=(1,1,0)^\mathsf{T}/\sqrt{2}\), \(v_2=(1,-1,0)^\mathsf{T}/\sqrt{2}\), and \(v_3=(0,0,1)^\mathsf{T}\). Variable variances are \(6.5,6.5,1\). Calculate the covariance-scaled and correlation loadings for the first two variables on PC 1, and the correlation loading for variable 1 on PC 2.
Method. For PC 1, \(\lambda_1=9\), so the covariance-scaled loading for either of the first two variables is \(\sqrt{9}(1/\sqrt{2})=3/\sqrt{2}\), approximately \(2.121\). The variable standard deviation is \(\sqrt{6.5}=\sqrt{13/2}\). Divide the covariance-scaled loading by that standard deviation: \((3/\sqrt{2})/\sqrt{13/2}=3/\sqrt{13}\), approximately \(0.832\). Both PC 1 correlation loadings are \(0.832\).
For variable 1 on PC 2, the coefficient is \(1/\sqrt{2}\) and the eigenvalue is 4. The covariance-scaled loading is \(\sqrt{4}(1/\sqrt{2})=\sqrt{2}\), approximately \(1.414\). The correlation loading is \(\sqrt{2}/\sqrt{6.5}=2/\sqrt{13}\), approximately \(0.555\). These decimals are rounded to three places.
Check. The first two variables have equal variances and equal PC 1 coefficients, so their PC 1 correlation loadings match. The PC 2 coefficient for variable 1 is positive, so its correlation loading is positive. A correlation loading must lie between \(-1\) and \(1\); both computed values do.
Common mistake avoided: The covariance-scaled loading \(3/\sqrt{2}\) is not the correlation. It exceeds 1 here, which is possible for a covariance-scaled quantity but impossible for a correlation.
Checks and Exam Traps
Several quick checks help detect confusion or arithmetic errors:
- Check that each coefficient vector has unit length. For example, \((1/\sqrt{2},1/\sqrt{2})^\mathsf{T}\) has squared length \(1/2+1/2=1\).
- For centered data, each score column has mean zero. This follows because the rows of \(X\) sum to the zero vector, so the rows of \(Xv_j\) sum to zero.
- Using the sample covariance convention \(S=X^\mathsf{T}X/(n-1)\), the sample variance of score column \(j\) is \(\lambda_j\). Dividing by \(n\) instead would not match this convention.
- Correlation loadings are correlations and must be between \(-1\) and \(1\). Covariance-scaled loadings are not bounded that way.
- State whether the PCA uses a covariance matrix or a correlation matrix, and state what “loading” means in the calculation. Without those conventions, a loading value can be ambiguous.
The key relationship is that coefficients generate scores by projection, while loadings summarize associations between variables and scores. Keeping these roles separate makes PCA output easier to interpret and prevents a component’s direction from being mistaken for the location of an observation.
Check Your Understanding
For each problem, use the definitions and conventions stated in the question.
- A centered observation is \(x=(1,2)^\mathsf{T}\), and a unit component coefficient vector is \(v=(3/5,4/5)^\mathsf{T}\). Calculate the score. (Answer: \(11/5=2.2\))
- A component has eigenvalue \(4\) and unit coefficient vector \(v=(1/\sqrt{2},-1/\sqrt{2})^\mathsf{T}\). Calculate its covariance-scaled loadings. (Answer: \((\sqrt{2},-\sqrt{2})^\mathsf{T}\))
- In covariance-based PCA, a variable has variance 9, and its coefficient on a component with eigenvalue 4 is \(1/2\). Calculate its correlation loading. (Answer: \(1/3\))
- For centered data with \(n=5\), a component’s score column has sum of squares 12. Using \(S=X^\mathsf{T}X/(n-1)\), what eigenvalue corresponds to this component? (Answer: 3)
- A covariance-based PCA has a variable with variance 4, component eigenvalue 9, and coefficient \(-1/3\) for that variable. Calculate the covariance-scaled loading and correlation loading, and state which can be outside the interval from \(-1\) to \(1\). (Answer: Covariance-scaled loading \(=-1\); correlation loading \(=-1/2\); only the covariance-scaled loading is not generally bounded to that interval.)