Eigenvectors, Loadings, and Score Coefficients

Three Quantities That Are Easy to Confuse

A principal component can be described in several numerical ways. Its eigenvector gives the coefficients used to form a score; the score is the resulting random variable or observed data column; and the covariances between that score and the original variables describe how the variables relate to it. These quantities are connected, but they are not interchangeable. In particular, an eigenvector entry is not generally the covariance between that variable and its component.

We work with a random vector \(X=(X_1,\ldots,X_p)^T\) whose mean vector \(\mu=E[X]\) and covariance matrix \(\Sigma=\operatorname{Cov}(X)\) are finite. The principal components considered here are those of the covariance matrix. Since \(\Sigma\) is real and symmetric, it has orthonormal eigenvectors. Let \(q_j\) be a unit eigenvector with eigenvalue \(\lambda_j\), so \(\Sigma q_j=\lambda_j q_j\).

Centering matters because PCA describes variation around the mean. For an observed vector \(x\), the component score is the number \(q_j^T(x-\mu)\), not the coefficient vector \(q_j\). A score is one value for one observation; the coefficient vector specifies how to calculate that value.

What the Eigenvector Determines

The eigenvector is chosen to describe a direction of variation in the original variable coordinates. Its eigenvalue is the variance of the score in that direction. The following identities also give the covariances with each original variable.

Proof. Write \(X_c=X-\mu\), so \(E[X_c]=0\) and \(\operatorname{Cov}(X_c)=\Sigma\). By bilinearity of covariance,

because \(q_j\) is a unit vector. For the vector of covariances, taking covariance with each coordinate of \(X_c\) gives

Finally,

when \(q_j^Tq_k=0\). This proves all three identities. \(\square\)

The second identity is the key distinction: the covariance between variable \(X_i\) and score \(Z_j\) is the \(i\)th entry of \(\lambda_jq_j\), or \(\lambda_j q_{ij}\). It equals the eigenvector entry multiplied by the eigenvalue. The vector \(q_j\) itself consists of score coefficients, not variable-score covariances.

Covariance Loadings and Correlation Loadings

The phrase “loading” is not used identically in every PCA text or software package. One precise option is to call \(\operatorname{Cov}(X_i,Z_j)=\lambda_j q_{ij}\) the covariance loading of variable \(i\) on component \(j\). Another commonly reported quantity is the correlation between the variable and the score. If \(\operatorname{Var}(X_i)=\sigma_{ii}>0\) and \(\lambda_j>0\), that correlation is

The formula follows by substituting \(\operatorname{Cov}(X_i,Z_j)=\lambda_jq_{ij}\) and \(\operatorname{Var}(Z_j)=\lambda_j\) into the definition of correlation. Some sources call these correlations “loadings”; others use “loading” for a rescaled coefficient such as \(\sqrt{\lambda_j}q_{ij}\). State the formula being used rather than inferring a meaning from the label alone.

For PCA of standardized variables, the same identities apply to the standardized vector and its correlation matrix. In that setting each original standardized variable has variance one, so the correlation between it and the score is \(\sqrt{\lambda_j}q_{ij}\). This is not generally equal to \(q_{ij}\). In covariance PCA with variables in different units, the marginal standard deviations must remain in the denominator when calculating correlations.

From a Model to an Observed Score

A score coefficient is a rule applied to centered data; it is not itself an observation. For an observed vector \(x\), subtract the model mean first and take the inner product with the eigenvector. The following example carries out those steps and checks the associated covariance vector.

The Same Identities for Sample PCA

For sample PCA, let \(x_1,\ldots,x_n\) be \(p\)-dimensional observations, let \(\bar{x}\) be their sample mean, and define the centered data matrix \(X_c\) to have rows \((x_i-\bar{x})^T\). With sample covariance \(S=X_c^TX_c/(n-1)\), suppose \(Sq_j=\hat{\lambda}_j q_j\), where \(q_j\) is a unit eigenvector. The vector of observed scores is \(t_j=X_cq_j\). Directly,

The first expression is the sample variance of the score column; the second gives the sample covariances of the original variables with that score. These are the sample counterparts of the population identities. If a different covariance divisor is used, the eigenvalues and covariance calculations must be interpreted with that same convention.

Common Mistake and Why the Distinction Matters

Wrong reasoning: “The first eigenvector is \((3/5,4/5)^T\), so the covariances of the two variables with the first score are \(3/5\) and \(4/5\).” This treats score coefficients as variable-score covariances. Correction: multiply the unit eigenvector by its eigenvalue: with eigenvalue \(9\), those covariances are \(27/5\) and \(36/5\). If correlations are wanted instead, divide each covariance by the product of the corresponding standard deviations.

Keeping these quantities separate prevents errors in reporting and interpretation. Coefficients tell how a score is constructed; the score gives a coordinate for an observation; covariance loadings quantify variable-score covariation in the variables' original units; and correlation loadings standardize that association. Always identify which quantity is being reported and use its defining formula.