Why PCA Plots Need a Scale Convention
A PCA plot can show variables as arrows, observations as points, or both. These pictures are useful summaries, but their coordinates are not self-explanatory: the same principal components can be plotted using different scalings, and a two-dimensional display omits information whenever more than two components matter. Correct interpretation begins by asking what each plotted coordinate represents.
This tutorial focuses on correlation circles and biplots. It uses standardized variables and PCA from the correlation matrix. Let \(Z_j\) denote standardized variable \(j\), with sample variance 1, and let \(R\) be the sample correlation matrix. Write \(v_k\) for a unit eigenvector of \(R\), with eigenvalue \(\lambda_k\), and define the score of observation \(i\) on component \(k\) by \(t_{ik}=v_k^\mathsf{T}z_i\). Thus the score has sample variance \(\lambda_k\). The sign of an eigenvector, and therefore of its scores and coordinates, may be reversed without changing the PCA.
Deriving the Correlation Coordinates
The formula follows directly from covariance and variance. Since the standardized variables have covariance matrix \(R\), the covariance between \(Z_j\) and \(T_k=v_k^\mathsf{T}Z\) is the \(j\)th entry of \(Rv_k\). The eigenvector equation gives \(Rv_k=\lambda_kv_k\), while \(\operatorname{Var}(Z_j)=1\) and \(\operatorname{Var}(T_k)=\lambda_k\). If \(\lambda_k>0\), dividing the covariance by the product of standard deviations gives:
The restriction matters. At \(\lambda_k=0\), the score has variance zero, so the denominator in the correlation definition is zero. Although \(\operatorname{Cov}(Z_j,T_k)=0\), that fact does not make the correlation zero: correlation is undefined when either variable has zero variance.
Worked Example: Reading a Two-Variable Correlation Circle
Suppose two standardized variables have a correlation matrix with eigenvalues \(\lambda_1=1.6\) and \(\lambda_2=0.4\), and corresponding unit eigenvectors \(v_1=(1/\sqrt{2},1/\sqrt{2})^\mathsf{T}\) and \(v_2=(1/\sqrt{2},-1/\sqrt{2})^\mathsf{T}\). Find their two-dimensional correlation-circle coordinates and recover their correlation.
Method. Apply \(c_{jk}=\sqrt{\lambda_k}v_{jk}\) to each variable and component. Both eigenvalues are positive, so both component correlations are defined.
For component 1, \(\sqrt{1.6}=1.2649\) to four decimals and \(1/\sqrt{2}=0.7071\). Each variable therefore has first coordinate \(1.2649(0.7071)=0.8944\). For component 2, \(\sqrt{0.4}=0.6325\), so the coordinates are \(0.6325(0.7071)=0.4472\) for variable 1 and \(0.6325(-0.7071)=-0.4472\) for variable 2.
The plotted vectors are therefore \((0.8944,0.4472)\) and \((0.8944,-0.4472)\). Each has squared length \(0.8944^2+0.4472^2=0.8000+0.2000=1.0000\), so each reaches the unit circle. Their dot product is \(0.8944(0.8944)+0.4472(-0.4472)=0.8000-0.2000=0.6000\). With both components included, this dot product equals the correlation between the two variables: \(0.6000\).
Check and trap avoided. The correlation matrix implied by the eigenpairs has diagonal entries 1 and off-diagonal entry 0.6, consistent with the unit arrow lengths and the calculation. Do not infer a positive correlation merely because the first coordinates point in the same direction; the second coordinates contribute negatively.
What Arrow Lengths and Angles Tell You
For a variable \(Z_j\), its squared length in the first two correlation-circle coordinates is \(c_{j1}^2+c_{j2}^2\). Each coordinate is a correlation, so the length is at most 1. A length near 1 means the variable is well represented by the first two components; a shorter arrow indicates that a larger share of its variation lies in later components. An arrow near the origin is therefore not evidence that the original variable has little variance: the variable was standardized to variance 1, and the plot is showing only its representation in the selected component plane.
For two variables \(Z_a\) and \(Z_b\), the full set of component coordinates recovers their correlation:
This identity follows from \(R=\sum_k\lambda_kv_kv_k^\mathsf{T}\), with zero-eigenvalue terms contributing zero to the matrix product. If only the first two components are plotted, the arrow dot product \(c_{a1}c_{b1}+c_{a2}c_{b2}\) is a two-component approximation to the correlation. It is exact when those components capture all relevant nonzero components, but can differ from the full correlation otherwise.
The angle between two arrows is most informative when both arrows are long. Their dot product equals the product of their lengths and the cosine of their angle. If both vectors are well represented, their angle usually gives a useful visual indication of association: a small angle suggests positive association, an angle near 90 degrees suggests little association, and an angle near 180 degrees suggests negative association. If either arrow is short, its direction can be unstable or misleading. In particular, the angle alone is not the correlation unless both arrows have unit length and the plotted plane captures the full relationship.
Worked Example: A Short Arrow and an Approximate Correlation
In a two-component correlation circle, variable A has coordinates \((0.8,0.0)\), and variable B has coordinates \((0.5,0.3)\). The full correlation between A and B, including omitted components, is known to be \(0.70\). Find the plotted lengths, the two-component dot-product approximation, and the contribution of omitted components.
Method. Compute lengths and the dot product from the displayed coordinates. The full correlation is not determined by a two-dimensional plot if other components contribute.
The length of A is \(\sqrt{0.8^2+0^2}=\sqrt{0.64}=0.8\). The length of B is \(\sqrt{0.5^2+0.3^2}=\sqrt{0.34}=0.5831\) to four decimals. The displayed dot product is \(0.8(0.5)+0(0.3)=0.40\). Thus the first two components account for 0.40 of the full correlation, and the omitted components contribute \(0.70-0.40=0.30\).
The angle’s cosine is \(0.40/(0.8\sqrt{0.34})=0.8575\) to four decimals, corresponding to an angle of about \(31.0\) degrees. The arrows point in a similar direction, but B is relatively short, so the angle should not be treated as a precise summary of the full correlation.
Check and trap avoided. The answer distinguishes the plotted dot product, 0.40, from the full correlation, 0.70. Equating them would silently assume that the omitted-component contribution is zero.
Biplots: Observation Scores and Variable Arrows
A biplot places observation points and variable vectors in the same display. There is no single universal biplot scaling: a plotting method allocates component scale between the row coordinates for observations and the column coordinates for variables. Consequently, distances between observations, arrow lengths, and observation-to-arrow projections must be interpreted according to the stated convention.
One useful convention for standardized data makes the variable arrows equal to correlation-circle coordinates and rescales observation scores to preserve the rank-two reconstruction through dot products. For the first two positive-eigenvalue components, define observation coordinate \(a_{ik}=t_{ik}/\sqrt{\lambda_k}\) and variable coordinate \(c_{jk}=\sqrt{\lambda_k}v_{jk}\). Then:
The final sum is the rank-two PCA reconstruction of standardized observation \(i\) on variable \(j\). Thus, under this explicitly defined convention, the dot product of an observation coordinate and a variable arrow approximates the standardized value. It is not generally the original value in its measurement units. If a different software display uses different biplot scalings, this dot-product interpretation and the relative lengths may change.
Worked Example: Recovering a Standardized Value From a Biplot
For one observation, the scores on two retained components are \(t_{i1}=1.5\) and \(t_{i2}=-0.5\). Their eigenvalues are \(\lambda_1=2\) and \(\lambda_2=0.5\). For variable \(j\), the corresponding eigenvector entries are \(v_{j1}=0.6\) and \(v_{j2}=0.8\). Using the biplot convention just defined, calculate the observation coordinates, the variable arrow, and the rank-two reconstructed standardized value.
Method. Divide each score by the square root of its eigenvalue for the observation coordinates, and multiply each eigenvector entry by that square root for the variable coordinates.
The observation coordinates are \(a_{i1}=1.5/\sqrt{2}=1.0607\) and \(a_{i2}=-0.5/\sqrt{0.5}=-0.7071\), rounded to four decimals. The variable arrow coordinates are \(c_{j1}=\sqrt{2}(0.6)=0.8485\) and \(c_{j2}=\sqrt{0.5}(0.8)=0.5657\).
The biplot dot product is \(1.0607(0.8485)+(-0.7071)(0.5657)=0.9000-0.4000=0.5000\), using unrounded values for the products. Directly from the scores and eigenvector entries, the PCA reconstruction is \(1.5(0.6)+(-0.5)(0.8)=0.9000-0.4000=0.5000\). The two methods agree. This value is the two-component approximation; any contribution from later components is excluded.
Check and trap avoided. The displayed observation coordinates are not the original component scores: each score has been divided by \(\sqrt{\lambda_k}\). Using raw scores with these correlation arrows would not preserve the stated dot-product reconstruction.
Zero Eigenvalues and Plotting Limits
A zero eigenvalue can occur when standardized variables are linearly dependent. The corresponding component score is constant at zero for centered data, so its variance is zero. Its correlation with any variable is undefined, and it must not be plotted as a zero-valued correlation coordinate. The component contributes no covariance to the correlation matrix, however, because its eigenvalue is zero. This distinction between an undefined correlation and a zero covariance contribution is important when interpreting singular correlation matrices.
Worked Example: A Zero-Eigenvalue Component
Suppose three standardized variables have correlation matrix \(R=\begin{pmatrix}1&1&0\\1&1&0\\0&0&1\end{pmatrix}\). The first two variables are identical after standardization. The eigenpairs are \(\lambda_1=2\), \(v_1=(1,1,0)^\mathsf{T}/\sqrt{2}\); \(\lambda_2=1\), \(v_2=(0,0,1)^\mathsf{T}\); and \(\lambda_3=0\), \(v_3=(1,-1,0)^\mathsf{T}/\sqrt{2}\). Find the defined correlation coordinates and explain the third component.
For component 1, \(\sqrt{\lambda_1}=\sqrt{2}\). Multiplying by \(v_1\) gives coordinates \((1,1,0)\). For component 2, \(\sqrt{\lambda_2}=1\), so the coordinates are \((0,0,1)\). These are the defined coordinates for the first two components.
For component 3, \(T_3=(Z_1-Z_2)/\sqrt{2}=0\) for every observation because \(Z_1=Z_2\). Hence \(\operatorname{Var}(T_3)=0\), and \(\operatorname{Corr}(Z_j,T_3)\) is undefined for all three variables. It would be incorrect to label the third-component coordinates as correlations of zero. Yet the component’s covariance contribution is zero because \(\lambda_3v_3v_3^\mathsf{T}\) is the zero matrix. For example, the correlation of variables 1 and 2 is recovered from the positive-eigenvalue terms: \(1(1)+0(0)=1\).
Check and trap avoided. The matrix’s exact duplicate-variable relationship agrees with the recovered correlation of 1. The zero-eigenvalue component contributes nothing to that reconstruction, but has no defined correlation coordinates.
Common Interpretation Errors
- Calling every zero coordinate a zero correlation. A coordinate is zero when the corresponding correlation is defined and equals zero. For a zero-variance component, correlation is undefined instead.
- Treating a two-dimensional dot product as the full correlation. It is only the contribution from the displayed components unless all omitted contributions are known to vanish.
- Using arrow angle without checking lengths. Short arrows are poorly represented in the displayed plane, so their directions may not summarize the full relationship well.
- Assuming all biplots use the same scaling. Check whether the plotted row coordinates are raw scores, rescaled scores, or another convention before interpreting distances or projections.
- Forgetting that PCA signs are arbitrary. Reversing one component’s eigenvector reverses that axis and all coordinates on it; the underlying variable relationships and reconstruction do not change.
These distinctions make correlation circles and biplots useful diagnostic summaries rather than pictures to interpret by appearance alone. State the coordinate definition, check which components are retained, and keep correlation, covariance, and reconstruction claims separate. This approach is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.
Check Your Understanding
For each problem, identify which plotted quantities are exact and which depend on the displayed components or scaling convention.
- A variable has correlation-circle coordinates \((0.6,0.8)\) on two retained positive-eigenvalue components. Find its plotted length and state what that length implies about the omitted components. (Answer: The length is \(1.0000\); the variable is fully represented in the displayed plane, so its squared correlation-coordinate contribution from omitted components is zero.)
- Two variables have coordinates \((0.7,0.2)\) and \((0.4,-0.1)\). Find their two-component dot product. If their full correlation is \(0.25\), find the omitted-component contribution. (Answer: The dot product is \(0.26\); the omitted contribution is \(-0.01\).)
- A component has eigenvalue \(0.25\), and an eigenvector entry for variable \(j\) is \(-0.8\). Find the variable’s correlation coordinate. (Answer: \(\sqrt{0.25}(-0.8)=-0.4000\).)
- A standardized observation has component scores \(t_{i1}=2\), \(t_{i2}=1\); eigenvalues \(\lambda_1=4\), \(\lambda_2=1\); and variable entries \(v_{j1}=0.5\), \(v_{j2}=-0.2\). Under the correlation-arrow biplot convention, find the observation coordinates, arrow, and reconstructed value. (Answer: Observation coordinates \((1,1)\), arrow \((1,-0.2)\), and reconstructed standardized value \(1(1)+1(-0.2)=0.8\).)
- A component has eigenvalue zero, and its score is constant for every observation. Is a correlation coordinate for a variable on this component zero or undefined, and what is its covariance contribution to the correlation matrix? (Answer: The correlation coordinate is undefined; the covariance contribution is zero.)