Turning PCA Output Into an Exam Answer
A principal components analysis (PCA) calculation is only part of a complete answer. You must also identify which matrix was analyzed, interpret the resulting directions, calculate the variance represented by the retained components, and justify whether the variables should be standardized. These decisions matter because PCA finds directions of greatest variance, and the meaning of variance depends on the scale of the variables.
The calculations below use a centered data matrix \(X\), whose rows are observations and whose columns are variables. For covariance PCA, the matrix analyzed is the sample covariance matrix \(S\). For correlation PCA, each variable is centered and divided by its sample standard deviation before calculating the covariance matrix; equivalently, PCA is performed on the sample correlation matrix \(R\). In either case, the principal component directions are unit eigenvectors of the matrix analyzed.
An eigenvector gives the coefficients of a component, not the component's value for a particular observation. To obtain that value, project the centered (or standardized) observation onto the eigenvector. The sign of an eigenvector and its component scores can be reversed together without changing the PCA result.
Why the Eigenvector Gives the Maximum-Variance Direction
The defining optimization helps connect the calculation to its interpretation. Suppose \(S\) is a sample covariance matrix and \(a\) is any unit vector. Projecting the centered observations onto \(a\) produces a variable with sample variance \(a^\mathsf{T}Sa\). PCA seeks the unit direction that maximizes this variance.
Because \(S\) is symmetric, it has an orthonormal eigenvector basis. Write \(a=\sum_j c_jq_j\), where \(q_j\) is an eigenvector with eigenvalue \(\lambda_j\). The unit-length condition gives \(\sum_j c_j^2=1\). Substituting the expansion into the variance gives
Equality is attained when \(a=q_1\), an eigenvector associated with the largest eigenvalue. Thus the first component has variance \(\lambda_1\). Repeating the maximization among directions orthogonal to \(q_1\) gives the second component, and so on. The eigenvalues therefore give component variances, which is why their sum is the total variance represented by the analyzed matrix.
Worked Example: Covariance PCA, Scores, and Variance Explained
A hypothetical portfolio study uses two centered measurements: annual change in a loss ratio, in percentage points, and annual change in an expense index, in index points. Suppose their sample covariance matrix is
Method. Find the eigenpairs of \(S\), project the specified observation onto the component directions, and divide eigenvalues by total variance. The given matrix is a covariance matrix, so do not standardize the variables during this calculation.
Step 1: Find the eigenvalues. Solve \(\det(S-\lambda I)=0\):
Thus \(\lambda_1=7\) and \(\lambda_2=1\). For \(\lambda_1=7\), the equation \((S-7I)q_1=0\) gives equal coordinates, so a unit eigenvector is \(q_1=(1,1)^\mathsf{T}/\sqrt{2}\). For \(\lambda_2=1\), the coordinates have opposite signs, giving \(q_2=(1,-1)^\mathsf{T}/\sqrt{2}\).
Step 2: Calculate scores. Consider a centered observation \(x=(2,0)^\mathsf{T}\), meaning its loss-ratio change is 2 percentage points above the sample mean and its expense-index change is at its sample mean. Its first score is \(q_1^\mathsf{T}x=(2+0)/\sqrt{2}=\sqrt{2}\approx1.414\). Its second score is \(q_2^\mathsf{T}x=(2-0)/\sqrt{2}=\sqrt{2}\approx1.414\).
Step 3: Interpret and quantify. The first component gives equal weight to the two centered measurements and describes their joint movement. The second contrasts them. Total variance is \(7+1=8\); the first component explains \(7/8=0.875\), or 87.5%, and the second explains \(1/8=0.125\), or 12.5%.
Check. The two unit directions are perpendicular, and the explained-variance proportions sum to 100%. This avoids the common mistake of treating a component coefficient as an observation's score: the score requires the projection calculation.
Choosing Covariance or Correlation PCA
Covariance PCA retains the original measurement units. A variable with a numerically large variance can therefore dominate the first component, even if that dominance reflects the units used rather than a substantive reason to give the variable greater weight. Correlation PCA standardizes each variable to have sample variance one before analysis. It is often suitable when variables use different units or when equal scale treatment is intended. It is not automatically preferable: if the original units and their variance differences are meaningful, removing those differences may discard useful information.
The scale decision should be justified from the problem context. State whether you use covariance or correlation PCA, identify what the resulting variance measures, and explain why that scale is appropriate. Standardization changes the question PCA answers; it is not merely a formatting step.
Worked Example: How Standardization Changes the Result
A hypothetical risk study measures annual claim-count change and annual premium-volume change. Their variances are 100 and 4, respectively, and their covariance is zero. Compare covariance PCA with correlation PCA, then explain when each analysis might be defensible.
Method. Use the given covariance matrix for the first analysis, then standardize each variable to form the correlation matrix. The covariance matrix is diagonal:
Step 1: Covariance PCA. The eigenvalues are 100 and 4, with coordinate directions \((1,0)^\mathsf{T}\) and \((0,1)^\mathsf{T}\). Total variance is \(100+4=104\). The first component explains \(100/104=25/26\approx0.9615\), or 96.15%, and is the claim-count variable alone. The second explains \(4/104=1/26\approx0.0385\), or 3.85%.
Step 2: Correlation PCA. The standard deviations are \(\sqrt{100}=10\) and \(\sqrt{4}=2\). Because the covariance is zero, the correlation is \(0/(10\cdot2)=0\), and
Both eigenvalues are 1, so each component direction has an explained-variance proportion of \(1/2=50\%\). There is no unique first direction: every unit direction is an eigenvector of this identity matrix, and all have the same variance.
Step 3: Decide what the analysis means. Covariance PCA is defensible if the original variances, in the stated units, are intended to determine importance. Correlation PCA is defensible if the variables' differing scales are arbitrary or if equal unit-variance treatment is intended. It does not reveal a uniquely dominant direction here. The calculation avoids the trap of assuming standardization always produces one clear leading component.
Check. For covariance PCA, \(25/26+1/26=1\); for correlation PCA, the equal eigenvalues give \(1/2+1/2=1\). The matrices answer different scale-weighting questions.
Interpreting Components and Choosing a Retained Set
An eigenvector's entries describe the direction of a component in the variables analyzed. For correlation PCA, equal entries with the same sign indicate a direction in which standardized variables move together; opposite signs indicate a contrast. For covariance PCA, the coefficients still define the direction, but variable units and scales remain part of the analysis. Interpret the direction in context rather than treating the component as a new original variable.
To choose a number of components using an explained-variance target, add the ordered eigenvalues until their cumulative proportion reaches the target. The target is a decision rule supplied by the analysis, not a universal PCA law. A component count chosen this way should be distinguished from a claim that the retained components preserve every feature of the data.
Worked Example: A Three-Variable Correlation PCA
Suppose standardized measures of three hypothetical portfolio characteristics have correlation matrix
A risk analyst wants at least 80% cumulative explained variance. Find the leading component, decide how many components meet the target, and score a particular standardized observation \(z=(2,0,-1)^\mathsf{T}\).
Step 1: Eigenvalues and the leading direction. Each row sums to 2, so \((1,1,1)^\mathsf{T}\) is an eigenvector with eigenvalue 2. Any vector whose entries sum to zero is multiplied by 0.5 by \(R\), so the other two eigenvalues are 0.5 and 0.5. A unit eigenvector for the leading component is \(q_1=(1,1,1)^\mathsf{T}/\sqrt{3}\). It describes a common direction: the three standardized measures receive equal, same-sign coefficients.
Step 2: Apply the 80% rule. The total variance is the trace \(1+1+1=3\), also equal to the eigenvalue sum \(2+0.5+0.5=3\). The first component explains \(2/3\approx0.6667\), or 66.67%, which is below 80%. The first two explain \((2+0.5)/3=2.5/3=5/6\approx0.8333\), or 83.33%. Therefore two components are required.
Step 3: Score the observation. The first score is \(q_1^\mathsf{T}z=(2+0-1)/\sqrt{3}=1/\sqrt{3}\approx0.577\). One valid second direction is \(q_2=(1,0,-1)^\mathsf{T}/\sqrt{2}\): its entries sum to zero, it is perpendicular to \(q_1\), and it has eigenvalue 0.5. The second score is \(q_2^\mathsf{T}z=(2-(-1))/\sqrt{2}=3/\sqrt{2}\approx2.121\).
Interpretation and check. This observation has a modest positive score on the common-movement direction and a larger positive score on a direction contrasting the first and third measures. The repeated eigenvalue 0.5 means that the second and third directions are not individually unique; any orthonormal basis for their two-dimensional subspace is valid. The eigenvalue ordering, cumulative proportion, and score calculations are consistent. This avoids claiming a unique interpretation for a direction within a tied-eigenvalue subspace.
Common Calculation Traps
- Using the wrong matrix. Covariance PCA and correlation PCA need not give the same directions or explained-variance proportions. Name the matrix and scale before interpreting the output.
- Using the wrong denominator. The total variance is the sum of all eigenvalues of the matrix analyzed, not just the eigenvalues retained. For correlation PCA with \(p\) variables, the total is \(p\).
- Confusing coefficients with scores. An eigenvector describes a direction. A score requires multiplying that direction by the centered observation, or by its standardized version for correlation PCA.
- Over-interpreting signs or tied directions. Reversing all signs in an eigenvector and its scores changes no PCA result. If eigenvalues are tied, a particular direction within the tied subspace may not be unique.
A complete exam-style response should therefore name the analyzed matrix, show enough eigenvalue and eigenvector work to support the result, calculate scores by projection when requested, and interpret both the direction and the variance proportion. For a scale choice, explain what the units imply and why standardization is or is not appropriate. This problem-solving approach is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.
Check Your Understanding
For each problem, identify the scale or calculation being used and check that the explained-variance proportions use the full total variance.
- A covariance matrix is \(\begin{pmatrix}9&0\\0&1\end{pmatrix}\). Find the eigenvalues, the first component direction, and its explained-variance proportion. (Answer: eigenvalues 9 and 1; first direction \((1,0)^\mathsf{T}\); 90%)
- A correlation matrix is \(\begin{pmatrix}1&0.4\\0.4&1\end{pmatrix}\). Find the eigenvalues and the proportion explained by the first component. (Answer: eigenvalues 1.4 and 0.6; first direction \((1,1)^\mathsf{T}/\sqrt{2}\); 70%)
- A PCA has ordered eigenvalues 3, 1, 0.5, and 0.5. How many components are needed to explain at least 75% of total variance? (Answer: 2 components; total variance is 5, and the first two explain 4/5 = 80%)
- Two variables have variances 25 and 1, with covariance 0. State the correlation matrix and explain why correlation PCA does not produce a unique leading direction. (Answer: the correlation matrix is the identity matrix; both eigenvalues are 1, so every unit direction is an eigenvector and explains 50%)
- A centered observation is \(x=(2,0)^\mathsf{T}\), and its first component direction is \(q_1=(1,1)^\mathsf{T}/\sqrt{2}\). Calculate its score. If its covariance matrix is \(\begin{pmatrix}5&1\\1&5\end{pmatrix}\), also give the first component's variance proportion. (Answer: score \(2/\sqrt{2}=\sqrt{2}\approx1.414\); eigenvalues are 6 and 4, so the first component explains 60%)