Why Unusual Observations Need Two Diagnostics
PCA summarizes variation by projecting observations onto a set of component directions. An observation can be unusual in two different ways: it can lie far from the center along the directions retained in the analysis, or it can lie far from the lower-dimensional space spanned by those directions. These are distinct diagnostics. Looking only at a plot of the first few component scores may miss observations that fit the retained components poorly.
The distinction matters in actuarial work because an unusual record may reflect a data error, a genuinely different risk, or a process that the current PCA representation does not capture. A distance is a screening signal, not a reason by itself to delete a record. In particular, an observation can both be extreme and strongly affect the fitted principal directions.
Definitions and Computation
Suppose PCA is fitted to \(p\) variables after the chosen centering and, if appropriate, scaling. Let \(\mathbf{x}_i\) be observation \(i\) on that analysis scale, and let \(\bar{\mathbf{x}}\) be the vector of variable means on the same scale. The component loading vectors \(\mathbf{v}_1,\ldots,\mathbf{v}_p\) are orthonormal eigenvectors, ordered by decreasing eigenvalue \(\lambda_1\ge \cdots\ge\lambda_p\). The score of observation \(i\) on component \(j\) is \(t_{ij}=\mathbf{v}_j^\mathsf{T}(\mathbf{x}_i-\bar{\mathbf{x}})\). Retain the first \(k\) components, where \(1\le k<p\), and assume their eigenvalues are positive.
The score distance uses the retained scores, scaled by their component variances. The residual vector is the part of the centered observation left after projecting onto the retained component space. Its length is the residual distance, also called orthogonal distance.
The final equality for \(OD_i\) follows because the full set of PCA loading vectors is an orthonormal basis: the omitted component scores are the coordinates of the residual in the omitted directions. These expressions use eigenvalues from the covariance matrix on the analysis scale. If PCA was fitted to standardized variables, the scores, residuals, and eigenvalues must all refer to that standardized scale.
Worked Examples
Worked Example: Separating an Extreme Score From a Poor Fit
A two-variable PCA has eigenvalues \(\lambda_1=9\) and \(\lambda_2=1\). Only the first component is retained. Two observations have component scores \((6,0.5)\) and \((9,2)\), respectively. For this screening exercise, flag a record if \(SD_i>2.5\) or \(OD_i>1.5\). Calculate both distances and classify each record.
Method. With \(k=1\), the score distance uses only the first score and \(\lambda_1=9\). The residual distance is the absolute value of the second score, because that score is omitted.
For the first observation, the retained score is \(t_{i1}=6\), so \(SD_i=\sqrt{6^2/9}=\sqrt{36/9}=\sqrt{4}=2\). Its omitted score is \(0.5\), so \(OD_i=\sqrt{0.5^2}=\sqrt{0.25}=0.5\). Since \(2\) is not greater than \(2.5\), and \(0.5\) is not greater than \(1.5\), this observation is not flagged by the stated rule.
For the second observation, \(t_{i1}=9\), so \(SD_i=\sqrt{9^2/9}=\sqrt{81/9}=\sqrt{9}=3\). Its omitted score is \(2\), so \(OD_i=\sqrt{2^2}=2\). Since \(3>2.5\) and \(2>1.5\), it is flagged on both diagnostics.
Check. The first record is not far from the retained component center and has a small omitted score. The second is far along the retained direction and also far from the retained one-dimensional space. The mistake this method avoids is treating a large first-component score as the only possible form of unusualness.
Worked Example: Finding an Observation Poorly Represented by Retained Components
A three-variable PCA has eigenvalues \((4,1,0.25)\), and the first two components are retained. Three observations have scores \(A=(2,1,0.2)\), \(B=(0.5,0.2,1.2)\), and \(C=(5,0,0.1)\). A diagnostic rule flags score distance above \(2\) and residual distance above \(0.8\). Identify which records are flagged and why.
Method. For each record, use the first two scores and eigenvalues \(4\) and \(1\) in the score-distance formula. Since only the first two components are retained, the third score alone determines residual distance.
For \(A\), \(SD_A=\sqrt{2^2/4+1^2/1}=\sqrt{1+1}=\sqrt{2}\approx1.4142\), and \(OD_A=\sqrt{0.2^2}=0.2\). Neither value exceeds its stated threshold, so \(A\) is not flagged.
For \(B\), \(SD_B=\sqrt{0.5^2/4+0.2^2/1}=\sqrt{0.25/4+0.04}=\sqrt{0.0625+0.04}=\sqrt{0.1025}\approx0.3202\). Its residual distance is \(OD_B=\sqrt{1.2^2}=1.2\). Thus \(B\) is flagged for residual distance, not for score distance.
For \(C\), \(SD_C=\sqrt{5^2/4+0^2/1}=\sqrt{25/4}=\sqrt{6.25}=2.5\), and \(OD_C=\sqrt{0.1^2}=0.1\). Thus \(C\) is flagged for score distance, not residual distance.
Check. \(A\) is not flagged; \(B\) is poorly represented by the retained space; and \(C\) is unusually far out within that space. The calculation avoids the mistake of calling \(B\) typical simply because its retained scores are small.
Worked Example: Seeing an Extreme Value Rotate a PCA Direction
Consider the hypothetical two-variable data set with observations \((-1,0)\), \((0,0)\), and \((1,0)\). Then add a fourth observation \((10,10)\). Using the sample covariance matrix in each case, compare the leading principal direction before and after adding the fourth observation. This example illustrates sensitivity; it is not a rule for deciding whether a real record should be removed.
Before adding the fourth observation. The mean is \((0,0)\). The centered sums of squares and cross-products are \(\sum x_i^2=1+0+1=2\), \(\sum y_i^2=0\), and \(\sum x_i y_i=0\). Dividing by \(n-1=2\), the sample covariance matrix is \(\begin{pmatrix}1&0\\0&0\end{pmatrix}\). Its leading eigenvector is \((1,0)\), so the first principal direction is the first-variable axis.
After adding the fourth observation. The new mean is \((2.5,2.5)\). The four centered observations are \((-3.5,-2.5)\), \((-2.5,-2.5)\), \((-1.5,-2.5)\), and \((7.5,7.5)\). The centered sums are \(\sum x_i^2=12.25+6.25+2.25+56.25=77\), \(\sum y_i^2=6.25+6.25+6.25+56.25=75\), and \(\sum x_i y_i=8.75+6.25+3.75+56.25=75\). Dividing each by \(n-1=3\) gives the covariance matrix \(\begin{pmatrix}25.6667&25\\25&25\end{pmatrix}\), with entries rounded here to four decimal places.
For a symmetric two-by-two matrix with entries \(a,b,b,d\), the leading eigenvector angle \(\theta\) from the first-variable axis satisfies \(2\theta=\operatorname{atan2}(2b,a-d)\). Here \(2b=50\) and \(a-d=25.6667-25=0.6667\), giving \(\theta\approx44.6^\circ\). The leading direction has shifted from the first-variable axis to a direction close to the diagonal.
Check. The change is not just a large score assigned to the added observation: the fitted mean and covariance, and therefore the PCA directions themselves, have changed. The arithmetic avoids the common mistake of calculating a new score using old directions and concluding that the PCA fit was unchanged.
Why PCA Is Sensitive to Extreme Values
PCA directions are chosen to explain variation in the data, and variation is measured through squared deviations from the mean. A point far from the center can therefore contribute large squared deviations to the covariance matrix. It can also move the mean, changing the deviations of the other observations. Because the eigenvectors are calculated from that covariance matrix, an extreme observation may rotate the directions or change which directions account for the most variance.
The preceding calculation shows both effects: the new observation moved the mean from \((0,0)\) to \((2.5,2.5)\) and changed the leading direction. This sensitivity is a property of ordinary PCA, not evidence that the observation is invalid. A rare but genuine claim pattern or unusual operating condition may be precisely the part of the population an analyst needs to understand.
Reading Score and Residual Displays
A score plot places observations using selected component scores, such as PC1 against PC2. It helps show variation within those displayed directions, but does not directly show how far observations lie outside the displayed component space. A score-distance-versus-residual-distance display makes the two diagnostics explicit. If the display uses squared distances rather than distances, its axis values and thresholds must be interpreted as squared quantities.
| Score distance | Residual distance | Diagnostic interpretation |
|---|---|---|
| Not large | Not large | Close to the center within the retained space and represented reasonably well by that space. |
| Large | Not large | Unusual along retained component directions, but still represented by those directions. |
| Not large | Large | Not extreme within retained scores, but poorly represented by the retained components. |
| Large | Large | Unusual within the retained space and also poorly represented by it. |
A threshold is a screening convention, not a universal constant. Its suitability depends on the data, the number of retained components, the scaling used, and the purpose of the analysis. Under a multivariate normal model, score distances can be compared with a chi-square reference under suitable population and estimation assumptions; in applied PCA, estimated directions, finite samples, and non-normal data make automatic use of such a reference hazardous. Residual-distance cutoffs also depend on the residual distribution. Use an appropriate reference or empirical diagnostic strategy, and report how the cutoff was chosen.
A flagged record should prompt investigation: check units, coding, data-entry errors, unusual combinations of variables, and whether the record belongs to the population being analyzed. Do not delete a record merely to make the PCA plot look cleaner. If the record is valid, retaining it may be important; if it is an error, document the correction. If extreme-value sensitivity is a serious concern, a robust PCA method may be considered, but it answers a different modeling question and should not be presented as ordinary PCA.
Common Exam Traps
- Using only score plots. A point can have ordinary retained scores and still have a large residual distance. Inspect both forms of unusualness.
- Comparing unscaled scores across components. A score of 3 on a high-variance component is not equivalent to a score of 3 on a low-variance component. Score distance divides each retained squared score by its eigenvalue.
- Confusing unusualness with influence. A large distance identifies an unusual observation relative to a fitted representation. To assess whether a record changes the PCA fit, compare the fit with and without it, taking care that centering, scaling, and the fitted components are recalculated consistently.
- Assuming a flag proves a data error. A flagged record may be correct and substantively important. Investigate it before deciding how to handle it.
- Forgetting the analysis scale. If variables were standardized before PCA, calculate scores and residuals on that same standardized scale. Mixing raw-scale observations with standardized-scale loadings gives invalid distances.
Check Your Understanding
For each question, distinguish the two distances and pay attention to which components are retained.
- A two-variable PCA has \(\lambda_1=4\), \(\lambda_2=1\), and retains only PC1. An observation has scores \((4,0.6)\). Calculate its score and residual distances. (Answer: \(SD=2\), \(OD=0.6\).)
- A three-variable PCA has eigenvalues \((9,4,1)\) and retains two components. An observation has scores \((3,4,2)\). Calculate both distances and state which part of the representation is larger. (Answer: \(SD=\sqrt{3^2/9+4^2/4}=\sqrt{5}\approx2.2361\), \(OD=2\); the score distance is larger numerically.)
- A PCA retains two components with eigenvalues \(4\) and \(1\). A record has retained scores \((0,2)\) and omitted-score magnitude \(0.3\). A rule flags \(SD>1.5\) or \(OD>0.5\). Is it flagged, and why? (Answer: Yes; \(SD=\sqrt{0/4+4/1}=2\), while \(OD=0.3\), so it is flagged for score distance only.)
- Explain why a valid extreme observation can change the PCA directions even if the analyst later focuses on the scores of the other records. (Answer: PCA recomputes the mean and covariance from all observations; an extreme value can change both through its large squared deviations, changing the covariance eigenvectors and hence the directions used for every record.)
- A record is not far from the center in retained score space but has a large residual distance. What does this imply, and does it establish that the record is erroneous? (Answer: It is poorly represented by the retained component space, but this alone does not establish an error; the record should be investigated.)