PCA As Orthogonal Projection

Introduction

PCA replaces a collection of measured variables with directions through the data that summarize variation. Once those directions have been found, each observation can be described by its coordinate along each direction. That coordinate is its principal component score.

Geometrically, a score is a projection: it measures how far an observation lies along a component direction. The observation must first be expressed relative to the data center used to fit PCA. For correlation-based PCA, it must also be expressed in the standardized units used to fit the components.

Projection onto a Component Direction

A direction is represented by a unit vector \(v\), so \(v^\mathsf{T}v=1\). The scalar projection of a centered observation \(z\) onto that direction is \(z^\mathsf{T}v\). The corresponding vector projection is \((z^\mathsf{T}v)v\). The scalar is the PCA score; the vector points from the origin to the projected location.

PCA directions obtained from a covariance matrix or correlation matrix are eigenvectors. For a symmetric matrix, its eigenvectors can be chosen orthonormal. Thus, for two distinct component directions \(v_1\) and \(v_2\), \(v_1^\mathsf{T}v_2=0\), and each direction has unit length. These conditions are what make the component scores coordinates along perpendicular axes.

Why the Dot Product Gives the Projection

The score formula follows from the definition of an orthogonal projection. Among all points on the line through the origin in direction \(v\), write a candidate point as \(cv\), where \(c\) is a scalar. The squared distance from \(z\) to this candidate is

where \(v^\mathsf{T}v=1\) has been used. This quadratic is minimized when its derivative with respect to \(c\) is zero:

Therefore the closest point on the component line is \((z^\mathsf{T}v)v\), and its signed coordinate is \(z^\mathsf{T}v\). The perpendicularity condition can also be checked directly:

The residual from the projection is perpendicular to the direction, as required. If a direction vector is not unit length, the coefficient is instead \(z^\mathsf{T}v/(v^\mathsf{T}v)\). PCA eigenvectors are conventionally normalized to unit length, so the simpler score formula applies.

Worked Two-Dimensional Projections

In two dimensions, write a centered observation as \(z=(z_1,z_2)^\mathsf{T}\) and a component direction as \(v=(v_1,v_2)^\mathsf{T}\). The score is \(z_1v_1+z_2v_2\). Before taking that dot product, identify whether the fitted PCA used centered measurements or standardized measurements.

Scores for Several Observations

The same projection calculation applies row by row. If the centered data matrix is \(Z\) and the matrix \(V\) has the component directions as its columns, the score matrix is \(T=ZV\). Its entry in row \(i\), column \(j\) is the dot product of centered observation \(i\) with direction \(j\). For standardized PCA, \(Z\) here denotes the matrix after applying the training means and standard deviations.

This matrix expression is a compact way to perform separate projections, not a different definition. Each row is multiplied by the direction vectors, so the output records each observation's coordinates along those directions.

Checks, Conditions, and Exam Traps

Projection scores are straightforward to compute, but a few conditions matter. The direction vector must be the one fitted by the PCA being applied, and the observation must be centered or standardized consistently with that fit. If the vector supplied is not unit length, the ordinary dot product is not the projection coefficient; divide by its squared length. If only some component directions are used, the resulting scores give coordinates along those directions only.

When all \(p\) orthonormal directions are included, the squared length of the centered or standardized observation equals the sum of its squared scores:

This follows by expressing \(z\) in an orthonormal basis: its coordinate in direction \(v_j\) is \(t_j=z^\mathsf{T}v_j\). The identity is a useful arithmetic check, but it requires the complete basis. It need not hold if only a subset of component directions is included.

For an observation from outside the training data, do not recalculate the training means or standard deviations using that observation. Apply the fitted preprocessing and directions. This keeps the new score on the same coordinate system as the scores of the observations used to fit the PCA.

These calculations are aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus. A reliable solution states the preprocessing, writes the centered or standardized vector, and then shows the dot products. In two dimensions, checking that the directions are unit and perpendicular and that the projected coordinates are consistent with the input can catch arithmetic and scaling errors.