What Principal Components Optimize

Why PCA Is an Optimization Problem

A data set may contain several measured variables that vary together. Principal components analysis (PCA) replaces those variables with new directions through the multivariable data space. The first direction is chosen to capture as much variation as possible; the next captures as much as possible subject to being orthogonal to the first. This gives a compact description of variation without requiring each original variable to be considered separately.

PCA is a descriptive method. It does not use an outcome variable, establish a causal relationship, or guarantee good prediction of a future response. Its central question is narrower: among directions of a specified length, which direction has the greatest variance when the observations are projected onto it?

For the optimization below, suppose there are \(p\) variables and \(n\) observed feature vectors. Assume the variables have been centered, so their sample means are zero, and let \(S\) denote their sample covariance matrix. The centering procedure is not part of this discussion. No normal distribution assumption is required for the variance maximization.

Definition: A direction is represented by a vector \(a\) with \(p\) entries. The projected value for observation \(i\) is \(a^\top x_i\), where \(x_i\) is that observation's centered feature vector. A unit direction satisfies \(a^\top a=1\). Its projected sample variance is \(a^\top S a\).

The unit-length condition matters. Without it, multiplying a direction by a constant would multiply every projected value by that constant, and its variance by the square of that constant. The apparent maximum could then be made arbitrarily large without changing the direction itself.

What the First Component Maximizes

The first principal direction solves the constrained optimization problem

\max_{a:\,a^\top a=1} a^\top S a.

Here \(a^\top S a\) is the variance of the projected observations. The constraint limits the search to unit directions, so the criterion compares directions rather than their arbitrary scales. Since \(S\) is a symmetric covariance matrix, this maximum is attained in a direction associated with its largest eigenvalue. The maximum value is that eigenvalue.

The result follows directly from a constrained optimization argument. Use a Lagrange multiplier \(\lambda\) for the condition \(a^\top a=1\), and define

L(a,\lambda)=a^\top S a-\lambda(a^\top a-1).

Differentiating with respect to \(a\) and setting the derivative equal to zero gives \(2Sa-2\lambda a=0\), or \(Sa=\lambda a\). Thus a maximizing direction must be an eigenvector of \(S\). For a unit eigenvector, its projected variance is

a^\top S a=a^\top(\lambda a)=\lambda a^\top a=\lambda.

Therefore, among the eigenvector directions, the one with the largest eigenvalue gives the greatest projected variance. This argument explains the optimization criterion; it does not require a particular numerical algorithm for finding the directions.

Worked Example: Finding the Direction of Greatest Variation

A hypothetical insurer summarizes two centered experience measures: variation in claim frequency and variation in claim severity. Suppose their sample covariance matrix, in the stated squared units, is

S= \begin{pmatrix} 5&2\\ 2&2 \end{pmatrix}.

Find the unit direction that maximizes projected variance and calculate that variance. Also compare it with projecting onto the first variable alone.

Method. Find the eigenvalues to identify the largest possible variance, then find and normalize its direction. The common mistake to avoid is reporting the variance for an unnormalized direction.

Step 1: Find the eigenvalues. Solve \(\det(S-\lambda I)=0\):

\begin{aligned} (5-\lambda)(2-\lambda)-4&=0,\\ \lambda^2-7\lambda+6&=0,\\ (\lambda-6)(\lambda-1)&=0. \end{aligned}

The largest eigenvalue is \(6\), so the maximum projected variance is \(6\).

Step 2: Find its direction. For \(\lambda=6\), the first row of \((S-6I)a=0\) gives \(-a_1+2a_2=0\), so \(a_1=2a_2\). A direction is \((2,1)^\top\), whose length is \(\sqrt{2^2+1^2}=\sqrt{5}\). A unit direction is therefore \(a=(2,1)^\top/\sqrt{5}\).

Step 3: Verify the variance. Multiplication gives \(Sa=(12,6)^\top/\sqrt{5}=6a\). Hence \(a^\top Sa=6a^\top a=6\), as required. By comparison, projecting onto the first variable alone uses \(a=(1,0)^\top\), giving \(a^\top Sa=5\). The optimized direction captures more sample variance than that single-variable direction.

Check. The unit direction has squared length \((4+1)/5=1\), and the computed variance agrees with the largest eigenvalue.

How Successive Directions Are Chosen

PCA does not select every direction independently. If every direction were allowed to equal the first, each would repeat the same variation. Instead, the second direction must be perpendicular to the first, and each later direction must be perpendicular to all directions already selected. For unit vectors \(a_j\), perpendicularity means \(a_j^\top a_k=0\) whenever \(j\ne k\).

Definition: The \(k\)th principal direction maximizes \(a^\top S a\) over unit vectors \(a\) that are orthogonal to the first \(k-1\) principal directions. The resulting directions are mutually orthogonal. If the maximum is tied, more than one direction can achieve it, and an orthonormal choice can be made within the tied directions.

For the second direction, the optimization is the same variance maximization as for the first, but with the additional constraint \(a^\top a_1=0\). The restriction removes the first direction from consideration. For a symmetric covariance matrix, the optimizing directions can be chosen as mutually orthogonal eigenvectors, ordered by decreasing eigenvalue. If eigenvalues are equal, the choice within the corresponding tied directions is not unique, but the maximum variance is unchanged.

The total sample variance across the original variables is the sum of their individual sample variances, which is the trace of \(S\), the sum of its diagonal entries. In an orthonormal coordinate system made from principal directions, that same total is the sum of the variances along those directions. Consequently, the fraction of total variance captured by the first \(k\) directions is the sum of their projected variances divided by the total variance. This is a descriptive summary, not a measure of predictive accuracy.

Worked Example: Maximizing Variation After the First Direction

A hypothetical portfolio monitoring system records three centered risk indicators. Its sample covariance matrix is

S= \begin{pmatrix} 9&0&0\\ 0&4&0\\ 0&0&1 \end{pmatrix}.

Find the first two principal directions and their projected variances.

Method. Maximize the variance over unit vectors for the first direction, then repeat under the constraint of orthogonality to the first. The trap to avoid is selecting the first direction again when looking for the second.

Step 1: First direction. For a unit vector \(a=(a_1,a_2,a_3)^\top\), the projected variance is \(9a_1^2+4a_2^2+a_3^2\), with \(a_1^2+a_2^2+a_3^2=1\). Since each coefficient is at most \(9\), this variance is at most \(9(a_1^2+a_2^2+a_3^2)=9\). Equality is attained at \(a_1=(1,0,0)^\top\). The first direction has variance \(9\).

Step 2: Second direction. Orthogonality to \(a_1\) requires \(a_1^\top a=0\), which here means the first coordinate of the candidate vector is zero. The variance becomes \(4a_2^2+a_3^2\), with \(a_2^2+a_3^2=1\). It is at most \(4(a_2^2+a_3^2)=4\), and equality is attained at \(a_2=(0,1,0)^\top\). The second direction has variance \(4\).

Step 3: Check the restriction. The dot product \(a_1^\top a_2=0\), so the directions are orthogonal, and each has length one. Their variances are in decreasing order, \(9\) then \(4\).

Comparing a Principal Direction With a Candidate

A direction that is meaningful for interpretation is not necessarily the direction that captures the most variation. The optimization criterion provides a precise comparison: calculate \(a^\top S a\) for each unit candidate and compare it with the maximum. For a one-component summary, the captured proportion is the component variance divided by the total variance.

Worked Example: Comparing Captured Variance and Total Variation

Suppose two centered measurements from a hypothetical group of policy records have sample covariance matrix

S= \begin{pmatrix} 10&3\\ 3&2 \end{pmatrix}.

A reviewer proposes using the first variable alone, direction \(b=(1,0)^\top\). Find the optimal first direction, compare its variance with that of \(b\), and calculate the proportion of total variance captured by the optimal direction.

Method. Use the largest eigenvalue for the maximum, evaluate the proposed unit direction directly, and divide the maximum by the total variance. The common mistake to avoid is dividing by only one diagonal entry instead of by total variance.

Step 1: Find the maximum. The characteristic equation is

\det(S-\lambda I)=(10-\lambda)(2-\lambda)-9 =\lambda^2-12\lambda+11 =(\lambda-11)(\lambda-1)=0.

The largest eigenvalue is \(11\). For \(\lambda=11\), the first row of \((S-11I)a=0\) gives \(-a_1+3a_2=0\), so \(a_1=3a_2\). Normalizing \((3,1)^\top\), whose length is \(\sqrt{10}\), gives the unit direction \(a=(3,1)^\top/\sqrt{10}\). Its projected variance is \(11\).

Step 2: Evaluate the proposed direction. Since \(b\) is unit length, \(b^\top S b=10\). The optimal direction therefore captures \(11-10=1\) more squared unit of sample variance than the proposed direction.

Step 3: Calculate the captured proportion. The total variance is the trace, \(10+2=12\). Thus the optimal first direction captures \(11/12=0.9167\), or \(91.67\%\) of total variance when expressed as a percentage.

Check. The two eigenvalues sum to \(11+1=12\), matching the total variance from the diagonal entries. The proposed direction's variance, \(10\), is below the maximum, \(11\).

Common Mistakes and Interpretation

PCA is sensitive to the variables supplied and to how their scales are represented. A covariance-based analysis uses the covariance matrix as given; a variable measured on a much larger numerical scale can contribute strongly to the variance criterion. Choosing to rescale variables changes the optimization problem, so it should be a deliberate modeling decision rather than an unnoticed change of units.

Using a non-unit direction. If \(c\) is a unit direction, then \((kc)^\top S(kc)=k^2c^\top Sc\). The resulting number is not comparable as a directional variance unless the vector is normalized.
Ignoring the orthogonality restriction. The second component is not simply another unrestricted search for the largest variance. It must be perpendicular to the first direction.
Calling high variance predictive value. PCA maximizes variation in the input measurements, not association with a response. A direction with high variance may have little value for predicting an outcome.
Assuming a unique direction in a tie. If multiple directions attain the same maximum, the maximizing direction need not be unique, even though the maximum variance is well defined.

The optimization view is useful because it gives PCA a clear purpose and a checkable criterion. Each component is a unit direction; its value is the variance of the data projected onto it; and successive directions are restricted to be orthogonal to those already chosen. This interpretation is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.

Check Your Understanding

For each hypothetical covariance matrix, identify the requested direction or variance. Treat all stated directions as unit vectors unless the question asks you to normalize one.

A centered two-variable data set has covariance matrix \(S=\begin{pmatrix}7&0\\0&3\end{pmatrix}\). Find the first principal direction and its projected variance. (Answer: direction \((1,0)^\top\), up to sign; variance \(7\).)
A hypothetical pair of centered claim indicators has \(S=\begin{pmatrix}4&1\\1&4\end{pmatrix}\). Find a unit first principal direction and its projected variance. (Answer: direction \((1,1)^\top/\sqrt{2}\), up to sign; variance \(5\).)
For \(S=\operatorname{diag}(8,5,2)\), the first direction is \((1,0,0)^\top\). Find the second direction under the PCA orthogonality restriction and its variance. (Answer: direction \((0,1,0)^\top\), up to sign; variance \(5\).)
A covariance matrix is \(S=\begin{pmatrix}6&2\\2&3\end{pmatrix}\). Find the maximum projected variance and a corresponding unit direction. (Answer: maximum variance \(7\); direction \((2,1)^\top/\sqrt{5}\), up to sign.)
A centered two-variable data set has \(S=\begin{pmatrix}3&1\\1&3\end{pmatrix}\). Find the maximum variance and the proportion of total variance it captures. (Answer: maximum variance \(4\); total variance \(6\); proportion \(4/6=2/3\), or \(66.67\%\).)

What You'll Learn