From a direction to projected variance
Principal components provide new directions in which to describe variation in a set of numeric variables. For the first principal component, the central question is precise: which unit-length coefficient vector makes the projected observations as variable as possible? The unit-length condition matters because otherwise the same direction could be multiplied by an arbitrarily large number, changing the variance just by changing the scale of the coefficients.
Assume the data have already been centered, as in the earlier material on centering data for PCA. Let \(X\) be the \(n \times p\) matrix whose rows are observations and whose columns are centered variables. For a coefficient vector \(a\), each projected value is the corresponding row of \(X\) multiplied by \(a\). The resulting vector of projected values is \(Xa\).
Write the sample covariance matrix of the centered variables as \(S=X^\mathsf{T}X/(n-1)\). The sample variance of the projected values is
Thus the problem is to maximize \(a^\mathsf{T}Sa\) subject to \(a^\mathsf{T}a=1\). This uses the covariance matrix of the variables supplied to PCA. When those variables have been standardized, \(S\) is their correlation matrix; standardization changes the matrix and therefore can change the maximizing direction.
Why the length constraint is essential
The quadratic form \(a^\mathsf{T}Sa\) measures projected variance. If a nonzero vector \(a\) is multiplied by a constant \(c\), the projected values are also multiplied by \(c\), and their variance becomes \(c^2a^\mathsf{T}Sa\). For a nonzero covariance matrix, there is some direction with positive variance, so allowing its length to increase without restriction would make the objective unbounded above. If \(S\) is the zero matrix, however, \(a^\mathsf{T}Sa=0\) for every \(a\), and the objective remains identically zero even without a length restriction. The unit-length constraint defines a meaningful search over directions in either case.
The constraint fixes the total squared size of the coefficients, not the variance of the projection. It also does not require each coefficient to be positive. A direction and its negative give projected values with opposite signs but exactly the same variance, so both represent the same maximum-variance axis.
Deriving the constrained maximum
The covariance matrix \(S\) is symmetric. To find stationary directions under the unit-length constraint, form a Lagrangian, which incorporates the constraint using a multiplier \(\lambda\):
Differentiating with respect to \(a\) gives \(2Sa-2\lambda a=0\), or \(Sa=\lambda a\). Multiplying on the left by \(a^\mathsf{T}\) and using \(a^\mathsf{T}a=1\) shows that the projected variance at such a direction is \(\lambda\). This condition identifies stationary directions, but a stationary value must still be selected to maximize the objective.
The maximum exists: the unit-length constraint describes a closed and bounded set, and the quadratic objective is continuous. In two variables, write \(S=\begin{pmatrix}s_{11}&s_{12}\\s_{12}&s_{22}\end{pmatrix}\). A nonzero stationary direction requires the system \(Sa=\lambda a\) to have a nonzero solution. Setting its determinant to zero gives
Solving this quadratic gives two stationary values:
The larger value is the maximum projected variance. In a specific calculation, the associated coefficient ratio follows by substituting that value into either row of \(Sa=\lambda a\); then normalize the coefficients so their squared sum is one. If a row gives no information about the ratio, use the other row.
Worked Example: Maximizing a Given Covariance Matrix
Suppose two centered numeric features in a hypothetical insurance-pricing data set have sample covariance matrix \(S=\begin{pmatrix}5&1\\1&2\end{pmatrix}\). Find the first principal-component direction and its projected sample variance. Name the method and identify the rounding trap.
Method. Use the constrained maximum condition \(Sa=\lambda a\), select the larger stationary value, solve for the coefficient ratio, and normalize. The matrix equation is \( \begin{pmatrix}5&1\\1&2\end{pmatrix}\begin{pmatrix}a_1\\a_2\end{pmatrix} =\lambda\begin{pmatrix}a_1\\a_2\end{pmatrix}\).
The two-variable determinant equation is \((5-\lambda)(2-\lambda)-1=0\), which expands to \(\lambda^2-7\lambda+9=0\). Its roots are \(\lambda=(7\pm\sqrt{13})/2\). The larger one is \(\lambda_{\max}=(7+\sqrt{13})/2\approx5.3027756\), so the maximum projected variance is about \(5.3028\).
From the first row, \(5a_1+a_2=\lambda_{\max}a_1\). Therefore, \(a_2/a_1=\lambda_{\max}-5=(\sqrt{13}-3)/2\approx0.3027756\). Set \(a_1=1\) temporarily, so the unnormalized direction is approximately \((1,0.3027756)\). Its length is \(\sqrt{1+0.3027756^2}\approx1.0448\). Dividing both entries by this length gives the unit direction approximately \((0.9571,0.2898)\). Either its negative or this direction describes the same axis.
As a check, using the exact ratio in the stationary equation gives the variance \(\lambda_{\max}\), as required. Using the displayed rounded coefficients in the quadratic form instead gives \(5(0.9571^2)+2(0.9571)(0.2898)+2(0.2898^2)=5.30290529\), which rounds to \(5.3029\). The small difference from \(5.3028\) is due to rounding the coefficients: their displayed squared lengths sum to \(1.00002445\), not exactly \(1\).
Common mistake avoided: Do not report the rounded-coefficient substitution as \(5.3028\); it evaluates to \(5.3029\). The exact unit direction gives the maximum \((7+\sqrt{13})/2\), which rounds to \(5.3028\).
Working from observations rather than a supplied matrix
When observations are given, first center each variable and calculate the sample covariance matrix using the same \(1/(n-1)\) convention throughout. Then solve the constrained problem. This order matters: the matrix entries summarize the variation and joint variation that determine the projection variance.
Worked Example: Building the Covariance Matrix Before Optimizing
Four hypothetical policy records have centered values \((x,y)\) equal to \((-1,-2),(-1,0),(1,0),(1,2)\). Find the first principal-component direction and its projected sample variance.
Method. Compute the sample covariance entries from the centered observations, then apply the two-variable constrained optimization. The sample means are already zero. The sum of squared \(x\)-values is \(1+1+1+1=4\); the sum of squared \(y\)-values is \(4+0+0+4=8\); and the sum of cross-products is \(2+0+0+2=4\). Since \(n-1=3\),
The determinant equation is \((4/3-\lambda)(8/3-\lambda)-(4/3)^2=0\). Equivalently, factor \(4/3\) from the matrix and solve using \(\begin{pmatrix}1&1\\1&2\end{pmatrix}\). Its determinant equation is \((1-\mu)(2-\mu)-1=0\), or \(\mu^2-3\mu+1=0\), so the larger value is \(\mu=(3+\sqrt5)/2\). Rescaling by \(4/3\) gives \(\lambda_{\max}=(4/3)(3+\sqrt5)/2=(6+2\sqrt5)/3\approx3.4907\).
For the matrix with entries \(1,1,2\), the first row of its stationary equation gives \(a_1+a_2=\mu a_1\), so \(a_2/a_1=\mu-1=(1+\sqrt5)/2\approx1.6180\). Normalizing the direction \((1,1.6180)\) gives approximately \((0.5257,0.8507)\). Its squared length is approximately \(0.5257^2+0.8507^2=0.2764+0.7237=1.0001\), with the small excess caused by rounding.
The maximizing variance is about \(3.4907\), the larger stationary value. This is consistent with the direction having substantial weight on \(y\), whose sample variance is \(8/3\), while also using the positive covariance to capture joint variation.
Common mistake avoided: Dividing the sums of squares and cross-products by \(n\) would produce a different matrix and incorrectly scaled projected variance. These are sample covariances, so the denominator is \(n-1=3\).
Standardized variables and the role of covariance
The optimization depends on the covariance matrix of the inputs. If the variables are standardized to have sample variance one, the covariance matrix becomes their correlation matrix. This makes the optimization less sensitive to differences in measurement units, but it does not change the constrained objective: it remains the variance of a unit-length projection in the chosen data representation.
Worked Example: Finding the Direction from a Correlation Matrix
Suppose two standardized variables in a hypothetical claim-screening analysis have correlation \(0.6\). Their correlation matrix is \(S=\begin{pmatrix}1&0.6\\0.6&1\end{pmatrix}\). Find the first direction and compare its variance with the variance from using just one standardized variable.
Method. Apply the stationary condition to the correlation matrix, then evaluate the quadratic form for the resulting unit vector. The determinant equation is \((1-\lambda)^2-0.6^2=0\). Thus the stationary values are \(1.6\) and \(0.4\), and the maximum is \(1.6\).
For \(\lambda=1.6\), the first row gives \(a_1+0.6a_2=1.6a_1\), so \(a_2=a_1\). A unit vector in this direction is \((1/\sqrt2,1/\sqrt2)\), since its squared length is \(1/2+1/2=1\). Its projected variance is \(1(1/2)+2(0.6)(1/2)+1(1/2)=0.5+0.6+0.5=1.6\).
Using just the first standardized variable corresponds to direction \((1,0)\), whose projected variance is \(1\). Thus the joint direction captures more variance than either single variable alone. The variance for direction \((1/\sqrt2,-1/\sqrt2)\) is \(0.5-0.6+0.5=0.4\), confirming that the opposite-sign direction is the lower-variance one.
Common mistake avoided: The unit-length condition does not mean each coefficient is one or that the projected variance must be one. It means the sum of squared coefficients is one; the projected variance here is \(1.6\).
How to check a proposed direction
A calculated direction should satisfy both parts of the optimization. First, verify its length: \(a_1^2+a_2^2=1\). Second, evaluate \(a^\mathsf{T}Sa\) and confirm that it matches the larger stationary value. These checks catch coefficient-normalization errors and mistakes in the covariance cross-term.
For two variables, the quadratic form is \(s_{11}a_1^2+2s_{12}a_1a_2+s_{22}a_2^2\). The cross-term has a factor of two because both off-diagonal entries contribute when multiplying \(a^\mathsf{T}Sa\). If \(s_{12}\) is negative, a direction with opposite-sign coefficients can make that term positive and increase the projected variance. Ignoring the sign or omitting the factor of two can therefore change the answer.
Use the covariance matrix of centered variables, or the correlation matrix when the inputs have been standardized.
Maximize \(a^\mathsf{T}Sa\) subject to \(a^\mathsf{T}a=1\).
Use \(Sa=\lambda a\); in two variables, solve the determinant equation and use the larger stationary value.
Scale the coefficient vector to unit length, then check its quadratic-form variance against the maximum.
The optimization perspective is useful because it explains what the first component is doing: among all directions with the same coefficient length, it selects the one with the greatest sample variation. It is not automatically a prediction target, a causal effect, or a guarantee of useful information for a later model. The choice of covariance versus correlation matrix is consequential, so the data preparation must match the intended scale interpretation.
Check Your Understanding
For each problem, use the unit-length constraint and distinguish the maximizing variance from the variance in any proposed direction.
- For \(S=\begin{pmatrix}3&0\\0&1\end{pmatrix}\), find the maximum projected variance and a unit direction that attains it. (Answer: maximum variance \(3\); direction \((1,0)\) or \((-1,0)\).)
- For \(S=\begin{pmatrix}2&1\\1&2\end{pmatrix}\), find the maximum variance and a corresponding unit direction. (Answer: maximum variance \(3\); direction \((1/\sqrt2,1/\sqrt2)\) or its negative.)
- For \(S=\begin{pmatrix}6&-2\\-2&3\end{pmatrix}\), find the maximum variance and a corresponding unit direction. (Answer: maximum variance \(7\); direction \((2/\sqrt5,-1/\sqrt5)\) or its negative.)
- For \(S=\begin{pmatrix}4&1\\1&4\end{pmatrix}\), evaluate the projected variance for \(a=(0.6,0.8)\), and state whether it is maximal. (Answer: variance \(4.96\); it is not maximal, since the maximum is \(5\) in direction \((1/\sqrt2,1/\sqrt2)\) or its negative.)
- A centered two-variable data set has zero covariance matrix. What is \(a^\mathsf{T}Sa\) for every unit vector, and is there a unique maximizing direction? (Answer: the projected variance is \(0\) for every unit vector; there is no unique maximizing direction.)