Scaling Choices Before PCA Feature Creation

Why Scaling Changes the PCA Features

PCA creates new predictors by finding directions of variation in the input variables. The direction can depend substantially on the scale of those variables. For example, a measurement recorded in dollars rather than thousands of dollars has the same underlying information, but its numerical variance is multiplied by a million. Covariance-based PCA responds to that change in units; correlation-based PCA does not.

This is a modeling choice, not a mechanical rule that every predictor must be standardized. If input variables have different units, standardization can prevent a large-unit variable from dominating merely because of its measurement scale. If their variances have meaningful substantive differences, standardization can instead give a low-variance variable influence comparable to a high-variance one. The appropriate choice depends on what variation the PCA features should represent.

Definition: For centered predictors with covariance matrix \(\Sigma\), covariance-based PCA uses the eigenvectors of \(\Sigma\). Correlation-based PCA first divides each centered predictor by its standard deviation and uses the eigenvectors of the resulting correlation matrix. A component score is the projection of a centered record (or standardized record) onto a chosen eigenvector.

Suppose there are \(p\) predictors and \(n\) training records. Let \(x_j\) be predictor \(j\), with training mean \(\bar{x}_j\) and sample standard deviation \(s_j\). Covariance PCA works with the centered values \(x_j-\bar{x}_j\). Correlation PCA works with the standardized values \((x_j-\bar{x}_j)/s_j\). In both cases the PCA directions are obtained from the training data. When the resulting scores are later used as predictors, the same fitted means, standard deviations, and directions define how to calculate them.

\Sigma_{jk}=\operatorname{Cov}(X_j,X_k),\qquad R_{jk}=\frac{\Sigma_{jk}}{s_j s_k},\qquad z_j=\frac{x_j-\bar{x}_j}{s_j}.

Here \(\Sigma\) is the covariance matrix, \(R\) is the correlation matrix, and \(z_j\) is the standardized value of predictor \(j\). The diagonal entries of \(\Sigma\) are the predictor variances \(s_j^2\); the diagonal entries of \(R\) are 1. A unit-length eigenvector \(v\) gives a component score \(v^\mathsf{T}x_c\) for a centered record \(x_c\) in covariance PCA, or \(v^\mathsf{T}z\) in correlation PCA. The two score formulas use different input coordinates, so their numerical score scales need not agree.

Why the Leading Direction Maximizes Variance

PCA does not use the outcome being predicted to select its directions. The leading direction is the unit-length linear combination of inputs whose scores have the largest variance. This explains both the eigenvector calculation and why scaling the inputs can change the result.

Derivation: For centered input vector \(X_c\), the score in a direction \(v\) is \(v^\mathsf{T}X_c\). Its variance is \(v^\mathsf{T}\Sigma v\). To find the direction with largest variance while fixing its length, maximize \(v^\mathsf{T}\Sigma v\) subject to \(v^\mathsf{T}v=1\). The Lagrangian is \(L(v,\lambda)=v^\mathsf{T}\Sigma v-\lambda(v^\mathsf{T}v-1)\). Differentiating with respect to \(v\) gives \(2\Sigma v-2\lambda v=0\), so \(\Sigma v=\lambda v\). Thus a maximizing direction is an eigenvector of \(\Sigma\); its variance is the corresponding eigenvalue. The maximum is the largest eigenvalue. Replacing \(\Sigma\) with \(R\) gives the analogous result for correlation PCA.

This argument assumes finite variances and a covariance matrix formed from the predictors under analysis. It identifies a direction with maximum input variance, not necessarily maximum predictive value for the response. PCA may be useful for creating compact predictors, but a high-variance component is not automatically a strong predictor of the outcome.

Worked Comparisons

Worked Example: Same Correlation, Unequal Predictor Scales

Suppose two centered hypothetical predictors have covariance matrix \(\begin{pmatrix}100&6\\6&1\end{pmatrix}\). A centered record has coordinates \((10,-1)\). Find its first covariance-based and correlation-based PCA scores. Use the leading eigenvector for each method, and report final scores to four decimal places.

Method and covariance PCA. The covariance matrix has trace \(101\) and determinant \(100(1)-6(6)=64\). Its eigenvalues are

\lambda=\frac{101\pm\sqrt{101^2-4(64)}}{2} =\frac{101\pm\sqrt{9945}}{2}.

Since \(\sqrt{9945}\approx99.7246\), the larger eigenvalue is \((101+99.7246)/2\approx100.3623\). From the first row of \((\Sigma-\lambda I)v=0\), the ratio of the second eigenvector coordinate to the first is \((\lambda-100)/6\approx0.0604\). Normalizing gives the leading direction approximately \((0.9982,0.0603)^\mathsf{T}\).

Using those displayed, rounded coordinates consistently, the score is \(0.9982(10)+0.0603(-1)=9.9820-0.0603=9.9217\). Using more precise direction coordinates, approximately \((0.998182,0.060275)^\mathsf{T}\), gives \(0.998182(10)-0.060275=9.981820-0.060275=9.921545\), or \(9.9215\) to four decimal places.

Correlation PCA. The standard deviations are \(10\) and \(1\), so the correlation matrix is \(\begin{pmatrix}1&0.6\\0.6&1\end{pmatrix}\). Its eigenvalues are \(1.6\) and \(0.4\), and a unit leading eigenvector is \((1/\sqrt{2},1/\sqrt{2})^\mathsf{T}\approx(0.7071,0.7071)^\mathsf{T}\). The record's standardized coordinates are \((10/10,-1/1)=(1,-1)\). Its score is \(0.7071(1)+0.7071(-1)=0\).

Check and trap avoided. The results differ because covariance PCA gives much more weight to the first predictor's larger variance, while correlation PCA gives both standardized predictors equal variance. A common mistake is to apply the correlation eigenvector directly to the raw record \((10,-1)\). Correlation PCA requires standardized inputs; doing so would produce an incorrect score.

Worked Example: Converting a Predictor's Units

Let centered predictors \(Z_1,Z_2\) have variances \(4,1\) and covariance \(1\). Define \(W_1=10Z_1\), so the first predictor is now expressed in units ten times as large. Find the leading covariance-PCA direction for \((W_1,Z_2)\), compare it with the correlation-PCA direction, and calculate both scores for a record with \((Z_1,Z_2)=(2,1)\).

Method and covariance matrix. Scaling \(Z_1\) by 10 multiplies its variance by \(10^2\) and its covariance with \(Z_2\) by 10. Therefore,

\operatorname{Var}(W_1)=100(4)=400,\qquad \operatorname{Cov}(W_1,Z_2)=10(1)=10,\qquad \Sigma= \begin{pmatrix}400&10\\10&1\end{pmatrix}.

The trace is \(401\), and the determinant is \(400(1)-10(10)=300\). The eigenvalue discriminant is \(401^2-4(300)=160801-1200=159601\), so the larger eigenvalue is

\lambda_1=\frac{401+\sqrt{159601}}{2} \approx\frac{401+399.5009}{2} \approx400.2505.

The ratio of the second coordinate to the first in the leading eigenvector is \((\lambda_1-400)/10\approx0.025047\). Normalizing gives approximately \((0.999686,0.025039)^\mathsf{T}\). The record has \(W_1=10(2)=20\), so its covariance-based score is \(0.999686(20)+0.025039(1)=19.99372+0.025039\approx20.0188\).

Correlation PCA. The correlation is \(10/\sqrt{400(1)}=0.5\), giving \(R=\begin{pmatrix}1&0.5\\0.5&1\end{pmatrix}\). Its leading unit eigenvector is \((0.7071,0.7071)^\mathsf{T}\). The standardized record is \((20/20,1/1)=(1,1)\), so its correlation-based score is \(0.7071+0.7071=1.4142\).

Check and trap avoided. The discriminant for a two-by-two symmetric matrix \(\begin{pmatrix}a&b\\b&d\end{pmatrix}\) is \((a-d)^2+4b^2\), which here is \(399^2+4(10^2)=159201+400=159601\). A common arithmetic trap is to substitute the trace or an incorrectly expanded expression for this discriminant. The unit conversion changes the covariance direction, but not the correlations or the correlation-PCA direction. The two scores should not be compared as if they were in the same units: each was calculated in a different coordinate system.

Worked Example: When Variance Differences Carry Meaning

A hypothetical analyst studies two centered predictors measured in the same units. Their covariance matrix is \(\begin{pmatrix}9&2.4\\2.4&1\end{pmatrix}\). A particular record has centered coordinates \((2,1)\), and the corresponding standard deviations are \(3\) and \(1\). Find the leading directions and the record's score under both PCA choices. Explain which choice preserves the greater absolute variation in the original units.

Method and covariance PCA. The trace is \(10\), and the determinant is \(9(1)-2.4^2=9-5.76=3.24\). The eigenvalue discriminant is \(10^2-4(3.24)=100-12.96=87.04\), with square root approximately \(9.3295\). Thus the larger eigenvalue is \((10+9.3295)/2\approx9.6648\). The eigenvector coordinate ratio is \((9.6648-9)/2.4\approx0.2770\). Normalization gives approximately \((0.9637,0.2670)^\mathsf{T}\). The record's covariance score is \(0.9637(2)+0.2670(1)=1.9274+0.2670=2.1944\).

Correlation PCA. The correlation is \(2.4/(3\cdot1)=0.8\), so \(R=\begin{pmatrix}1&0.8\\0.8&1\end{pmatrix}\). Its leading eigenvalue is \(1.8\) and its leading direction is \((0.7071,0.7071)^\mathsf{T}\). The standardized record is \((2/3,1)\). Its score is \(0.7071(2/3)+0.7071(1)\approx0.4714+0.7071=1.1785\).

Decision and check. If the larger variance in the first predictor represents meaningful variation the analysis should retain, covariance PCA is aligned with that goal. Correlation PCA instead treats one standard deviation of either predictor as comparable, regardless of its original scale. The two choices are not competing calculations of one universal answer: they encode different definitions of important variation. A common mistake is to standardize automatically and thereby erase a meaningful difference in absolute variability.

Choosing a Scaling Rule

A practical decision begins with the meaning of the input variables, not with the eigenvalues alone. Covariance PCA is often reasonable when predictors share units and their absolute variances are meaningfully comparable. Correlation PCA is often reasonable when predictors use different units, or when the analysis is intended to give each predictor equal initial variance. Neither choice ensures that the resulting components will predict an outcome well; predictive performance must be assessed as part of the modeling process.

Question	Implication for the scaling choice
Are predictor units very different, such as dollars and years?	Consider correlation PCA so numerical units alone do not drive the directions.
Do the predictors share units and does larger absolute variation matter?	Consider covariance PCA to preserve that difference in variability.
Could a low-variance predictor be mostly measurement noise?	Standardizing it may amplify that noise; inspect data quality and context before choosing correlation PCA.
Will the PCA scores be used as predictors?	Compare the chosen construction against a suitable baseline using the intended predictive evaluation.

Standardization is not the only possible scaling rule. A practitioner might use a prespecified scale based on subject-matter units, but that rule must be defined clearly and applied consistently. Whatever scaling is chosen, record its definition. For a new record, use the same centering and scaling values used to construct the PCA directions; otherwise the score no longer has the intended coordinate meaning. In particular, calculating a new record's standardization with a different set of means or standard deviations changes its score.

Common Mistakes and Practical Checks

Assuming that PCA is unit-invariant. Covariance PCA generally changes when a predictor is converted to different units. Correlation PCA is invariant to positive changes of measurement units because the standard deviation changes by the same factor as the centered values.
Mixing raw and standardized coordinates. A correlation-PCA direction must be applied to standardized values, not unscaled centered values.
Treating standardized variance as automatically better. Standardization can make a noisy low-variance variable as influential initially as a stable high-variance variable.
Comparing component scores across scaling methods by magnitude. The coordinate systems and score units differ. Compare the construction and its predictive utility, not just the numerical size of one score.
Interpreting an eigenvalue as predictive strength. It measures variance in the input direction, not association with the response.

Key takeaway: Covariance PCA emphasizes absolute variation in the original centered units. Correlation PCA emphasizes variation after each predictor is expressed in standard-deviation units. Choose the rule that matches the intended meaning of variation, then calculate and use scores in that same coordinate system.

For an actuarial prediction task, document the variables included, the scaling rule, and why it is suitable. This makes the feature construction interpretable and reproducible, while keeping the distinction clear between explaining variation among predictors and improving prediction of the response. This treatment is aligned with the SOA Exam PA (Predictive Analytics) syllabus.

Check Your Understanding

For each problem, identify the relevant scaling implication or carry out the requested calculation.

A covariance matrix is \(\begin{pmatrix}16&0\\0&1\end{pmatrix}\). What is a leading covariance-PCA direction? If both variables are standardized, what is the correlation matrix, and is its leading direction unique? (Answer: Covariance PCA has leading direction \((1,0)^\mathsf{T}\), up to sign. The correlation matrix is the identity, so all unit directions are tied and the leading direction is not unique.)
Two predictors have standard deviations 5 and 2, and covariance 6. What is their correlation? If their correlation matrix has positive off-diagonal correlation, what is its leading direction? (Answer: The correlation is \(6/(5\cdot2)=0.6\); the leading correlation-PCA direction is \((0.7071,0.7071)^\mathsf{T}\), up to sign.)
A two-variable correlation matrix is \(\begin{pmatrix}1&-0.4\\-0.4&1\end{pmatrix}\). Give the leading eigenvalue and a unit leading direction. (Answer: The leading eigenvalue is \(1.4\), and a leading direction is \((0.7071,-0.7071)^\mathsf{T}\), up to sign.)
A predictor \(X_1\) is converted to \(X_1^*=100X_1\), with the other predictor unchanged. Which PCA construction is generally affected by this conversion, and which is unchanged? (Answer: Covariance PCA is generally affected; correlation PCA is unchanged for a positive unit conversion, provided the same records and variables are used.)
For covariance matrix \(\begin{pmatrix}4&1\\1&1\end{pmatrix}\), find the leading eigenvalue and an approximate unit leading direction. (Answer: \(\lambda_1=(5+\sqrt{13})/2\approx4.3028\); the coordinate ratio is \((\lambda_1-4)\approx0.3028\), giving a unit direction approximately \((0.9571,0.2898)^\mathsf{T}\), up to sign.)

What You'll Learn