From Predictors to Component Scores
Principal components analysis (PCA) can turn a set of correlated predictors into a new set of numerical features. Each new feature, called a principal-component score, is a weighted combination of the original predictors. A supervised model can then use some of these scores to predict an outcome.
There are two distinct operations. First, PCA constructs scores using only the predictor data: it finds directions of variation among those predictors. Second, a supervised method uses the scores together with the observed outcomes to estimate a prediction rule. Keeping these operations separate matters: PCA does not choose directions because they predict the outcome well.
The same preprocessing and loading vectors define the scores for every record to which this PCA mapping is applied. In a prediction workflow, those quantities are estimated from the training predictors and then held fixed when transforming other records. This is consistent with the training-only transformation principle: the data used to evaluate predictions must not be used to redefine the feature mapping.
A PCA score is not a new observed measurement. It is a calculated coordinate, in a direction specified by a loading vector. If the model uses only a subset of components, it uses the corresponding columns of \(T\). The outcome does not enter that score calculation; it enters later, when the supervised model is fitted.
Why PCA Produces These Directions
Suppose the processed training predictors are centered, and let \(S\) be their sample covariance matrix. A direction \(a\), represented by a vector of predictor weights, gives the score \(Za\). Its sample variance is \(a^\mathsf{T}Sa\). PCA finds a unit-length direction that maximizes this variance, then finds further unit-length directions subject to being orthogonal to the earlier ones.
The first direction must have unit length because otherwise multiplying all its weights by a large constant would make the score variance arbitrarily large without changing the direction. The eigenvector condition follows from maximizing the variance under that length constraint.
To see the implication, form the Lagrangian \(a^\mathsf{T}Sa-\lambda(a^\mathsf{T}a-1)\). Its derivative with respect to \(a\) is \(2Sa-2\lambda a\). Setting that derivative to zero gives \(Sa=\lambda a\). Thus the maximizing directions are eigenvectors of \(S\), and their score variances are the corresponding eigenvalues.
If \(Q\) contains orthonormal eigenvectors and \(\Lambda\) is the diagonal matrix of their eigenvalues, then the score covariance matrix is
This establishes that distinct PCA scores are uncorrelated in the training data when all components are retained. It does not establish that they are independent, nor that a component with high variance is necessarily useful for predicting the outcome.
Worked Example: Constructing Scores With Training Preprocessing
A hypothetical pricing team has two training predictors, \(x_1\) and \(x_2\). From the training data, their means are 10 and 20, and their standard deviations are 2 and 4. PCA was fitted to the standardized training predictors, producing loadings \(q_1=(0.8,0.6)^\mathsf{T}\) and \(q_2=(-0.6,0.8)^\mathsf{T}\). Calculate both scores for a record with predictors \((14,28)\), and then calculate the scores for a new record with predictors \((12,16)\). The loadings are orthonormal because \(0.8^2+0.6^2=1\) and \(0.8(-0.6)+0.6(0.8)=0\).
Method. Standardize each predictor with the training mean and standard deviation, then take the dot product with each loading vector. The common mistake this avoids is projecting raw measurements directly onto loadings that were obtained from standardized predictors.
For the first record, the standardized predictors are \(z_1=(14-10)/2=2\) and \(z_2=(28-20)/4=2\). Therefore,
For the new record, use the same training means and standard deviations: \(z_1=(12-10)/2=1\) and \(z_2=(16-20)/4=-1\). Its scores are
As a check on the first calculation, multiplying its score vector by the loading matrix reconstructs its standardized predictor vector: \(0.8(2.8)-0.6(0.4)=2\), and \(0.6(2.8)+0.8(0.4)=2\). Thus the calculated scores are consistent with the stated loadings and standardized values.
Separating the PCA Step From the Prediction Step
Once scores have been created, they can be used as inputs to a supervised model. For example, a regression might model a continuous outcome \(Y\) as a function of selected scores. In that second step, outcomes determine the fitted regression coefficients. They do not retroactively change the PCA loading vectors or the scores already constructed.
This distinction is especially important when describing the procedure. “PCA was fitted to the predictors, and a regression of the outcome on selected PCA scores was fitted afterward” names two operations accurately. Saying “PCA predicted the outcome” blurs the roles: PCA creates features, while the supervised model estimates the relationship between those features and the outcome.
Worked Example: Finding PCA Directions From a Covariance Matrix
Suppose two centered training predictors have sample covariance matrix \(S=\begin{pmatrix}4&3\\3&4\end{pmatrix}\). Find the two loading directions and the scores for a centered record with predictor deviations \((2,0)\). This hypothetical calculation concerns the predictor data only; no outcome is supplied or needed.
Method. Find the eigenvalues and normalized eigenvectors of \(S\), then project the centered record onto each eigenvector. This avoids treating the covariance entries themselves as loadings.
The characteristic equation is
For \(\lambda_1=7\), solving \((S-7I)q_1=0\) gives equal coordinates, so the unit eigenvector is \(q_1=(1,1)^\mathsf{T}/\sqrt{2}\). For \(\lambda_2=1\), solving \((S-I)q_2=0\) gives opposite coordinates, so a unit eigenvector is \(q_2=(1,-1)^\mathsf{T}/\sqrt{2}\). The eigenvalues sum to 8, which equals the trace \(4+4\) of \(S\), providing a check.
The centered record is \(z=(2,0)^\mathsf{T}\). Its scores are
The two scores are equal for this record, even though their directions differ. The score on each direction is the record’s projection onto that direction; it is not the eigenvalue and is not a measure of the outcome.
Worked Example: Fitting a Regression to Component Scores
A hypothetical portfolio study has already created two PCA scores for four training records. The score pairs and observed claim-cost indices are \((1,0),12\), \((-1,0),8\), \((0,1),7\), and \((0,-1),13\), where each pair lists \((t_1,t_2)\) followed by \(Y\). Fit an ordinary least squares model with an intercept and both scores, then predict for a record with scores \((0.5,-0.25)\).
Method. Fit the outcome model by least squares using the scores as predictors. The common mistake this avoids is assuming that because PCA is unsupervised, the later regression also ignores the outcomes.
The mean outcome is \(\bar Y=(12+8+7+13)/4=10\). Each score has mean zero. The score columns are orthogonal: \(\sum_i t_{i1}t_{i2}=0\). Their sums of squares are \(\sum_i t_{i1}^2=1+1=2\) and \(\sum_i t_{i2}^2=1+1=2\). Therefore, the intercept is 10, and the least-squares coefficients are
The fitted model is \(\hat Y=10+2t_1-3t_2\). It reproduces all four observed outcomes: the predictions are 12, 8, 7, and 13, respectively, so the residual sum of squares is zero. For scores \((0.5,-0.25)\), the predicted value is \(10+2(0.5)-3(-0.25)=10+1+0.75=11.75\). The arithmetic check confirms the negative second coefficient increases the prediction when \(t_2\) is negative.
Here the supervised fit uses \(Y\) to estimate \(\hat\beta_1\), \(\hat\beta_2\), and the intercept. The PCA score definitions, by contrast, came from the predictor values and loading vectors alone.
Common Traps and Practical Interpretation
A frequent calculation error is to apply loadings to raw values when PCA was fitted to centered or standardized predictors. In the first worked example, the incorrect raw projection for the first record’s first component would be \(0.8(14)+0.6(28)=28\). The correct score is 2.8, because the loadings were fitted to standardized values \((2,2)\), not raw values \((14,28)\). The score formula and the preprocessing used to create the loadings must agree.
A second trap is to use \(Y\) while calculating PCA loadings in an attempt to make components more predictive. That is not ordinary PCA: its directions are defined by variation in the predictor data. To use outcome information, fit an appropriate supervised model after creating the scores. Changing the PCA directions based on outcomes changes the feature-construction procedure and must not be described as outcome-free PCA.
- Do not confuse a loading with a score. A loading is a weight defining a component direction; a score is a record’s projection onto that direction.
- Do not equate high predictor variance with predictive value. PCA ranks directions by predictor variance, not by their association with the outcome.
- Do not interpret a score coefficient as an original-predictor coefficient. It is the estimated effect of a one-unit change in a component score, with the other included model inputs held fixed.
- Do not recalculate the mapping for each set of records. Reusing a fitted PCA transformation means applying the same preprocessing parameters and loading vectors.
PCA scores are useful when they provide a compact representation of correlated predictors, but the supervised model remains responsible for learning the outcome relationship. Any performance claim must therefore concern the full workflow—score construction followed by outcome-based fitting—not PCA in isolation.
Check Your Understanding
Work through each problem by separating the predictor transformation from any outcome-based fit.
- A training predictor has mean 6 and standard deviation 2. A record has value 10, and its loading on a component is 1. What is its standardized value and component score? (Answer: standardized value \(2\); score \(2\).)
- For a two-predictor record with processed values \((1,2)\), the loading vector is \((0.6,0.8)^\mathsf{T}\). Calculate the component score. (Answer: \(0.6(1)+0.8(2)=2.2\).)
- A covariance matrix is \(\begin{pmatrix}2&0\\0&5\end{pmatrix}\). State the eigenvalues and one unit loading vector for the direction with the larger variance. (Answer: eigenvalues \(5\) and \(2\); a loading vector for the larger-variance direction is \((0,1)^\mathsf{T}\).)
- Four records have one score \(t=(-1,0,0,1)\) and outcomes \(Y=(4,5,7,10)\). In an ordinary least squares model with an intercept and this score, calculate the intercept, slope, and fitted value at \(t=2\). (Answer: intercept \(6.5\); slope \(3\); fitted value \(12.5\).)
- A team uses outcomes to choose PCA loading vectors, then says the PCA features were constructed without outcome information. Is that description correct? State the distinction that should be made. (Answer: No. Ordinary PCA loadings are estimated from predictors without \(Y\); using \(Y\) to choose directions makes feature construction outcome-informed, distinct from fitting a supervised model to fixed PCA scores.)