Why PCA Features Can Look Different Across Fits
Principal component analysis (PCA) creates new predictors by projecting centered or scaled input variables onto directions called principal components. A downstream regression model may use the resulting component scores. The numerical values of those scores are useful, but their signs and labels are not unique: two valid PCA outputs can describe the same component directions with opposite signs or list the same directions in a different order.
This matters when reviewing model coefficients. A coefficient that changes from positive to negative may reflect only a sign convention, not a changed relationship or changed prediction. To interpret or compare coefficients, first check whether the PCA coordinates have been aligned.
The PCA directions are eigenvectors of a covariance or correlation matrix, depending on the chosen preprocessing. If \(v\) is an eigenvector with eigenvalue \(\lambda\), then \(-v\) is also an eigenvector with the same eigenvalue. Consequently, the data do not determine whether that direction is reported as \(v\) or \(-v\). Component order is also a labeling convention: software commonly sorts components by decreasing eigenvalue, but a valid output can list the same components in another order.
For the equivalence results below, assume that both fits use the same input variables and the same centering and scaling, and that their score matrices differ only by component sign changes and/or reordering. Also assume that the downstream model uses those scores with coefficients transformed consistently. A different preprocessing transformation, a different selected component set, or a model refit with different specifications is not covered by this equivalence.
Transforming Coefficients Along With the Scores
A sign or order change to scores must be accompanied by the matching change to coefficients. Otherwise, the linear predictor generally changes. The simplest case is one sign reversal: if \(z_j\) becomes \(-z_j\), its coefficient \(\beta_j\) must also become \(-\beta_j\). Their product is unchanged.
For a general combination of sign changes and reordering, let \(Q\) be a signed permutation matrix: it permutes coordinates and may multiply selected coordinates by \(-1\). Such a matrix is orthogonal, so \(QQ^\mathsf{T}=I\). The new score vector is \(z'=zQ\), using row-vector notation for an individual record. If the new coefficient vector is \(\beta'=Q^\mathsf{T}\beta\), then:
This is the key justification: the coordinate representation changes, but the matched products contributing to the linear predictor do not. For a generalized linear model, the response-scale prediction is obtained by applying the inverse link function to \(\eta\). Since \(\eta'=\eta\), the response-scale predictions are also identical.
Determine which score columns were sign-reversed and how the columns were reordered.
Reverse the sign of coefficients for reversed score columns and move each coefficient with its component.
Substitute the new scores and coefficients into the model and verify that the intercept and total component contribution are unchanged.
Worked Examples
Worked Example: Reversing One Component in a Logistic Model
A hypothetical claims-screening model uses two PCA scores. For one record, the scores are \(z=(2,-1)\). The fitted logistic model has intercept \(\beta_0=0.1\) and coefficients \(\beta=(0.3,-0.2)\). A second valid PCA output reverses the sign of the first component but leaves the second unchanged. Find the transformed coefficients and verify the predicted probability.
Method. Reverse the score and coefficient for the first component together. First calculate the original linear predictor:
For a logistic model, the predicted probability is \(1/(1+e^{-\eta})\). Thus the original probability is \(1/(1+e^{-0.9})\), which is \(0.7109\) to four decimal places.
The transformed scores are \(z'=(-2,-1)\). The first coefficient changes from \(0.3\) to \(-0.3\), while the second remains \(-0.2\), so \(\beta'=(-0.3,-0.2)\). Substitution gives:
The transformed model therefore gives \(1/(1+e^{-0.9})=0.7109\), to four decimal places, as before. The check is that both the linear predictor and the probability agree. Common mistake avoided: reversing a score but leaving its coefficient unchanged would change that component's contribution from \(0.6\) to \(-0.6\).
Worked Example: Reordering Two Components
For a hypothetical pricing model with an identity link, one record has component scores \(z=(2,-1)\), coefficients \(\beta=(0.25,0.4)\), and intercept \(\beta_0=-0.1\). A second output lists the same two components in reverse order. Calculate the linear predictor before and after reordering.
Method. First compute the original predictor:
After reordering, the scores are \(z'=(-1,2)\). The first new score is the old second score, so its coefficient is \(0.4\); the second new score is the old first score, so its coefficient is \(0.25\). Thus \(\beta'=(0.4,0.25)\), and:
The predictors agree, so the identity-link predictions agree as well. Common mistake avoided: treating the coefficient vector as fixed by position rather than moving each coefficient with its component. The check is that the two component contributions are still \(-0.4\) and \(0.5\), even though their positions have swapped.
Worked Example: Combining a Reordering and a Sign Reversal
A model uses three component scores. For one record, \(z=(1,-2,0.5)\), \(\beta=(0.6,-0.1,0.4)\), and \(\beta_0=0.2\). A new PCA output puts the old third component first, the old first component second with its sign reversed, and the old second component third. Find the matching coefficients and verify the predictor.
Method. The original predictor is:
The new scores are \(z'=(0.5,-1,-2)\): the old third score comes first, the negative of the old first score comes second, and the old second score comes third. The matching coefficients are \(\beta'=(0.4,-0.6,-0.1)\). The middle coefficient is negative because the corresponding score was sign-reversed. Now calculate:
The predictor is unchanged. The check is that the three component contributions, in their new order, are \(0.2\), \(0.6\), and \(0.2\), the same contributions as before. Common mistake avoided: changing the order correctly but forgetting that the second new score also has its sign reversed.
What Coefficient Comparisons Can and Cannot Tell You
A coefficient is attached to a particular coordinate, not to a component name that is intrinsically fixed across outputs. If a component's sign is reversed, the coefficient's sign reverses too. If components are reordered, their coefficients move with them. Comparing raw coefficient vectors without accounting for these changes can therefore make equivalent models appear different.
To compare fitted component coefficients, first match the score columns across outputs, then account for any sign reversals, and only then compare the corresponding coefficient values. Loadings should be checked as well: a loading vector and its negative describe the same direction. If the score columns are not equivalent up to sign and order, the simple coefficient mapping above does not establish that the predictions match.
The equivalence also depends on keeping the component set fixed. If a model uses a different set of components, or if the preprocessing or downstream model changes, the scores may not be related by a signed reordering. In that case, applying a sign-and-order correction alone is not enough to claim prediction invariance. Equal eigenvalues can also allow more than sign or order differences, because directions within the tied eigenspace may be represented differently; the simple signed-permutation check does not cover every such case.
Common Exam Trap
A frequent error is to reorder scores but retain coefficients in their old positions. Consider the worked example with \(z=(2,-1)\), \(\beta=(0.25,0.4)\), and intercept \(-0.1\). The correct reordered scores are \((-1,2)\), and the coefficients must be \((0.4,0.25)\), giving \(-0.1-0.4+0.5=0\). If the old coefficients are incorrectly left as \((0.25,0.4)\), the result is \(-0.1+(-1)(0.25)+(2)(0.4)=0.45\), which falsely suggests a changed prediction. The error is not a subtle modeling effect: the coefficients have been paired with the wrong score columns.
Check Your Understanding
For each problem, track the score-to-coefficient pairing and verify the final predictor or probability.
- A linear model has intercept \(0.2\), scores \((3,-1)\), and coefficients \((0.1,0.5)\). The second score is sign-reversed. What coefficient vector preserves the predictor, and what is that predictor? (Answer: coefficients \((0.1,-0.5)\); predictor \(0\).)
- A model has intercept \(0.1\), scores \((1,4)\), and coefficients \((-0.2,0.15)\). The score order is reversed. Give the reordered coefficients and verify the predictor. (Answer: coefficients \((0.15,-0.2)\); predictor \(0.5\).)
- A model has intercept \(-0.3\), scores \((-2,1,3)\), and coefficients \((0.4,-0.2,0.1)\). The new order is old component 3, then old component 1 sign-reversed, then old component 2. Give the new scores, coefficients, and predictor. (Answer: scores \((3,2,1)\); coefficients \((0.1,-0.4,-0.2)\); predictor \(-1\).)
- A logistic model has intercept \(-0.1\), scores \((2,-1)\), and coefficients \((0.3,0.2)\). The first score is sign-reversed. What is the matching coefficient vector, and what is the predicted probability to four decimal places? (Answer: coefficients \((-0.3,0.2)\); probability \(0.5744\).)
- A model has intercept \(-0.1\), scores \((2,-1)\), and coefficients \((0.25,0.4)\). The score order is reversed. Give the reordered coefficients and calculate the predictor; check that the original predictor agrees. (Answer: reordered coefficients \((0.4,0.25)\); both predictors are \(0\).)