Factor Score Estimates by Regression and Bartlett Methods

From a Fitted Factor Model to Factor Scores

A factor model describes how observed variables relate to latent factors, but fitting the model does not directly reveal each observation’s factor values. Factor-score methods provide estimates of those values from an observed response vector. The estimates depend on the fitted loadings, factor covariance, and uniquenesses; they are not additional parameters already determined uniquely by the covariance model.

Let $x$ be a centered response vector for one observation, with $p$ measured variables and $q$ factors. Treat the fitted model parameters as fixed while calculating scores. Write the model as $x=\Lambda f+\varepsilon$, where $\Lambda$ is the $p\times q$ loading matrix, $f$ is the $q$-dimensional factor vector, and $\varepsilon$ is the $p$-dimensional unique-error vector. Assume $E[f]=0$, $\operatorname{Cov}(f)=\Phi$, $E[\varepsilon]=0$, $\operatorname{Cov}(\varepsilon)=\Psi$, and $\operatorname{Cov}(f,\varepsilon)=0$. The matrix $\Psi$ is positive definite, and $\Phi$ is positive definite. Thus the model-implied covariance of $x$ is $\Sigma=\Lambda\Phi\Lambda'+\Psi$.

Definition: The regression factor-score estimate is the conditional mean of the factors given the response under a centered Gaussian factor model: $$ \hat f_R(x)=E[f\mid x]=\Phi\Lambda'\Sigma^{-1}x. $$ The Bartlett factor-score estimate is the minimizer, over factor vectors $g$, of the uniqueness-weighted residual sum of squares: $$ \hat f_B(x)=\underset{g\in\mathbb R^q}{\operatorname{argmin}}\;(x-\Lambda g)'\Psi^{-1}(x-\Lambda g). $$

For the regression formula as a conditional mean, assume in addition that $f$ and $\varepsilon$ are jointly Gaussian (in particular, it suffices that they are independent Gaussian vectors). The Bartlett estimate requires no Gaussian assumption for its weighted least-squares definition. It is unique when $\Lambda$ has full column rank, since $\Psi$ is positive definite. For an uncentered observation $z$ with fitted mean $\mu$, apply these formulas to $x=z-\mu$.

Deriving the Regression Score

Under the Gaussian model, the joint distribution of $f$ and $x$ is Gaussian. Their covariance matrices are $\operatorname{Cov}(f)=\Phi$, $\operatorname{Cov}(x)=\Sigma$, and $\operatorname{Cov}(f,x)=\Phi\Lambda'$. To verify the conditional-mean formula, define $r=f-\Phi\Lambda'\Sigma^{-1}x$. Direct calculation gives

\operatorname{Cov}(r,x) =\Phi\Lambda'-\Phi\Lambda'\Sigma^{-1}\Sigma =0.

Because $r$ and $x$ are jointly Gaussian and uncorrelated, they are independent. Also $E[r]=0$. Therefore $E[r\mid x]=0$, so $E[f\mid x]=\Phi\Lambda'\Sigma^{-1}x$, as claimed. This derivation explains why the regression score depends on the factor covariance as well as on the loadings and uniquenesses.

A second expression follows by completing the square in the conditional Gaussian density. Apart from terms not involving $f$, its negative twice log-density is

f'\Phi^{-1}f+(x-\Lambda f)'\Psi^{-1}(x-\Lambda f).

Collecting terms involving $f$ gives quadratic coefficient $A=\Phi^{-1}+\Lambda'\Psi^{-1}\Lambda$ and linear coefficient $\Lambda'\Psi^{-1}x$. Completing the square shows that the conditional mean is $A^{-1}\Lambda'\Psi^{-1}x$. Consequently, the regression score can also be written as

\hat f_R(x) =(\Phi^{-1}+\Lambda'\Psi^{-1}\Lambda)^{-1}\Lambda'\Psi^{-1}x.

This form makes the role of the factor distribution explicit: the term $\Phi^{-1}$ contributes information from the factor covariance, while the data enter through the uniqueness-weighted loading and response terms.

Worked Example: Orthogonal Factors and a Corrected Bartlett Calculation

Let $q=2$, $p=3$, $\Phi=I$, $\Psi=I$, and \[ \Lambda=\begin{pmatrix}1&0\\0&1\\1&-1\end{pmatrix}, \qquad x=\begin{pmatrix}1\\2\\0\end{pmatrix}. \] Calculate both scores. First, \[ \Lambda'\Lambda=\begin{pmatrix}2&-1\\-1&2\end{pmatrix}, \qquad \Lambda'x=\begin{pmatrix}1\\2\end{pmatrix}. \]

For Bartlett scoring, the normal equations give \[ \hat f_B=(\Lambda'\Lambda)^{-1}\Lambda'x =\frac13\begin{pmatrix}2&1\\1&2\end{pmatrix} \begin{pmatrix}1\\2\end{pmatrix}. \] The matrix-vector product is $(4,5)'$, so $\hat f_B=(4/3,5/3)'$. Equivalently, using the loading matrix above, $\Lambda'\Lambda$ has off-diagonal entry $-1$, not $+1$, and its inverse is $\frac13\begin{pmatrix}2&1\\1&2\end{pmatrix}$. Thus this calculation follows the stated loading matrix and response exactly.

For regression scoring, $\Phi^{-1}+\Lambda'\Psi^{-1}\Lambda=I+\Lambda'\Lambda=\begin{pmatrix}3&-1\\-1&3\end{pmatrix}$, whose inverse is $\frac18\begin{pmatrix}3&1\\1&3\end{pmatrix}$. Hence \[ \hat f_R=\frac18\begin{pmatrix}3&1\\1&3\end{pmatrix} \begin{pmatrix}1\\2\end{pmatrix} =\begin{pmatrix}5/8\\7/8\end{pmatrix}. \] The two methods differ: Bartlett fits the response by weighted least squares, whereas regression also accounts for the factor distribution.

Deriving the Bartlett Score

The Bartlett method chooses factor values that make the observed response close to its common-factor part, measuring residuals in units of uniqueness. Define $Q(g)=(x-\Lambda g)'\Psi^{-1}(x-\Lambda g)$. Expanding and differentiating with respect to $g$ gives

\nabla_g Q(g)=-2\Lambda'\Psi^{-1}x+ 2\Lambda'\Psi^{-1}\Lambda g.

Setting the gradient to zero yields the normal equations. If $\Lambda$ has full column rank, then for every nonzero vector $a$, $a'\Lambda'\Psi^{-1}\Lambda a=(\Lambda a)'\Psi^{-1}(\Lambda a)>0$. Thus $\Lambda'\Psi^{-1}\Lambda$ is positive definite, the objective is strictly convex, and the unique minimizer is

Result: Under positive-definite $\Psi$ and full column rank of $\Lambda$, the Bartlett factor-score estimate is $$ \hat f_B(x)=(\Lambda'\Psi^{-1}\Lambda)^{-1}\Lambda'\Psi^{-1}x. $$

A useful property follows by substituting the model $x=\Lambda f+\varepsilon$ into this expression. Conditional on a fixed factor vector $f$, and assuming $E[\varepsilon\mid f]=0$,

E[\hat f_B(x)\mid f] =(\Lambda'\Psi^{-1}\Lambda)^{-1}\Lambda'\Psi^{-1}\Lambda f =f.

So Bartlett scores are conditionally unbiased for $f$ under these assumptions. This does not mean each score equals the latent factor: the realized error term still contributes to the estimate. Nor does conditional unbiasedness make Bartlett scoring a conditional mean or a minimum mean-squared-error predictor.

Worked Example: Oblique Factors Change the Regression Score

Suppose $p=q=2$, $\Lambda=I$, $\Psi=I$, and \[ \Phi=\begin{pmatrix}1&1/2\\1/2&1\end{pmatrix}, \qquad x=\begin{pmatrix}2\\0\end{pmatrix}. \] The factors are correlated, so $\Phi$ is not the identity. Here $\Sigma=\Phi+I=\begin{pmatrix}2&1/2\\1/2&2\end{pmatrix}$, and \[ \Sigma^{-1}=\frac1{15}\begin{pmatrix}8&-2\\-2&8\end{pmatrix}, \] since the determinant of $\Sigma$ is $4-1/4=15/4$.

The regression score is $\Phi\Sigma^{-1}x$. Multiplying the matrices gives \[ \Phi\Sigma^{-1}=\frac1{15}\begin{pmatrix}7&2\\2&7\end{pmatrix}, \qquad \hat f_R=\begin{pmatrix}14/15\\4/15\end{pmatrix}. \] For Bartlett scoring, $\Lambda=I$ makes the weighted least-squares objective $(x-g)'(x-g)$, whose unique minimizer is $g=x$. Thus $\hat f_B=(2,0)'$. This example shows that oblique factor correlations enter the regression score directly, while Bartlett scoring uses the loading and uniqueness matrices in its residual-fitting objective.

Objectives and Practical Differences

Regression and Bartlett scores answer different questions. Regression estimates the conditional mean of the latent factors given the response under the specified Gaussian model. It is a predictor that uses the assumed distribution of the factors as well as the measurement model. In particular, conditional expectation minimizes expected squared prediction error among predictors with finite second moments: if $m(x)=E[f\mid x]$ and $a(x)$ is another such predictor, then

E[\|f-a(x)\|^2] =E[\|f-m(x)\|^2]+E[\|m(x)-a(x)\|^2].

To see this, write $f-a=(f-m)+(m-a)$, expand the squared norm, and condition the cross term on $x$. It vanishes because $E[f-m\mid x]=0$. The remaining terms give the displayed identity. Thus the regression score is optimal for this prediction criterion under the model.

Bartlett scoring instead focuses on reproducing the observed response through $\Lambda g$, penalizing residuals more when their uniqueness variance is small. Its conditional unbiasedness is a different property from minimizing expected squared prediction error. The contrast is especially transparent for orthogonal factors, where $\Phi=I$: regression still incorporates the factor distribution and typically shrinks scores, whereas Bartlett estimates solve the weighted residual-fitting problem without that same predictive adjustment.

Worked Example: Unequal Uniquenesses for One Factor

Consider one factor with loading vector $\Lambda=(1,1,1)'$, factor variance $\Phi=1$, uniqueness matrix $\Psi=\operatorname{diag}(1,2,1)$, and response $x=(2,1,0)'$. The uniqueness weights are $\Psi^{-1}=\operatorname{diag}(1,1/2,1)$.

For Bartlett scoring, the denominator is \[ \Lambda'\Psi^{-1}\Lambda=1+\frac12+1=\frac52, \] and the numerator is \[ \Lambda'\Psi^{-1}x=2+\frac12+0=\frac52. \] Therefore $\hat f_B=(5/2)/(5/2)=1$.

For regression scoring, the equivalent precision-form expression has denominator $\Phi^{-1}+\Lambda'\Psi^{-1}\Lambda=1+5/2=7/2$, with the same numerator $5/2$. Hence $\hat f_R=(5/2)/(7/2)=5/7$. The second response contributes half as much as the first or third in the weighted residual calculation because its uniqueness variance is twice as large. The additional factor-variance term in the regression denominator gives a different estimate.

Common Pitfall

A frequent mistake is to treat the two formulas as interchangeable because both are linear functions of $x$. They are not. Bartlett scoring uses $(\Lambda'\Psi^{-1}\Lambda)^{-1}\Lambda'\Psi^{-1}x$; regression scoring uses the factor distribution as well, through $\Phi$, or equivalently through $\Phi^{-1}+\Lambda'\Psi^{-1}\Lambda$ in the precision form. Replacing one matrix by the other without checking the scoring objective can change the answer substantially. A second calculation pitfall is to multiply matrix entries without checking signs: in the first worked example, the off-diagonal entries of $\Lambda'\Lambda$ are negative, so its inverse has positive off-diagonal entries. Verifying the product against the original normal equations catches such errors.

Key takeaway: Regression scores are model-based conditional means under Gaussian assumptions; Bartlett scores are uniqueness-weighted least-squares fits and are conditionally unbiased under the stated model conditions. State the chosen objective and use the fitted $\Phi$, $\Lambda$, and $\Psi$ consistently.

Check Your Understanding

Use the definitions and derivations above to answer these questions.

Which Gaussian-model assumption allows the regression score to be interpreted as $E[f\mid x]$?
What full-rank condition ensures the Bartlett estimate is unique, and why does it make the objective strictly convex?
In the oblique worked example, why does the regression score depend on the off-diagonal entries of $\Phi$?
For Bartlett scoring, explain why a variable with larger uniqueness variance receives less weight in the residual objective.
Does conditional unbiasedness imply that a Bartlett score equals the latent factor for every observation? Explain.