Deriving Conditional Factor Score Uncertainty

Why a Factor Score Needs an Uncertainty Measure

A regression factor score assigns a value to each latent factor based on the observed variables. But the same observed vector can be compatible with multiple factor values: measurement error and overlap among factor effects leave uncertainty even after the data are observed. A score without that uncertainty measure can look more definitive than the model warrants.

Under a Gaussian factor model, the conditional distribution of the factors given the observations is available in closed form. Its mean is the regression factor score, and its covariance quantifies the remaining uncertainty. The formulas apply to orthogonal and oblique factors, provided the factor covariance is handled consistently.

Gaussian Factor-Model Setup

Let \(X\) be a \(p\)-dimensional observed random vector and \(F\) a \(q\)-dimensional latent factor vector. Assume the model

where \(\mu\) is a \(p\)-vector, \(\Lambda\) is a \(p\times q\) loading matrix, \(F\sim N(0,\Phi)\), and \(\varepsilon\sim N(0,\Psi)\). Assume \(F\) and \(\varepsilon\) are independent, and that \(\Phi\) and \(\Psi\) are positive definite. In the usual factor model, \(\Psi\) is diagonal with positive uniqueness variances. These assumptions make the observed covariance \(\Sigma=\Lambda\Phi\Lambda'+\Psi\) positive definite as well. Throughout, model parameters are treated as known.

The positive-definiteness assumptions are useful both mathematically and practically: they ensure the inverses in the formulas below exist. The derivation uses the full covariance \(\Phi\); setting \(\Phi=I\) is appropriate only for an orthogonal, unit-variance factor parameterization.

Conditional Mean and Covariance

Write \(y=x-\mu\) for the centered observed vector. The joint distribution of \(F\) and \(X\) is Gaussian, with \(E[F]=0\), \(E[X]=\mu\), \(\operatorname{Cov}(X)=\Sigma\), and \(\operatorname{Cov}(F,X)=\Phi\Lambda'\). Conditioning a jointly Gaussian vector therefore gives the following result.

The same conditional covariance has a useful precision-form expression:

Here \(C\) does not depend on \(x\). Different observations can produce different conditional means, but under a fixed model they share the same conditional covariance.

Deriving the Conditional Distribution

A direct derivation makes clear why the precision form involves both the prior factor covariance and the measurement error covariance. For a realized factor vector \(f\), the density of \(F\) is proportional to \(\exp(-\tfrac12 f'\Phi^{-1}f)\). Given \(F=f\), the observed centered vector \(y\) has density proportional to \(\exp(-\tfrac12(y-\Lambda f)'\Psi^{-1}(y-\Lambda f))\). Independence and Bayes’ rule imply that the conditional density of \(F\) given \(X=x\), as a function of \(f\), is proportional to their product.

For completeness, these precision-form expressions equal the covariance-form expressions in the theorem. The matrix identity \((\Phi^{-1}+\Lambda'\Psi^{-1}\Lambda)\Phi\Lambda' =\Lambda'\Psi^{-1}(\Psi+\Lambda\Phi\Lambda')\) shows, after multiplying by the appropriate inverses, that \(C\Lambda'\Psi^{-1}=\Phi\Lambda'\Sigma^{-1}\). Also, \(C=\Phi-\Phi\Lambda'\Sigma^{-1}\Lambda\Phi\): multiplying the right-hand side by \(A\) gives \(I\), using \(\Sigma=\Psi+\Lambda\Phi\Lambda'\), so it is \(A^{-1}\). This proves both expressions for the conditional mean and covariance. \(\square\)

Worked Calculations

Why Conditional Uncertainty Cannot Exceed Prior Uncertainty

The conditional covariance has a precise comparison with the prior covariance. For symmetric matrices, \(B\) is positive semidefinite if \(v'Bv\geq 0\) for every vector \(v\). Write \(C\preceq\Phi\) when \(\Phi-C\) is positive semidefinite.

Proof. From the covariance form,

For any \(q\)-vector \(v\), \(v'(\Phi-C)v=(\Lambda\Phi v)'\Sigma^{-1}(\Lambda\Phi v)\geq 0\), because \(\Sigma\) is positive definite. Thus \(\Phi-C\) is positive semidefinite. Equivalently, \(\operatorname{Var}(v'F\mid X=x)=v'Cv\leq v'\Phi v=\operatorname{Var}(v'F)\). \(\square\)

This inequality does not say that every factor’s uncertainty must decrease strictly. If some factor direction is not informed by the observed variables, the reduction in that direction can be zero. The theorem also does not include uncertainty from estimating \(\mu\), \(\Lambda\), \(\Phi\), or \(\Psi\): those quantities were treated as fixed.

Reading and Reporting the Uncertainty

The diagonal entry \(C_{jj}\) is the conditional variance of factor \(j\); \(\sqrt{C_{jj}}\) is its conditional standard deviation. An off-diagonal entry \(C_{jk}\) is the conditional covariance between factors \(j\) and \(k\). It can matter when interpreting a joint score or a linear combination of factors, even when the marginal standard deviations are the main reporting focus.

A common mistake is to report the regression score as if it were the known value of the latent factor. The correction is to distinguish its conditional mean from the factor itself and, when uncertainty matters, report the corresponding conditional variance or covariance. A second mistake is to assume that the conditional covariance changes with the observed vector: in this fixed-parameter Gaussian model, the data value changes the conditional mean but not \(C\). A third is to use \(\Phi=I\) automatically; oblique solutions require the actual factor covariance matrix.