Diagonal Specific Covariance and Its Implications

Why the Specific Covariance Assumption Matters

A common-factor model separates variation in observed variables into a part associated with shared factors and a specific part. The diagonal specific-covariance assumption says that different variables’ specific components have zero covariance. This is an important restriction: it limits which covariances can be attributed to the specific components. But it is not, by itself, a claim that the specific components are independent, nor does it automatically settle what happens after conditioning on the factors.

Those distinctions matter in interpretation. A covariance model describes second moments; independence is a statement about the full joint distribution. Conditional dependence is a further question about the distribution of residual variation at fixed factor values. We will keep these three claims separate.

Model and Assumptions

Definition: In the common-factor model, an observed random vector $X$ of dimension $p$ is represented as $$ X=\mu+\Lambda F+\varepsilon, $$ where $\mu=E[X]$, $F$ is a $q$-dimensional vector of common factors, $\Lambda$ is a $p\times q$ loading matrix, and $\varepsilon$ is a $p$-dimensional vector of specific errors. The standard covariance assumptions are $E[F]=0$, $E[\varepsilon]=0$, $\operatorname{Cov}(F)=\Phi$, $\operatorname{Cov}(\varepsilon)=\Psi$, and $\operatorname{Cov}(F,\varepsilon)=0$. The diagonal specific-covariance assumption is $$ \Psi=\operatorname{diag}(\psi_1,\ldots,\psi_p). $$ All variables are assumed to have finite second moments.

The notation $\operatorname{Cov}(F,\varepsilon)=0$ means that every factor-specific cross-covariance is zero. The diagonal condition means that $\operatorname{Cov}(\varepsilon_i,\varepsilon_j)=0$ for distinct observed-variable indices $i$ and $j$. Since $\Psi$ is a covariance matrix, each $\psi_i$ is nonnegative. These assumptions do not require $\Phi$ to be nonsingular.

What Diagonality Implies for the Observed Covariance

Theorem: Under the common-factor covariance assumptions, $$ \operatorname{Cov}(X)=\Lambda\Phi\Lambda^{\mathsf T}+\Psi. $$ If $\Psi$ is diagonal, then for distinct $i,j$, $$ \operatorname{Cov}(X_i,X_j)=\lambda_i^{\mathsf T}\Phi\lambda_j, $$ where $\lambda_i^{\mathsf T}$ is row $i$ of $\Lambda$. Thus, all off-diagonal covariances in the model come from the common-factor part.

Proof. Since $E[F]=E[\varepsilon]=0$, the representation gives $E[X]=\mu$. Subtracting the mean, $X-E[X]=\Lambda F+\varepsilon$. Expand its covariance:

\begin{aligned} \operatorname{Cov}(X) &=E\big[(\Lambda F+\varepsilon)(\Lambda F+\varepsilon)^{\mathsf T}\big]\\ &=\Lambda E[FF^{\mathsf T}]\Lambda^{\mathsf T} +\Lambda E[F\varepsilon^{\mathsf T}] +E[\varepsilon F^{\mathsf T}]\Lambda^{\mathsf T} +E[\varepsilon\varepsilon^{\mathsf T}]\\ &=\Lambda\Phi\Lambda^{\mathsf T}+\Psi. \end{aligned}

The two cross terms vanish because $\operatorname{Cov}(F,\varepsilon)=0$ and both vectors have mean zero. In the $(i,j)$ entry, the specific contribution is $\Psi_{ij}$, which is zero when $i\ne j$. This proves both claims. $\square$

For the diagonal entries, $\operatorname{Var}(X_i)=\lambda_i^{\mathsf T}\Phi\lambda_i+\psi_i$. Thus the specific variance contributes to each variable’s own variance, while the diagonal-specific assumption excludes a specific contribution to covariance between different observed variables. It does not say that the observed variables are uncorrelated: their common-factor contribution can be nonzero.

Worked Example: Computing the Covariance Matrix

Let there be one factor with variance $1.5$, loading vector $\lambda=(1,2,-1)^{\mathsf T}$, and specific covariance matrix $\Psi=\operatorname{diag}(0.25,1,0.5)$. Find the covariance matrix of the three observed variables.

Here $\Phi=(1.5)$, so $\Lambda\Phi\Lambda^{\mathsf T}=1.5\lambda\lambda^{\mathsf T}$. First,

\lambda\lambda^{\mathsf T} = \begin{pmatrix} 1&2&-1\\ 2&4&-2\\ -1&-2&1 \end{pmatrix}, \qquad 1.5\lambda\lambda^{\mathsf T} = \begin{pmatrix} 1.5&3&-1.5\\ 3&6&-3\\ -1.5&-3&1.5 \end{pmatrix}.

Adding the specific covariance matrix gives

\operatorname{Cov}(X)= \begin{pmatrix} 1.75&3&-1.5\\ 3&7&-3\\ -1.5&-3&2 \end{pmatrix}.

For example, the covariance between variables 1 and 2 is $1\cdot 1.5\cdot 2=3$; there is no specific contribution to that off-diagonal entry. The second variable’s variance is $2^2(1.5)+1=7$, including its specific variance of 1.

Uncorrelated Does Not Mean Independent

A diagonal $\Psi$ establishes that the specific errors are pairwise uncorrelated. Independence is stronger: it rules out all forms of distributional dependence, not only linear association measured by covariance. Without additional assumptions on their joint distribution, zero covariance does not imply independence.

Worked Example: Dependent Errors With Diagonal Covariance

Let $U$ take values $-1,0,1$, each with probability $1/3$, and define $\varepsilon_1=U$ and $\varepsilon_2=U^2-2/3$. Verify that the errors are uncorrelated but dependent.

The means are $E[\varepsilon_1]=(-1+0+1)/3=0$ and $E[\varepsilon_2]=(1-2/3+1-2/3+0-2/3)/3=0$. Their covariance is $E[\varepsilon_1\varepsilon_2]$, because both means are zero. The three products are $-1/3,0,1/3$, so $E[\varepsilon_1\varepsilon_2]=(-1/3+0+1/3)/3=0$.

Their variances are $E[U^2]=2/3$ and $\operatorname{Var}(U^2)=E[U^4]-(E[U^2])^2=2/3-4/9=2/9$. Therefore $\Psi=\operatorname{diag}(2/3,2/9)$. Yet $\varepsilon_2$ is determined exactly by $\varepsilon_1$: it equals $1/3$ when $\varepsilon_1=\pm1$, and $-2/3$ when $\varepsilon_1=0$. The two errors are dependent despite having zero covariance.

This example isolates the key distinction: diagonal specific covariance is a second-moment restriction. It is not a general assertion that the specific errors have a product distribution.

Conditional Covariance and Conditional Independence

The phrase “given the factors” refers to the conditional distribution of $X$ at specified values of $F$. From the model, $X=\mu+\Lambda F+\varepsilon$, so the conditional covariance of $X$ given $F$ equals the conditional covariance of $\varepsilon$ given $F$. It does not follow from $\operatorname{Cov}(\varepsilon)=\Psi$ alone that this conditional covariance is $\Psi$. The unconditional covariance and the conditional covariance are different quantities.

Result: If the error vector $\varepsilon$ is independent of $F$, has mean zero, and has covariance $\Psi$, then $$ E[X\mid F]=\mu+\Lambda F,\qquad \operatorname{Cov}(X\mid F)=\Psi \quad\text{almost surely}. $$ Consequently, diagonal $\Psi$ implies that distinct observed variables are conditionally uncorrelated given $F$. It does not, by itself, imply their conditional independence.

To verify the result, independence makes the conditional distribution of $\varepsilon$ given $F$ equal to its marginal distribution almost surely. Hence $E[\varepsilon\mid F]=0$ and $\operatorname{Cov}(\varepsilon\mid F)=\Psi$. Adding the $F$-measurable vector $\mu+\Lambda F$ changes the conditional mean but not the conditional covariance, giving the displayed formulas. Conditional uncorrelatedness follows because the off-diagonal entries of $\Psi$ vanish.

The additional independence assumption in this result matters. The original covariance assumptions only make $F$ and $\varepsilon$ uncorrelated. They do not generally make them independent, or ensure that conditional error covariances are constant.

Worked Example: Uncorrelated Errors That Are Dependent Given the Factor

Let $F$ and $U$ be independent random variables taking values $-1$ and $1$ with equal probabilities. Set $\varepsilon_1=U$ and $\varepsilon_2=FU$. Check the covariance assumptions and then find the conditional covariance of the errors given $F$.

Both errors have mean zero and variance one. Their covariance is $E[\varepsilon_1\varepsilon_2]=E[FU^2]=E[F]=0$, so $\Psi=I_2$. Also, $E[F\varepsilon_1]=E[FU]=0$, and $E[F\varepsilon_2]=E[F^2U]=E[U]=0$. Thus the factor and error vector are uncorrelated, and the unconditional specific covariance is diagonal.

For a fixed value of $F$, both conditional error means are zero, while \[ E[\varepsilon_1\varepsilon_2\mid F] =E[FU^2\mid F]=F. \] Therefore $\operatorname{Cov}(\varepsilon_1,\varepsilon_2\mid F)=F$, which is either $1$ or $-1$, not zero. In fact, given $F$, $\varepsilon_2=F\varepsilon_1$, so the errors are conditionally completely dependent. This example shows why zero unconditional cross-covariances do not establish conditional uncorrelatedness.

What the Gaussian Assumption Adds

A useful stronger conclusion is available when the factor and error blocks are jointly Gaussian. The independence result below does not require a nonsingular covariance matrix. In particular, it remains valid if a factor is constant or if any covariance matrix involved is singular.

Theorem: If $(F^{\mathsf T},\varepsilon^{\mathsf T})^{\mathsf T}$ is jointly Gaussian and $\operatorname{Cov}(F,\varepsilon)=0$, then $F$ and $\varepsilon$ are independent. If also $\operatorname{Cov}(\varepsilon)=\Psi$ is diagonal, then the components of $\varepsilon$ are mutually independent. Thus, in this jointly Gaussian model, the observed variables are conditionally independent given $F$.

Proof. Write $t$ and $s$ for vectors of the dimensions of $F$ and $\varepsilon$. The characteristic function of a jointly Gaussian vector with mean zero and covariance matrix $V$ is $\exp(-u^{\mathsf T}Vu/2)$. This formula holds for positive semidefinite $V$, including singular $V$. Here the zero cross-covariance gives the block covariance matrix with diagonal blocks $\Phi$ and $\Psi$, so the joint characteristic function is

\begin{aligned} E\!\left[\exp\{i(t^{\mathsf T}F+s^{\mathsf T}\varepsilon)\}\right] &=\exp\!\left\{-\frac12(t^{\mathsf T}\Phi t+s^{\mathsf T}\Psi s)\right\}\\ &=\exp\!\left(-\frac12t^{\mathsf T}\Phi t\right) \exp\!\left(-\frac12s^{\mathsf T}\Psi s\right). \end{aligned}

The two factors on the last line are the marginal characteristic functions of $F$ and $\varepsilon$. Factorization of the joint characteristic function therefore proves their independence, including in the degenerate case. Applying the same argument to the Gaussian vector $\varepsilon$, a diagonal $\Psi$ makes its joint characteristic function factor into the product of its coordinate characteristic functions. Hence its components are mutually independent. Given $F$, each $X_i$ is a fixed shift of $\varepsilon_i$; independent errors therefore give conditional independence of the observed variables. $\square$

A common sufficient condition for the theorem is that the full vector $(F,\varepsilon)$ is jointly Gaussian. Merely knowing that each block is Gaussian separately is not enough to conclude joint Gaussianity or independence. Likewise, without joint Gaussianity, zero cross-covariance does not establish independence.

Common Pitfall

Incorrect reasoning: “The specific covariance matrix is diagonal, so the specific errors are independent, and the observed variables must be conditionally independent given the factors.”
Correction: Diagonal covariance gives pairwise uncorrelated errors. Conditional uncorrelatedness given factors requires more, such as independence of the error vector from the factors. Conditional independence requires still more, such as mutually independent errors; joint Gaussianity together with the stated zero-covariance conditions is one setting where these stronger conclusions hold.

Another tempting shortcut is to infer independence by writing down a joint density and factoring it. That argument requires the relevant distributions to have densities, which a Gaussian vector need not have when its covariance is singular. The characteristic-function proof above avoids that gap and covers degenerate Gaussian factors as well.

Why the Distinction Matters

The diagonal-specific assumption gives the covariance model a clear interpretation: cross-variable covariances must be represented by the common factors, while the specific terms contribute only to marginal variances. That statement is valid at the level of second moments under the stated assumptions. Predictions about conditional behavior or independence require distributional assumptions beyond covariance structure.

When interpreting a fitted factor model, state precisely which claim is being made. “The specific errors are uncorrelated” follows from diagonal $\Psi$. “The specific errors are independent” does not follow without more assumptions. “Observed variables are conditionally independent given the factors” requires both a suitable relationship between factors and errors and sufficient assumptions about the joint distribution of the errors.

Check Your Understanding

Use the assumptions carefully; distinguish covariance claims from distributional claims.

Under the common-factor covariance assumptions, derive the expression for $\operatorname{Cov}(X_i,X_j)$ when $i\ne j$.
Does a diagonal $\Psi$ imply that the specific errors are independent? Explain what it does imply.
What additional assumption ensures that $\operatorname{Cov}(X\mid F)=\Psi$ almost surely?
Under joint Gaussianity and zero factor-error cross-covariance, why does independence still follow if $\Phi$ is singular?
Can a diagonal unconditional specific covariance coexist with nonzero conditional covariance given $F$? Describe the mechanism.