Diagnosing Gaussian Independence and Conditioning Errors

Why Gaussian Conditioning Errors Are Easy to Make

Gaussian calculations often have compact formulas, but a correct-looking formula can answer the wrong question. Zero covariance is evidence of independence only under the joint Gaussian assumption; block independence requires every cross-covariance entry to vanish; and conditioning changes covariance. Singular covariance adds another issue: a conditioning value may be possible only on a lower-dimensional support.

A useful diagnostic habit is to name the random vectors being compared, state which variables are conditioned on, and check the relevant covariance matrix before simplifying. The examples below use these checks to expose errors rather than merely recite formulas.

Zero Covariance and Block Independence

Theorem: If $X$ and $Y$ are jointly Gaussian random vectors, then $X$ and $Y$ are independent if and only if $\operatorname{Cov}(X,Y)=0$, where zero means that every entry of the cross-covariance matrix is zero. Joint Gaussianity is essential; zero covariance alone does not imply independence for arbitrary random variables.

To prove the Gaussian statement, let the joint mean be $(\mu_X,\mu_Y)$, and let the covariance have diagonal blocks $\Sigma_{XX}$ and $\Sigma_{YY}$, with cross block $\Sigma_{XY}$. The characteristic function of a jointly Gaussian vector at arguments $s,t$ is

\varphi_{X,Y}(s,t) =\exp\!\left(i s^{\mathsf T}\mu_X+i t^{\mathsf T}\mu_Y -\frac12\left(s^{\mathsf T}\Sigma_{XX}s +2s^{\mathsf T}\Sigma_{XY}t +t^{\mathsf T}\Sigma_{YY}t\right)\right).

If $\Sigma_{XY}=0$, this expression factors as $\varphi_X(s)\varphi_Y(t)$. Factorization of the joint characteristic function into the two marginal characteristic functions is equivalent to independence. Conversely, independence implies zero covariance whenever the covariances exist, as they do for Gaussian vectors: $E[(X-\mu_X)(Y-\mu_Y)^{\mathsf T}]$ factors into the product of two zero means. This proves both directions, including for singular Gaussian distributions. $\square$

For blocks, a single zero covariance between a selected pair of coordinates is not enough. If $X=(X_1,X_2)^{\mathsf T}$ and $Y=(Y_1,Y_2)^{\mathsf T}$, their cross-covariance block has four entries. The theorem applies to the blocks only when the whole matrix is zero.

Worked Example: Zero Covariance Without Independence

Let $U$ be uniform on $(-1,1)$, and set $V=U^2$. Calculate their covariance and decide whether they are independent.

Symmetry gives $E[U]=0$ and $E[U^3]=0$, while $E[U^2]=1/3$. Therefore

\operatorname{Cov}(U,V) =E[UV]-E[U]E[V] =E[U^3]-0\cdot \frac13 =0.

Nevertheless, $V$ is determined exactly by $U$: given $U=u$, the conditional value of $V$ is $u^2$, whereas $V$ is not constant marginally. Thus $U$ and $V$ are dependent. The Gaussian theorem cannot be used here because $(U,V)$ is not jointly Gaussian. The diagnostic correction is not to reject the covariance calculation, but to reject the unsupported inference from zero covariance to independence.

Worked Example: A Cross-Covariance Block Is Not Zero

Suppose $(X_1,X_2,Y)$ is jointly Gaussian with covariance matrix

\operatorname{Var}(X_1,X_2,Y) = \begin{pmatrix} 1&0&0.3\\ 0&1&0\\ 0.3&0&1 \end{pmatrix}.

This is a valid covariance matrix: its eigenvalues are $0.7,1,1.3$, all positive. For $X=(X_1,X_2)^{\mathsf T}$, the cross-covariance with $Y$ is

\operatorname{Cov}(X,Y) = \begin{pmatrix}0.3\\0\end{pmatrix}.

Although $\operatorname{Cov}(X_2,Y)=0$, the block is not zero because $\operatorname{Cov}(X_1,Y)=0.3$. Hence $X$ and $Y$ are not independent. Checking just one coordinate pair would produce a false block-independence conclusion.

Conditional Covariance Is Not Marginal Covariance

For a jointly Gaussian pair of vectors $X,Z$, write $A=\operatorname{Var}(X)$, $S=\operatorname{Var}(Z)$, and $C=\operatorname{Cov}(X,Z)$. When $S$ is invertible, the conditional covariance of $X$ given $Z$ is $A-CS^{-1}C^{\mathsf T}$, not $A$. The subtracted term measures the covariance explained by the linear prediction from $Z$.

Conditional Gaussian formula: If $(X,Z)$ is jointly Gaussian and $S=\operatorname{Var}(Z)$ is invertible, then $$ E[X\mid Z=z]=\mu_X+CS^{-1}(z-\mu_Z), \qquad \operatorname{Var}(X\mid Z=z)=A-CS^{-1}C^{\mathsf T}. $$ The conditional covariance does not depend on $z$.

Here is a derivation that also explains the singular case. If $S$ is invertible, define the residual $R=X-\mu_X-CS^{-1}(Z-\mu_Z)$. Its covariance with $Z$ is $C-CS^{-1}S=0$. Since $R$ and $Z$ are jointly Gaussian, the result just proved makes them independent. Consequently the conditional distribution of $X$ given $Z=z$ is the distribution of $\mu_X+CS^{-1}(z-\mu_Z)+R$. Direct covariance calculation gives $\operatorname{Var}(R)=A-CS^{-1}C^{\mathsf T}$, proving the formula. $\square$

Worked Example: Correcting a Conditional-Variance Calculation

Let $Z=(Z_1,Z_2)^{\mathsf T}$ have covariance matrix $S=4I_2$. Suppose $X$ is jointly Gaussian with $Z$, with $\operatorname{Var}(X)=6$ and $\operatorname{Cov}(X,Z)=(2,2)$. The joint covariance is valid: its Schur complement is $6-(2,2)(4I_2)^{-1}(2,2)^{\mathsf T}=4>0$.

The covariance reduction from conditioning on both coordinates is

(2,2)S^{-1}(2,2)^{\mathsf T} =(2,2) \begin{pmatrix}1/4&0\\0&1/4\end{pmatrix} \begin{pmatrix}2\\2\end{pmatrix} =(2,2)\begin{pmatrix}1/2\\1/2\end{pmatrix} =2.

Thus $\operatorname{Var}(X\mid Z)=6-2=4$, not $6-4$. For example, if both means are zero and the observed value is $z=(1,1)^{\mathsf T}$, then the conditional mean is

(2,2)S^{-1}\begin{pmatrix}1\\1\end{pmatrix} =(2,2)\begin{pmatrix}1/4\\1/4\end{pmatrix} =1.

The conditional law is therefore $X\mid Z=z\sim N(1,4)$. The marginal variance remains $6$; it is not the variance to report after observing $Z$.

Singular Covariance and Feasible Conditioning Values

When $S$ is singular, the inverse in the conditional formula does not exist. For a jointly Gaussian pair, the corresponding formula uses the Moore–Penrose pseudoinverse $S^+$. The conditioning vector $z$ must lie in the affine support $\mu_Z+\operatorname{Range}(S)$, which has probability one. Values outside that support have probability zero, so a conditional law there is not determined by the joint distribution alone.

Singular Gaussian conditioning: If $(X,Z)$ is jointly Gaussian, $S=\operatorname{Var}(Z)$ may be singular. For $z\in\mu_Z+\operatorname{Range}(S)$, a version of the conditional Gaussian law has mean $\mu_X+CS^+(z-\mu_Z)$ and covariance $A-CS^+C^{\mathsf T}$. This law is specified almost surely in $Z$.

To justify the formula, positive semidefiniteness of the joint covariance implies that $C$ vanishes on the null space of $S$; equivalently, the rows of $C$ lie in $\operatorname{Range}(S)$. Thus $CS^+S=C$. Set $R=X-\mu_X-CS^+(Z-\mu_Z)$. Then $\operatorname{Cov}(R,Z)=C-CS^+S=0$, so the jointly Gaussian vectors $R,Z$ are independent. Also $S^+SS^+=S^+$, giving $\operatorname{Var}(R)=A-CS^+C^{\mathsf T}$. On the support of $Z$, the residual decomposition gives the stated conditional mean and covariance. $\square$

Worked Example: Conditioning on a Singular Gaussian Vector

Let $U,E$ be independent $N(0,1)$ variables, set $Z_1=U,\ Z_2=2U$, and let $X=3U+E$. Then $Z$ is supported on the line $z_2=2z_1$, with

S=\operatorname{Var}(Z)= \begin{pmatrix}1&2\\2&4\end{pmatrix}, \qquad C=\operatorname{Cov}(X,Z)=(3,6), \qquad A=\operatorname{Var}(X)=10.

The matrix $S$ has rank one and $S^+=S/25$. The covariance reduction is

CS^+C^{\mathsf T} =(3,6)\frac1{25} \begin{pmatrix}1&2\\2&4\end{pmatrix} \begin{pmatrix}3\\6\end{pmatrix} =(3,6)\begin{pmatrix}3/5\\6/5\end{pmatrix} =9.

Hence the conditional variance is $10-9=1$. For a feasible observed value $z=(1,2)^{\mathsf T}$, the conditional mean is $CS^+z=3$, so $X\mid Z=(1,2)^{\mathsf T}\sim N(3,1)$. A value such as $(1,1)^{\mathsf T}$ is off the support and cannot be treated as an ordinary possible observation of this $Z$.

Conditioning Sequentially Without Changing the Question

Conditioning first on one variable and then on another is valid when each update uses the distribution already conditioned on the information available. A common error is to take a conditional distribution after the first observation but then use a second update formula that still uses marginal covariances. The following calculation shows how the updated quantities fit together.

Worked Example: Two Sequential Gaussian Updates

Let $X,E_1,E_2$ be independent $N(0,1)$ variables and observe $Z_1=X+E_1$ and $Z_2=X+E_2$. First, $\operatorname{Var}(Z_1)=2$ and $\operatorname{Cov}(X,Z_1)=1$. Given $Z_1=1$,

E[X\mid Z_1=1]=\frac12, \qquad \operatorname{Var}(X\mid Z_1)=1-\frac{1^2}{2}=\frac12.

For the second update, compute covariances conditional on $Z_1$. Since $\operatorname{Cov}(X,Z_2)=1$, $\operatorname{Cov}(X,Z_1)=1$, $\operatorname{Cov}(Z_1,Z_2)=1$, and $\operatorname{Var}(Z_1)=2$,

\operatorname{Cov}(X,Z_2\mid Z_1)=1-\frac{1\cdot1}{2}=\frac12, \qquad \operatorname{Var}(Z_2\mid Z_1)=2-\frac{1^2}{2}=\frac32.

Also $E[Z_2\mid Z_1=1]=1/2$. On observing $Z_2=0$, the updated mean and variance are therefore

E[X\mid Z_1=1,Z_2=0] =\frac12+\frac{1/2}{3/2}\left(0-\frac12\right) =\frac13, \qquad \operatorname{Var}(X\mid Z_1,Z_2) =\frac12-\frac{(1/2)^2}{3/2} =\frac13.

As a check, the joint likelihood of the two observations gives posterior precision $1+1+1=3$ and posterior mean $(1+0)/3=1/3$. It agrees with the sequential result. The check would fail if the second update used marginal covariances while retaining the already-conditioned mean and variance.

Common Diagnostic Mistakes

Wrong: “The covariance is zero, so the variables are independent.” Correction: This conclusion requires joint Gaussianity (or another separately established independence condition); the uniform-square example shows why covariance alone is insufficient.
Wrong: “One coordinate in each block has zero covariance, so the blocks are independent.” Correction: Inspect the entire cross-covariance matrix. Every entry must be zero for jointly Gaussian blocks.
Wrong: “The covariance matrix is singular, so conditioning cannot be done.” Correction: Use the pseudoinverse formula and restrict the conditioning value to the affine support.
Wrong: “After observing $Z$, the uncertainty in $X$ is still its marginal variance.” Correction: Subtract the covariance reduction $CS^{-1}C^{\mathsf T}$, or $CS^+C^{\mathsf T}$ in the singular case.
Wrong: “Sequential updates can mix marginal and conditional quantities.” Correction: The second update must use covariances and means conditional on the information already observed.

Key takeaway: Before drawing an independence or conditioning conclusion, check the joint Gaussian assumption, the full cross-covariance block, the conditioning set, and whether the conditioned covariance is singular. Then use a covariance formula that matches those facts.

Check Your Understanding

For each question, identify the assumption or calculation that decides the answer.

Can zero covariance establish independence for arbitrary random variables? Give the relevant qualification.
If two jointly Gaussian vectors have a cross-covariance block with one nonzero entry, what can be concluded about their independence?
In the corrected variance example, why is the reduction from conditioning on both coordinates $2$, rather than $4$?
What must be true of an observed value $z$ when conditioning on a singular Gaussian vector $Z$?
In a sequential Gaussian update, which covariance quantities must be used for the second observation?