Gaussian Laws Given Linear Observations

From a Gaussian Vector to an Exact Linear Observation

A linear observation may reveal only a few combinations of the coordinates of a random vector. For example, a study might record the sum of two measurements rather than each measurement separately. Conditioning on that sum does not usually determine a unique vector; instead, it gives a Gaussian distribution over the vectors consistent with the observed value. The key calculations are the conditional mean and covariance, and the key geometric fact is that the conditional distribution lies on the observation constraint.

Let \(X\) be a \(p\)-dimensional Gaussian vector with mean \(\mu\) and positive-definite covariance matrix \(\Sigma\). Let \(A\) be an \(m\times p\) matrix of full row rank, where \(m\leq p\), and define the observed vector \(Y=AX\). Positive definiteness of \(\Sigma\) and full row rank of \(A\) ensure that \(Y\) has a nonsingular covariance matrix. These conditions allow the conditional law to be written using an ordinary matrix inverse.

Write \(M=A\Sigma A^T\). This is the covariance matrix of \(Y\), and it is invertible under the stated assumptions: for every nonzero \(u\in\mathbb{R}^m\), \(A^Tu\neq 0\), so \(u^TMu=(A^Tu)^T\Sigma(A^Tu)>0\). We can therefore define the gain matrix \(K=\Sigma A^TM^{-1}\). The conditional formula adjusts the original mean by the discrepancy between the observed value and its expectation, multiplied by this matrix.

Why the Conditional Formula Holds

The proof separates the part of \(X\) predicted linearly from \(Y\) from a residual that is independent of \(Y\). Define \(K=\Sigma A^TM^{-1}\), where \(M=A\Sigma A^T\), and let \(R=X-\mu-K(Y-A\mu)\). Both \(R\) and \(Y\) are affine transformations of a Gaussian vector, so their joint distribution is Gaussian. Their covariance is zero:

For completeness, zero cross-covariance implies independence here because the pair is jointly Gaussian. In fact, the characteristic function of any jointly Gaussian pair \((U,V)\), with means \(a,b\) and covariance blocks \(C_U,C_V,C_{UV}\), has exponent \(i(t^Ta+s^Tb)-\tfrac12(t^TC_Ut+2t^TC_{UV}s+s^TC_Vs)\). When \(C_{UV}=0\), this characteristic function factors into the characteristic functions of \(U\) and \(V\). The factorization proves independence. Applying this fact gives \(R\) independent of \(Y\).

Since \(X=\mu+K(Y-A\mu)+R\), conditioning on \(Y=y\) leaves only the residual \(R\) random. Its mean is zero, and its covariance is \[ \operatorname{Var}(R) =\Sigma-\Sigma A^TM^{-1}A\Sigma. \] To check this covariance directly, set \(D=K A\). Then \(R=(I-D)(X-\mu)\), so its covariance is \((I-D)\Sigma(I-D)^T\). Expanding and using \(K=\Sigma A^TM^{-1}\), \(M=A\Sigma A^T\), and the symmetry of \(M^{-1}\) gives \(\Sigma-\Sigma A^TM^{-1}A\Sigma\). Thus the conditional mean is \(\mu+K(y-A\mu)\) and the conditional covariance is the displayed residual covariance.

The formula specifies a conditional distribution even though each event \(Y=y\) has probability zero when \(Y\) has a continuous density. More precisely, it defines a regular conditional distribution: for each \(y\), take the stated Gaussian law with its specified mean and covariance. The identity \(X=\mu+K(Y-A\mu)+R\), with \(R\) independent of \(Y\), verifies that this kernel gives the conditional law.

Geometry and Checks on the Result

The conditional covariance is necessarily singular when the observation leaves fewer than \(p\) free directions. Indeed, the conditional random vector must satisfy \(AX=y\). Its mean satisfies this constraint because \[ A\{\mu+\Sigma A^TM^{-1}(y-A\mu)\} =A\mu+MM^{-1}(y-A\mu)=y. \] Its covariance \(S=\Sigma-\Sigma A^TM^{-1}A\Sigma\) obeys \(AS=0\), as follows by multiplying out and using \(M=A\Sigma A^T\). Thus any variation in the conditional vector is in the null space of \(A\).

The dimension of that residual variation can also be seen by whitening. Let \(\Sigma^{1/2}\) be the symmetric positive-definite square root of \(\Sigma\), and set \(B=A\Sigma^{1/2}\). Then \[ S=\Sigma^{1/2}\left[I-B^T(BB^T)^{-1}B\right]\Sigma^{1/2}. \] The matrix \(B^T(BB^T)^{-1}B\) is symmetric and idempotent, so it is the orthogonal projection onto the row space of \(B\). The bracketed matrix projects onto its orthogonal complement, which has dimension \(p-m\), since \(B\) has row rank \(m\). Consequently \(S\) is positive semidefinite of rank \(p-m\). This describes the remaining degrees of freedom after the \(m\) independent constraints are imposed.

Common Mistakes and Practical Checks

A frequent error is to treat conditioning on \(AX=y\) as though the conditional vector retained the original covariance \(\Sigma\). It does not: the observation fixes \(m\) linear combinations, and the conditional covariance must have zero variation in each observed direction. Another error is to use \(A\Sigma A^T\) without checking that it is invertible. Full row rank of \(A\) and positive definiteness of \(\Sigma\) guarantee invertibility here; if either condition is relaxed, the ordinary inverse formula may not apply.

A reliable calculation has three checks. First, the conditional mean must obey \(A E[X\mid Y=y]=y\). Second, the covariance must satisfy \(A S=0\). Third, for the assumptions in this tutorial, its rank must be \(p-m\). These are not substitutes for the derivation, but they quickly expose many matrix multiplication and sign errors.