Independent-Innovation Representations of Gaussian Blocks

Why Independent Innovations Are Useful

A Gaussian block can often be understood as two components: the part that is linearly predictable from another block, and a remaining fluctuation that is independent of that block. Writing the vector this way makes the conditional distribution transparent and gives a useful construction for simulation and modeling. The key is not merely to subtract a fitted linear term: for Gaussian vectors, the resulting uncorrelated residual is independent.

We develop the representation for possibly degenerate Gaussian vectors. This includes the familiar case of an invertible conditioning covariance and also covers a conditioning block whose coordinates obey exact linear constraints.

Definition and General Representation

Definition: Let \(X\in\mathbb{R}^p\) and \(Y\in\mathbb{R}^q\) be jointly Gaussian, with means \(\mu_X,\mu_Y\) and covariance blocks \(\Sigma_{XX}=\operatorname{Var}(X)\), \(\Sigma_{YY}=\operatorname{Var}(Y)\), and \(\Sigma_{XY}=\operatorname{Cov}(X,Y)\). The innovation of \(X\) relative to \(Y\) is the residual obtained after subtracting the linear predictor based on \(Y\). When \(\Sigma_{YY}\) is singular, the standard coefficient is \(\Sigma_{XY}\Sigma_{YY}^{+}\), where \(A^{+}\) denotes the Moore–Penrose inverse of \(A\).

The Moore–Penrose inverse is the unique matrix satisfying \(AA^{+}A=A\), \(A^{+}AA^{+}=A^{+}\), and symmetry of \(AA^{+}\) and \(A^{+}A\). For a symmetric positive semidefinite matrix \(A\), \(AA^{+}=A^{+}A\) is the orthogonal projection onto the range of \(A\). If \(A\) is invertible, \(A^{+}=A^{-1}\).

Theorem: For jointly Gaussian \(X,Y\), define \[ B=\Sigma_{XY}\Sigma_{YY}^{+},\qquad \varepsilon=X-\mu_X-B(Y-\mu_Y). \] Then \[ X=\mu_X+B(Y-\mu_Y)+\varepsilon, \] where \(\varepsilon\) is Gaussian, independent of \(Y\), and has mean zero and covariance \[ \Sigma_{\varepsilon} =\Sigma_{XX}-\Sigma_{XY}\Sigma_{YY}^{+}\Sigma_{YX}. \] Consequently, for \(Y\)-almost every \(y\), \[ X\mid Y=y\ \sim\ N\!\left(\mu_X+B(y-\mu_Y),\ \Sigma_{\varepsilon}\right). \] When \(\Sigma_{YY}\) is invertible, its Moore–Penrose inverse in these formulas is the ordinary inverse.

For singular \(\Sigma_{\varepsilon}\), the displayed normal distribution is understood as a possibly degenerate Gaussian law. If \(\Sigma_{YY}\) is singular, \(Y\) is supported on a lower-dimensional affine set; the conditional-distribution statement is for almost every \(y\) under that law, not for arbitrary points outside its support.

Why the Innovation Is Independent

The central steps are to show that the residual has zero covariance with \(Y\), then use the Gaussian property to turn zero cross-covariance into independence. We verify the range condition needed when \(\Sigma_{YY}\) is singular, so the argument applies to both singular and nonsingular blocks.

Write \(A=\Sigma_{YY}\) and \(C=\Sigma_{XY}\). If \(a\) is in the null space of \(A\), then \(\operatorname{Var}(a^{\mathsf T}(Y-\mu_Y))=a^{\mathsf T}Aa=0\). A random variable with zero variance equals its mean almost surely, so \(a^{\mathsf T}(Y-\mu_Y)=0\) almost surely. Therefore \(\operatorname{Cov}(X,a^{\mathsf T}Y)=Ca=0\). Since \(A\) is symmetric, its range is the orthogonal complement of its null space. Thus each row of \(C\) lies in the range of \(A\), which gives \(CA^{+}A=C\).

Now \(B=CA^{+}\). The residual has zero mean, and its cross-covariance with \(Y\) is \[ \operatorname{Cov}(\varepsilon,Y) =C-BA =C-CA^{+}A =0. \] Its covariance is \[ \begin{aligned} \operatorname{Var}(\varepsilon) &=\Sigma_{XX}-BA B^{\mathsf T}\\ &=\Sigma_{XX}-CA^{+}A A^{+}C^{\mathsf T}\\ &=\Sigma_{XX}-CA^{+}C^{\mathsf T}, \end{aligned} \] using \(A^{+}AA^{+}=A^{+}\). This also shows directly that the stated residual covariance is positive semidefinite: it is the covariance of \(\varepsilon\).

The vector \((\varepsilon,Y)\) is jointly Gaussian because it is an affine transformation of \((X,Y)\). A jointly Gaussian vector with zero cross-covariance between its two blocks has a characteristic function that factors into the characteristic functions of those blocks: the block-diagonal covariance makes the Gaussian exponent a sum of two terms. Factorization of the joint characteristic function implies independence. Hence \(\varepsilon\) and \(Y\) are independent.

Finally, the representation writes \(X\) as a function of \(Y\) plus an independent Gaussian vector. Conditional on \(Y=y\), the first term is fixed and the innovation retains its unconditional law. This gives the stated conditional mean and covariance and completes the proof. \(\square\)

Worked Representations

Worked Example: A Scalar Response and a Scalar Conditioning Variable

Suppose \((X,Y)\) is jointly Gaussian with \(E[X]=1\), \(E[Y]=2\), \(\operatorname{Var}(X)=9\), \(\operatorname{Var}(Y)=4\), and \(\operatorname{Cov}(X,Y)=3\). Find its independent-innovation representation and the conditional distribution of \(X\) given \(Y=y\).

The coefficient of \(Y-2\) is \(\operatorname{Cov}(X,Y)/\operatorname{Var}(Y)=3/4\). The innovation is therefore \[ \varepsilon=X-1-\frac{3}{4}(Y-2). \] Its covariance with \(Y\) is \(3-(3/4)4=0\), so Gaussianity gives independence. Its variance is \[ \operatorname{Var}(\varepsilon) =9-\frac{3^2}{4} =9-\frac{9}{4} =\frac{27}{4}. \] Thus \[ X=1+\frac{3}{4}(Y-2)+\varepsilon,\qquad \varepsilon\sim N\!\left(0,\frac{27}{4}\right),\qquad \varepsilon\ \text{independent of }Y. \] In particular, \(X\mid Y=y\sim N(1+\frac{3}{4}(y-2),27/4)\). The conditional variance is smaller than the marginal variance \(9\), while the independent innovation accounts for the remaining uncertainty.

Worked Example: One Response With a Two-Dimensional Conditioning Block

Let \(X\) be scalar and \(Y=(Y_1,Y_2)^{\mathsf T}\). Suppose their means are \(\mu_X=0\), \(\mu_Y=(1,-1)^{\mathsf T}\), and \[ \Sigma_{XX}=5,\qquad \Sigma_{YY}=\begin{pmatrix}4&0\\0&1\end{pmatrix},\qquad \Sigma_{XY}=\begin{pmatrix}2&1\end{pmatrix}. \] Find the innovation representation and conditional law.

Since \(\Sigma_{YY}^{-1}=\begin{pmatrix}1/4&0\\0&1\end{pmatrix}\), the coefficient row is \[ B=\Sigma_{XY}\Sigma_{YY}^{-1} =\begin{pmatrix}2&1\end{pmatrix} \begin{pmatrix}1/4&0\\0&1\end{pmatrix} =\begin{pmatrix}1/2&1\end{pmatrix}. \] The residual variance is \[ \Sigma_{\varepsilon} =5-\begin{pmatrix}2&1\end{pmatrix} \begin{pmatrix}1/4&0\\0&1\end{pmatrix} \begin{pmatrix}2\\1\end{pmatrix} =5-\left(1+1\right) =3. \] Consequently, \[ X=\frac{1}{2}(Y_1-1)+(Y_2+1)+\varepsilon,\qquad \varepsilon\sim N(0,3),\qquad \varepsilon\ \text{independent of }Y. \] For example, at \(y=(3,0)^{\mathsf T}\), the conditional mean is \(\frac12(3-1)+(0+1)=2\). Hence \(X\mid Y=(3,0)^{\mathsf T}\sim N(2,3)\). The calculation uses both conditioning coordinates; omitting either would change the residual covariance and the conditional mean.

Worked Example: A Singular Conditioning Block

Let \(Z\sim N(0,1)\) and \(U\sim N(0,4)\) be independent, and define \(Y=(Z,2Z)^{\mathsf T}\) and \(X=3Z+U\). Find the representation using the Moore–Penrose inverse and interpret conditioning on \(Y\).

The covariance blocks are \[ \Sigma_{YY}=\begin{pmatrix}1&2\\2&4\end{pmatrix},\qquad \Sigma_{XY}=\begin{pmatrix}3&6\end{pmatrix},\qquad \Sigma_{XX}=13. \] The matrix \(\Sigma_{YY}\) has rank one, so it is singular. It equals \(vv^{\mathsf T}\), where \(v=(1,2)^{\mathsf T}\) and \(v^{\mathsf T}v=5\). Its Moore–Penrose inverse is \[ \Sigma_{YY}^{+}=\frac{1}{25} \begin{pmatrix}1&2\\2&4\end{pmatrix}, \] as can be checked by multiplying: \(\Sigma_{YY}\Sigma_{YY}^{+}\Sigma_{YY}=\Sigma_{YY}\), and the product \(\Sigma_{YY}\Sigma_{YY}^{+}\) is the symmetric projection \(vv^{\mathsf T}/5\).

The coefficient is \[ B=\begin{pmatrix}3&6\end{pmatrix}\Sigma_{YY}^{+} =\begin{pmatrix}3/5&6/5\end{pmatrix}. \] Because the means are zero, \(B Y=(3/5)Z+(12/5)Z=3Z\). The residual covariance is \[ 13-\begin{pmatrix}3&6\end{pmatrix}\Sigma_{YY}^{+} \begin{pmatrix}3\\6\end{pmatrix} =13-9=4. \] Thus \(X=BY+\varepsilon=3Z+U\), with \(\varepsilon=U\sim N(0,4)\) independent of \(Y\). Since \(Y=(y_1,y_2)^{\mathsf T}\) lies on \(y_2=2y_1\), conditioning gives \(X\mid Y=y\sim N(3y_1,4)\) for \(y\) on that support. The representation does not assign a meaningful conditional law at points off the support.

Common Pitfall and Practical Use

A common mistake is to treat zero covariance as sufficient for independence for any pair of random vectors. That implication is not generally valid. It is valid here because the residual and conditioning block are jointly Gaussian; without that hypothesis, the factorization argument for the Gaussian characteristic function does not apply.

Another error is to use an ordinary inverse when \(\Sigma_{YY}\) is singular. This fails even when the joint Gaussian model is perfectly well-defined. The Moore–Penrose inverse supplies the coefficient on the range of the conditioning covariance, while the data themselves remain restricted to the affine support of \(Y\).

The representation is useful whenever a model needs a conditional Gaussian draw: first generate \(Y\), then independently generate \(\varepsilon\) with covariance \(\Sigma_{\varepsilon}\), and combine them using the displayed affine formula. It also separates the predictable component from the remaining variation without requiring a new joint simulation at every conditioning value.

Key takeaway: A jointly Gaussian block always admits a linear-plus-independent-innovation representation. Its coefficient is \(\Sigma_{XY}\Sigma_{YY}^{+}\), and its innovation covariance is \(\Sigma_{XX}-\Sigma_{XY}\Sigma_{YY}^{+}\Sigma_{YX}\); the inverse reduces to the ordinary inverse when the conditioning covariance is nonsingular.

Check Your Understanding

Use the block representation and verify each covariance calculation.

In the scalar example, calculate \(\operatorname{Cov}(\varepsilon,Y)\) directly from the definition of \(\varepsilon\).
Why does the proof need joint Gaussianity to conclude that the innovation is independent of \(Y\)?
If \(\Sigma_{YY}\) is invertible, what does the Moore–Penrose inverse become, and how does the coefficient formula simplify?
In the singular example, what linear constraint must every possible observed value of \(Y\) satisfy?
Suppose the computed innovation covariance is negative in a proposed jointly Gaussian model. What does this tell you to check about the covariance blocks or calculations?