Tutorials › Statistical Theory I - Probability and Distribution Theory › Gaussian Conditioning After Affine Reparameterization

Home  /  Tutorials  /  Gaussian Conditioning and Linear Maps

Gaussian Conditioning and Linear Maps · Tutorial 738 of 994

Gaussian Conditioning After Affine Reparameterization

Learn how to transform a Gaussian conditional law correctly when either the response coordinates or the conditioning coordinates are changed.

Advanced 10 min read

What You'll Learn

  • State the conditional Gaussian law under invertible affine changes to both coordinate blocks.
  • Track how a response transformation changes conditional means and covariances.
  • Explain why an invertible change of conditioning coordinates preserves the information conditioned upon.
  • Verify transformed conditional laws using block covariance calculations.
  • Identify common errors involving inverse transformations, offsets, and covariance matrices.

Why Coordinate Changes Matter

A conditional Gaussian law is often easiest to calculate in one set of coordinates and easiest to interpret in another. For example, a response vector may be shifted and rescaled, or the observed conditioning variables may be reported in transformed units. The conditional distribution must then be translated consistently: the observed conditioning value changes coordinates, the conditional mean changes with the response coordinates, and the conditional covariance changes by a matrix congruence.

The key distinction is between changing the response and changing the conditioning variables. An invertible affine change to the response transforms the conditional distribution itself. An invertible affine change to the conditioning coordinates does not change the information being conditioned on; it only changes how the observed value is written.

Definition: Let \(X\in\mathbb{R}^p\) and \(Y\in\mathbb{R}^q\) be jointly Gaussian, with means \(\mu_X,\mu_Y\) and covariance blocks \(\Sigma_{XX},\Sigma_{XY},\Sigma_{YX},\Sigma_{YY}\). Suppose \(\Sigma_{YY}\) is positive definite. For invertible matrices \(A\in\mathbb{R}^{p\times p}\) and \(B\in\mathbb{R}^{q\times q}\), and fixed vectors \(a\in\mathbb{R}^p\) and \(b\in\mathbb{R}^q\), define \(U=a+AX\) and \(V=b+BY\). These are invertible affine reparameterizations of the response and conditioning coordinates.

The positive-definiteness assumption ensures that the conditioning vector has a nonsingular Gaussian density. The formulas below specify a version of the conditional Gaussian law for every conditioning value \(y\) (or \(v\)), not merely almost every value. The response covariance may be singular; its conditional law can then be degenerate.

The Transformation Formula

Write

$$ C=\Sigma_{XY}\Sigma_{YY}^{-1}, \qquad S=\Sigma_{XX}-\Sigma_{XY}\Sigma_{YY}^{-1}\Sigma_{YX}. $$

The block conditional Gaussian formula gives \(X\mid Y=y\) a mean \(\mu_X+C(y-\mu_Y)\) and covariance \(S\). To condition on \(V=v\), first recover the corresponding original value \(y=B^{-1}(v-b)\). Then transform the conditional response by \(u=a+Ax\).

Theorem: Under the conditions above,
$$ U\mid V=v \sim N\!\left( a+A\left[\mu_X+C\bigl(B^{-1}(v-b)-\mu_Y\bigr)\right], \,ASA^T \right). $$

Thus the affine offset \(a\) shifts the conditional mean but does not affect the conditional covariance. The response matrix \(A\) transforms the covariance to \(ASA^T\). The conditioning matrix \(B\) changes the input value in the mean through \(B^{-1}(v-b)\), but does not otherwise change the conditional covariance.

Why the Formula Holds

A direct justification uses the Gaussian regression residual. Define \(R=X-\mu_X-C(Y-\mu_Y)\). Its covariance with \(Y\) is zero, since

$$ \operatorname{Cov}(R,Y) =\Sigma_{XY}-C\Sigma_{YY} =\Sigma_{XY}-\Sigma_{XY}\Sigma_{YY}^{-1}\Sigma_{YY} =0. $$

The pair \((R,Y)\) is jointly Gaussian, so zero cross-covariance implies independence. Also, expanding the covariance of \(R\) gives \(\operatorname{Var}(R)=S\). Consequently, conditionally on \(Y=y\), the random vector \(X\) has the distribution of the fixed vector \(\mu_X+C(y-\mu_Y)\) plus an independent Gaussian residual with covariance \(S\). Substituting \(y=B^{-1}(v-b)\) and applying \(u=a+Ax\) yields the stated mean and covariance.

The same formula can be checked by calculating the transformed covariance blocks. They are

$$ \begin{aligned} \mu_U&=a+A\mu_X, & \mu_V&=b+B\mu_Y,\\ \Sigma_{UU}&=A\Sigma_{XX}A^T, &\Sigma_{UV}&=A\Sigma_{XY}B^T,\\ &&\Sigma_{VV}&=B\Sigma_{YY}B^T. \end{aligned} $$

Because \(B\) and \(\Sigma_{YY}\) are invertible,

$$ \begin{aligned} \Sigma_{UV}\Sigma_{VV}^{-1} &=A\Sigma_{XY}B^T(B\Sigma_{YY}B^T)^{-1}\\ &=A\Sigma_{XY}\Sigma_{YY}^{-1}B^{-1} =ACB^{-1}. \end{aligned} $$

Here \((B\Sigma_{YY}B^T)^{-1}=B^{-T}\Sigma_{YY}^{-1}B^{-1}\), where \(B^{-T}=(B^T)^{-1}\). Substituting this transformed regression coefficient and the transformed means into the block conditional mean gives the theorem’s mean. For the covariance, the block formula gives

$$ \begin{aligned} \Sigma_{UU}-\Sigma_{UV}\Sigma_{VV}^{-1}\Sigma_{VU} &=A\Sigma_{XX}A^T -A\Sigma_{XY}\Sigma_{YY}^{-1}\Sigma_{YX}A^T\\ &=ASA^T. \end{aligned} $$

This second calculation confirms that the result agrees with conditioning directly in the transformed coordinates. \(\square\)

Worked Examples

Worked Example: A Shear of the Response and a Rescaling of the Conditioning Value

Suppose \(X\in\mathbb{R}^2\) and \(Y\in\mathbb{R}\) are jointly Gaussian, with

$$ \mu_X=\begin{pmatrix}1\\-1\end{pmatrix},\quad \mu_Y=3,\quad \Sigma_{XX}=\begin{pmatrix}2&0.5\\0.5&1\end{pmatrix},\quad \Sigma_{XY}=\begin{pmatrix}1\\0.5\end{pmatrix},\quad \Sigma_{YY}=2. $$

First calculate the original conditional law. The regression coefficient and residual covariance are

$$ \begin{aligned} C&=\Sigma_{XY}\Sigma_{YY}^{-1} =\begin{pmatrix}0.5\\0.25\end{pmatrix},\\ S&=\begin{pmatrix}2&0.5\\0.5&1\end{pmatrix} -\frac{1}{2} \begin{pmatrix}1\\0.5\end{pmatrix} \begin{pmatrix}1&0.5\end{pmatrix} =\begin{pmatrix}1.5&0.25\\0.25&0.875\end{pmatrix}. \end{aligned} $$

At \(y=5\), the conditional mean is

$$ \mu_X+C(5-\mu_Y) =\begin{pmatrix}1\\-1\end{pmatrix} +2\begin{pmatrix}0.5\\0.25\end{pmatrix} =\begin{pmatrix}2\\-0.5\end{pmatrix}. $$

Now take \(U=a+AX\) and \(V=b+BY\), where \(a=(0,1)^T\), \(A=\begin{pmatrix}1&2\\0&1\end{pmatrix}\), \(b=-1\), and \(B=3\). The value \(Y=5\) corresponds to \(V=-1+3(5)=14\). The transformed conditional mean is

$$ a+A\begin{pmatrix}2\\-0.5\end{pmatrix} =\begin{pmatrix}0\\1\end{pmatrix} +\begin{pmatrix}1&2\\0&1\end{pmatrix} \begin{pmatrix}2\\-0.5\end{pmatrix} =\begin{pmatrix}1\\0.5\end{pmatrix}. $$

For the covariance, multiply in two stages:

$$ \begin{aligned} AS &=\begin{pmatrix}1&2\\0&1\end{pmatrix} \begin{pmatrix}1.5&0.25\\0.25&0.875\end{pmatrix} =\begin{pmatrix}2&2\\0.25&0.875\end{pmatrix},\\ ASA^T &=\begin{pmatrix}2&2\\0.25&0.875\end{pmatrix} \begin{pmatrix}1&0\\2&1\end{pmatrix} =\begin{pmatrix}6&2\\2&0.875\end{pmatrix}. \end{aligned} $$

Therefore \(U\mid V=14\) has mean \((1,0.5)^T\) and covariance \(\begin{pmatrix}6&2\\2&0.875\end{pmatrix}\). Notice that the covariance changes even though the observed value \(V=14\) only determines which original conditioning value to use.

Worked Example: Changing Only the Conditioning Coordinates

Let \(X\) be scalar and \(Y\in\mathbb{R}^2\), with

$$ \mu_X=2,\quad \mu_Y=\begin{pmatrix}1\\0\end{pmatrix},\quad \operatorname{Var}(X)=9,\quad \Sigma_{XY}=\begin{pmatrix}2&1\end{pmatrix},\quad \Sigma_{YY}=\begin{pmatrix}2&0\\0&1\end{pmatrix}. $$

The original regression coefficient is \(\Sigma_{XY}\Sigma_{YY}^{-1}=(1,1)\), and the conditional variance is \(9-(2,1)\begin{pmatrix}1\\1\end{pmatrix}=6\). At \(Y=(3,2)^T\), the conditional mean is \(2+(1,1)(2,2)^T=6\).

Reparameterize the conditioning vector as \(V=b+BY\), with \(B=\begin{pmatrix}1&1\\0&2\end{pmatrix}\) and \(b=(-1,4)^T\); leave the response unchanged. The value \(Y=(3,2)^T\) becomes \(V=(4,8)^T\), while \(\mu_V=(0,4)^T\). The transformed covariance blocks are

$$ \Sigma_{XV}=(2,1)B^T=(3,2),\qquad \Sigma_{VV}=B\begin{pmatrix}2&0\\0&1\end{pmatrix}B^T =\begin{pmatrix}3&2\\2&4\end{pmatrix}. $$

The inverse of \(\Sigma_{VV}\) is \(\frac18\begin{pmatrix}4&-2\\-2&3\end{pmatrix}\), so the transformed regression coefficient is

$$ (3,2)\frac18\begin{pmatrix}4&-2\\-2&3\end{pmatrix}=(1,0). $$

It follows that the transformed conditional mean is \(2+(1,0)(4,4)^T=6\), and the conditional variance remains \(6\). Both match the calculation in the original coordinates.

Worked Example: Standardizing Both Scalar Variables

Suppose \(X\) and \(Y\) are jointly Gaussian with means \(10\) and \(4\), variances \(16\) and \(9\), and covariance \(6\). Define \(U=(X-10)/2\) and \(V=(Y-4)/3\). In the original coordinates, the conditional mean and variance are

$$ E[X\mid Y=y]=10+\frac{6}{9}(y-4), \qquad \operatorname{Var}(X\mid Y=y)=16-\frac{6^2}{9}=12. $$

If \(V=1\), then \(Y=4+3(1)=7\). Thus \(E[X\mid Y=7]=10+\frac23(3)=12\), and \(E[U\mid V=1]=(12-10)/2=1\). The conditional variance of \(U\) is \(12/2^2=3\).

The transformed covariance calculation checks both results: \(E[U]=E[V]=0\), \(\operatorname{Var}(U)=16/4=4\), \(\operatorname{Var}(V)=9/9=1\), and \(\operatorname{Cov}(U,V)=6/(2\cdot3)=1\). Hence the conditional mean of \(U\) given \(V=1\) is \(0+(1/1)(1)=1\), and its conditional variance is \(4-1^2/1=3\).

Common Mistakes and Practical Uses

Affine reparameterization is useful when units, centering, or coordinate systems change between a model and its interpretation. The formulas let one move between these representations without recalculating the underlying conditional relationship from scratch.

A frequent mistake is to substitute the transformed conditioning value directly into the original conditional mean. In the theorem, the original value is \(y=B^{-1}(v-b)\): the offset must be removed before applying the inverse matrix. Another is to transform a covariance as \(ASA\) rather than \(ASA^T\). For a nonsymmetric response transformation, only the congruence \(ASA^T\) gives the covariance of \(AX\). Finally, adding a response offset changes the mean but never adds a term to the covariance.

Invertibility matters. If \(B\) is invertible, knowing \(V=v\) is equivalent to knowing \(Y=B^{-1}(v-b)\). A noninvertible transformation can discard information, so conditioning on the transformed variable need not give the same law. That situation requires a different analysis.

Key takeaway: Under invertible affine changes, undo the conditioning transformation to identify the original conditioning value, then apply the response transformation to the original conditional mean and covariance. The conditional covariance becomes \(ASA^T\), and an invertible reparameterization of the conditioning coordinates alone leaves it unchanged.

Check Your Understanding

Use the transformation formula and show the key matrix or scalar calculations.

  1. If \(U=a+AX\) and only the response is transformed, which part of the conditional law is affected by \(a\), and how does \(A\) act on the covariance?
  2. For \(V=b+BY\), write the original conditioning value corresponding to \(V=v\). Why must \(B\) be invertible for this expression to be defined uniquely?
  3. A conditional response covariance is \(S=\begin{pmatrix}2&1\\1&2\end{pmatrix}\) and \(A=\begin{pmatrix}1&1\\0&1\end{pmatrix}\). Calculate \(ASA^T\).
  4. Explain why an invertible affine change of conditioning coordinates does not change the conditional covariance of the response.
  5. In the standardization example, verify the conditional variance of \(U\) using only the transformed variances and covariance.

Previous

Next

Back to Tutorials