Conditioning Gaussian Vectors on Sums and Contrasts

Why Sums and Contrasts Need Their Own View

A sum or difference of Gaussian coordinates is itself Gaussian, but conditioning on one changes the joint law of all the coordinates. The change depends on their variances and covariances: under correlation, observing a sum can shift the conditional means unevenly, and a difference can convey information about coordinates not appearing in it. A matrix formulation keeps those effects explicit.

A contrast is a linear combination that compares coordinates, commonly with coefficients summing to zero. Examples include $X_1-X_2$ and $X_1-(X_2+X_3)/2$. A sum, such as $X_1+X_2$, is not a zero-sum contrast in the strict sense, but it is also a linear observation and uses the same conditioning calculation.

Conditioning on Linear Observations

Definition: Let $X$ be a $p$-dimensional Gaussian vector with mean vector $\mu$ and positive-definite covariance matrix $\Sigma$. For an $r\times p$ matrix $A$, the linear observation is $Y=AX$. If $A$ has full row rank, then $A\Sigma A^{\mathsf T}$ is invertible, and the conditional law given $AX=y$ has mean and covariance $$ E[X\mid AX=y]=\mu+\Sigma A^{\mathsf T}(A\Sigma A^{\mathsf T})^{-1}(y-A\mu), $$ $$ \operatorname{Cov}(X\mid AX=y) =\Sigma-\Sigma A^{\mathsf T}(A\Sigma A^{\mathsf T})^{-1}A\Sigma. $$

The observed value $y$ is a vector of $r$ sums or contrasts. The quantity $y-A\mu$ is its departure from its mean. The matrix multiplying this departure describes how the observation shifts each coordinate. The conditional covariance does not depend on the observed value: once the model and observation matrix are fixed, it is the same for every $y$.

These formulas describe a conditional Gaussian distribution, including when its covariance is singular. In that case the distribution is supported on a lower-dimensional affine set rather than having a density on all of $\mathbb R^p$. In particular, conditioning on exact linear observations forces the resulting vector to satisfy those observations.

Worked Example: Two Correlated Coordinates Given Their Sum

Suppose $(X_1,X_2)^{\mathsf T}$ has mean $(1,2)^{\mathsf T}$ and covariance matrix $$ \Sigma= \begin{pmatrix} 4&3\\ 3&9 \end{pmatrix}. $$ Find its conditional law given $X_1+X_2=s$.

Write $A=(1,1)$. Then $A\mu=3$, and $$ A\Sigma A^{\mathsf T}=4+3+3+9=19,\qquad \Sigma A^{\mathsf T}= \begin{pmatrix}7\\12\end{pmatrix}. $$ The conditional mean is therefore $$ \begin{pmatrix}1\\2\end{pmatrix} +\frac{1}{19}\begin{pmatrix}7\\12\end{pmatrix}(s-3) = \begin{pmatrix} 1+\frac{7}{19}(s-3)\\ 2+\frac{12}{19}(s-3) \end{pmatrix}. $$

For the covariance, subtract the rank-one adjustment: $$ \begin{pmatrix}4&3\\3&9\end{pmatrix} -\frac{1}{19} \begin{pmatrix}49&84\\84&144\end{pmatrix} = \frac{1}{19} \begin{pmatrix}27&-27\\-27&27\end{pmatrix}. $$ Thus the conditional variance of each coordinate is $27/19$, and their conditional covariance is $-27/19$. Each realized pair satisfies $X_1+X_2=s$; accordingly, the sum of the entries in every row of the conditional covariance matrix is zero. The conditional variation is entirely in the direction that changes one coordinate up as the other goes down.

Why the Formula Holds

The conditional law can be derived by removing from $X$ the part linearly predicted by $Y=AX$. The remainder is uncorrelated with $Y$; because the pair is jointly Gaussian, it is independent of $Y$. The Gaussian fact used here is that jointly Gaussian vectors with zero cross-covariance are independent.

Theorem: Under the conditions in the definition, set $$ K=\Sigma A^{\mathsf T}(A\Sigma A^{\mathsf T})^{-1}, \qquad R=X-\mu-K(AX-A\mu). $$ Then $R$ is independent of $AX$, has mean zero, and has covariance $\Sigma-\Sigma A^{\mathsf T}(A\Sigma A^{\mathsf T})^{-1}A\Sigma$. Consequently, $X\mid AX=y$ is Gaussian with the mean and covariance stated above.

Proof. Write $Y=AX$. Since $X$ is Gaussian and $R$ and $Y$ are affine transformations of $X$, the pair $(R,Y)$ is jointly Gaussian. Also, $$ \operatorname{Cov}(X,Y)=\Sigma A^{\mathsf T}, \qquad \operatorname{Var}(Y)=A\Sigma A^{\mathsf T}. $$ It follows that $$ \operatorname{Cov}(R,Y) =\Sigma A^{\mathsf T}-K(A\Sigma A^{\mathsf T})=0. $$ Joint Gaussianity and zero cross-covariance give $R\perp Y$. Its mean is zero because $E[Y]=A\mu$.

To calculate its covariance, use $R=(I-KA)(X-\mu)$. Expanding gives $$ \operatorname{Var}(R) =\Sigma-\Sigma A^{\mathsf T}K^{\mathsf T} -KA\Sigma+KA\Sigma A^{\mathsf T}K^{\mathsf T}. $$ With $K=\Sigma A^{\mathsf T}(A\Sigma A^{\mathsf T})^{-1}$, the last three terms combine to $-\Sigma A^{\mathsf T}(A\Sigma A^{\mathsf T})^{-1}A\Sigma$. Thus $$ \operatorname{Var}(R) =\Sigma-\Sigma A^{\mathsf T}(A\Sigma A^{\mathsf T})^{-1}A\Sigma. $$ Finally, $X=\mu+K(Y-A\mu)+R$. Given $Y=y$, the first two terms are fixed and $R$ retains its unconditional distribution because it is independent of $Y$. This gives the asserted conditional mean and covariance. $\square$

Independent Coordinates and Multiple Contrasts

Worked Example: Independent Normal Coordinates Given Their Total

Let $X_1,X_2,X_3$ be independent $N(2,4)$ variables. Find the conditional law of $X=(X_1,X_2,X_3)^{\mathsf T}$ given $X_1+X_2+X_3=12$.

Here $\mu=2\mathbf 1$, $\Sigma=4I_3$, and $A=(1,1,1)$. The observation has mean $A\mu=6$ and variance $A\Sigma A^{\mathsf T}=12$. Also, $\Sigma A^{\mathsf T}=4\mathbf 1$. Since the observed total exceeds its mean by $12-6=6$, the conditional mean is $$ 2\mathbf 1+4\mathbf 1\frac{6}{12} = \begin{pmatrix}4\\4\\4\end{pmatrix}. $$ The conditional covariance is $$ 4I_3-\frac{(4\mathbf 1)(4\mathbf 1)^{\mathsf T}}{12} =4I_3-\frac{4}{3}\mathbf 1\mathbf 1^{\mathsf T} = \begin{pmatrix} 8/3&-4/3&-4/3\\ -4/3&8/3&-4/3\\ -4/3&-4/3&8/3 \end{pmatrix}. $$ The negative off-diagonal covariances express a constraint: if one coordinate is above its conditional mean, the other two must together be below theirs so that the total remains 12. Each row sums to zero, as required by the fixed-sum condition.

Worked Example: Two Contrasts of Three Coordinates

Let $X_1,X_2,X_3$ be independent standard normal variables. Find the conditional law given $X_1-X_2=2$ and $X_2-X_3=-1$.

Set $$ A=\begin{pmatrix}1&-1&0\\0&1&-1\end{pmatrix}, \qquad y=\begin{pmatrix}2\\-1\end{pmatrix}. $$ Since $\Sigma=I_3$, we have $$ A\Sigma A^{\mathsf T}=AA^{\mathsf T} =\begin{pmatrix}2&-1\\-1&2\end{pmatrix}, \qquad (AA^{\mathsf T})^{-1} =\frac{1}{3}\begin{pmatrix}2&1\\1&2\end{pmatrix}. $$ The mean vector is zero, so the conditional mean is $A^{\mathsf T}(AA^{\mathsf T})^{-1}y$. First, $$ (AA^{\mathsf T})^{-1}y =\frac{1}{3}\begin{pmatrix}2&1\\1&2\end{pmatrix} \begin{pmatrix}2\\-1\end{pmatrix} =\begin{pmatrix}1\\0\end{pmatrix}. $$ Therefore the conditional mean is $A^{\mathsf T}(1,0)^{\mathsf T}=(1,-1,0)^{\mathsf T}$.

The conditional covariance is $I_3-A^{\mathsf T}(AA^{\mathsf T})^{-1}A$, which evaluates to $$ \frac{1}{3} \begin{pmatrix}1&1&1\\1&1&1\\1&1&1\end{pmatrix}. $$ Thus the conditional vector has the representation $$ X= \begin{pmatrix}1\\-1\\0\end{pmatrix}+Z \begin{pmatrix}1\\1\\1\end{pmatrix}, \qquad Z\sim N(0,1/3). $$ Indeed, the two differences of the mean are $1-(-1)=2$ and $-1-0=-1$; adding the common random quantity $Z$ does not change either difference. The conditional support is a line: the contrasts fix two directions, while the common-level direction remains random.

Worked Example: A Difference in a Correlated Vector

Suppose $X=(X_1,X_2,X_3)^{\mathsf T}$ has mean zero and covariance $$ \Sigma=\begin{pmatrix}4&1&0\\1&3&1\\0&1&2\end{pmatrix}. $$ Find the conditional law given the single contrast $X_1-X_2=d$.

Take $A=(1,-1,0)$. Then $$ A\Sigma A^{\mathsf T}=4+3-2=5, \qquad \Sigma A^{\mathsf T}=\begin{pmatrix}3\\-2\\-1\end{pmatrix}. $$ The conditional mean is $(3,-2,-1)^{\mathsf T}d/5$. For the covariance, subtract $(\Sigma A^{\mathsf T})(\Sigma A^{\mathsf T})^{\mathsf T}/5$ from $\Sigma$: $$ \operatorname{Cov}(X\mid X_1-X_2=d) =\frac{1}{5}\begin{pmatrix}11&11&3\\11&11&3\\3&3&9\end{pmatrix}. $$ The first two conditional means differ by $d$, and the first two rows of the covariance matrix are identical. Hence $X_1-X_2$ has zero conditional variance, as it must. The third coordinate's conditional mean is $-d/5$: even though $X_3$ is not part of the observed contrast, its correlation with that contrast shifts its conditional distribution.

Rank, Support, and a Common Pitfall

If $A$ has full row rank and $\Sigma$ is positive definite, then $A\Sigma A^{\mathsf T}$ is positive definite: for every nonzero $v\in\mathbb R^r$, $v^{\mathsf T}A\Sigma A^{\mathsf T}v=(A^{\mathsf T}v)^{\mathsf T}\Sigma(A^{\mathsf T}v)>0$, since full row rank ensures $A^{\mathsf T}v\ne 0$. The conditional covariance can nevertheless be singular. The observation fixes $r$ independent linear combinations, so the conditional vector is supported on the affine set $\{x:Ax=y\}$, which has dimension $p-r$.

Common mistake: Treating a fixed sum as if the coordinates remain conditionally independent because they were independent before observing the sum. In the three-coordinate example, the original covariances were zero, but after conditioning on the total, every off-diagonal covariance is $-4/3$. The correction is to recompute the entire conditional covariance matrix; fixing a sum couples the coordinates.

Another algebraic pitfall is to use the displayed inverse when the observation rows are redundant. For example, recording both $X_1-X_2$ and $2X_1-2X_2$ does not provide two independent observations: $A\Sigma A^{\mathsf T}$ is singular. Retain one independent contrast, or choose a full-row-rank matrix with the same row space and impose the corresponding consistent observed values. Do not apply the ordinary inverse formula to the redundant matrix.

Key takeaway: Encode each sum or contrast as $AX$. The conditional mean adjusts according to the covariance with that observation; the conditional covariance removes the variation explained by it. Zero conditional variance in an observed direction records an exact constraint, not a calculation error.

Check Your Understanding

For each question, identify the observation matrix before attempting the calculation.

If $X$ is a Gaussian vector and $A$ has full row rank, what condition ensures that $A\Sigma A^{\mathsf T}$ is invertible?
In the first worked example, why must the rows of the conditional covariance matrix sum to zero?
For independent $N(0,1)$ coordinates $X_1,X_2,X_3$, what direction remains random after conditioning on both $X_1-X_2$ and $X_2-X_3$?
Why can conditioning create nonzero covariances between coordinates that were initially independent?
What should be done before using the inverse formula if an observation matrix contains redundant contrast rows?