Marginal Gaussian Laws from Coordinate Selection

From a Joint Gaussian Model to a Marginal

A jointly Gaussian vector describes several measurements together, but many questions concern only a subset of them. For example, an analysis may record four clinical measurements while focusing on two. The distribution of those two measurements is a marginal distribution: it describes the selected coordinates without conditioning on the values of the others.

For a Gaussian vector, finding this marginal does not require a new model or a complicated calculation. Coordinate selection is a linear transformation, so the selected vector remains Gaussian. Its mean and covariance are obtained by selecting the corresponding entries and covariance rows and columns.

Definition: Let $X=(X_1,\ldots,X_p)^T$ be a jointly Gaussian vector with mean $\mu$ and covariance matrix $\Sigma$. Choose $k$ distinct indices $I=(i_1,\ldots,i_k)$, in the order desired for the selected vector. The coordinate-selection matrix $S_I$ is the $k$ by $p$ matrix whose $r$th row is the row vector with a 1 in position $i_r$ and 0 in every other position. The selected vector is $Y=S_I X=(X_{i_1},\ldots,X_{i_k})^T$.

We write $X\sim N_p(\mu,\Sigma)$, allowing $\Sigma$ to be positive semidefinite and thus allowing a singular Gaussian law. If $\Sigma$ is positive definite, the law has a density; if it is singular, it does not have an ordinary density on all of $\mathbb{R}^p$. Coordinate selection is valid in either case.

The Coordinate-Selection Result

Theorem: If $X\sim N_p(\mu,\Sigma)$ and $S_I$ selects $k$ distinct coordinates, then $$ Y=S_I X\sim N_k(S_I\mu,\;S_I\Sigma S_I^T). $$ In particular, $E[Y]=S_I\mu$, and $\operatorname{Cov}(Y)=S_I\Sigma S_I^T$. The $r$th selected mean is $\mu_{i_r}$, and the $(r,s)$ entry of the selected covariance matrix is $\Sigma_{i_r i_s}$.

Proof. A Gaussian vector with possibly singular covariance has a representation in distribution $X=\mu+\Sigma^{1/2}Z$, where $Z\sim N_p(0,I_p)$ and $\Sigma^{1/2}$ is a symmetric square root of $\Sigma$. Therefore $$ Y=S_I\mu+S_I\Sigma^{1/2}Z. $$ This is an affine transformation of a standard Gaussian vector, so $Y$ is Gaussian. Its expectation is $S_I\mu$, since $E[Z]=0$. Its covariance is $$ \operatorname{Cov}(Y) =S_I\Sigma^{1/2}\operatorname{Cov}(Z)(\Sigma^{1/2})^T S_I^T =S_I\Sigma^{1/2}I_p\Sigma^{1/2}S_I^T =S_I\Sigma S_I^T. $$ The definition of $S_I$ then shows that the entries are exactly the means and covariances at the selected indices. $\square$

This result gives the marginal parameters directly. It also shows why the ordering in $I$ matters: changing the order changes the order of the selected mean entries and the rows and columns of the covariance matrix, but not the underlying coordinates being described.

Worked Example: Selecting Two Nonconsecutive Coordinates

Suppose $X=(X_1,X_2,X_3)^T\sim N_3(\mu,\Sigma)$, where $$ \mu= \begin{pmatrix}2\\-1\\4\end{pmatrix}, \qquad \Sigma= \begin{pmatrix} 4&1&2\\ 1&3&0\\ 2&0&5 \end{pmatrix}. $$ Find the marginal law of $(X_1,X_3)^T$.

The covariance matrix is valid: its leading principal minors are $4$, $4(3)-1=11$, and $\det(\Sigma)=43$, all positive, so $\Sigma$ is positive definite. The selection matrix for indices $I=(1,3)$ is $$ S_I= \begin{pmatrix} 1&0&0\\ 0&0&1 \end{pmatrix}. $$

Multiplying the mean gives $S_I\mu=(2,4)^T$. The covariance calculation selects rows 1 and 3, then columns 1 and 3: $$ S_I\Sigma S_I^T= \begin{pmatrix} 4&2\\ 2&5 \end{pmatrix}. $$ Thus $$ \begin{pmatrix}X_1\\X_3\end{pmatrix} \sim N_2\left( \begin{pmatrix}2\\4\end{pmatrix}, \begin{pmatrix}4&2\\2&5\end{pmatrix} \right). $$ The covariance $2$ between $X_1$ and $X_3$ remains in the marginal. The covariance with the omitted coordinate $X_2$ is not an entry of this two-coordinate covariance matrix.

Reading Marginals from a Partitioned Model

When coordinates are grouped into two subvectors, the selection rule takes a particularly simple form. Write $X=(U^T,V^T)^T$, where $U$ contains the $k$ coordinates of interest and $V$ contains the remaining coordinates. Partition the mean and covariance conformably: $$ \mu= \begin{pmatrix}\mu_U\\\mu_V\end{pmatrix}, \qquad \Sigma= \begin{pmatrix} \Sigma_{UU}&\Sigma_{UV}\\ \Sigma_{VU}&\Sigma_{VV} \end{pmatrix}. $$ Selecting $U$ means using $S=(I_k\;\;0)$. Consequently, the marginal mean is $\mu_U$ and the marginal covariance is $\Sigma_{UU}$. The other blocks are part of the joint model, but they are not substituted into the marginal covariance calculation.

Worked Example: Extracting a Marginal from a Partitioned Covariance

Let $X=(U_1,U_2,V_1,V_2)^T$ be jointly Gaussian with $$ \mu= \begin{pmatrix}120\\8\\5\\2\end{pmatrix}, \qquad \Sigma= \begin{pmatrix} 4&1&2&0\\ 1&3&0&1\\ 2&0&5&1\\ 0&1&1&4 \end{pmatrix}. $$ Find the joint marginal law of $U=(U_1,U_2)^T$.

The covariance matrix is symmetric and strictly diagonally dominant with positive diagonal entries: the diagonal entries $4,3,5,4$ exceed the respective sums of the absolute off-diagonal entries $3,2,3,2$. It is therefore positive definite. Partitioning after the first two coordinates gives $$ \mu_U=\begin{pmatrix}120\\8\end{pmatrix}, \qquad \Sigma_{UU}= \begin{pmatrix}4&1\\1&3\end{pmatrix}. $$ Equivalently, the selection matrix is $S=(I_2\;\;0)$, and multiplying $S\mu$ and $S\Sigma S^T$ yields these same quantities. Hence $$ \begin{pmatrix}U_1\\U_2\end{pmatrix} \sim N_2\left( \begin{pmatrix}120\\8\end{pmatrix}, \begin{pmatrix}4&1\\1&3\end{pmatrix} \right). $$ For example, the marginal variance of $U_1$ is $4$, and the marginal covariance of $U_1$ and $U_2$ is $1$. Neither calculation requires the covariance blocks involving $V$.

Result: Consistency under Further Selection. Suppose $Y=S_I X$ selects $I=(i_1,\ldots,i_k)$, and $T$ selects entries $r_1,\ldots,r_m$ from $Y$, in that order. Then $TY$ has the same law as direct selection of $(X_{i_{r_1}},\ldots,X_{i_{r_m}})^T$ from $X$.

Proof. The $a$th row of $T$ selects entry $r_a$ of $Y$, which is $X_{i_{r_a}}$. Thus the product $TS_I$ is exactly the selection matrix for the ordered indices $(i_{r_1},\ldots,i_{r_m})$. Since $TY=TS_I X$, both procedures give the same random vector, not merely vectors with the same mean and covariance. Applying the coordinate-selection theorem also gives mean $TS_I\mu$ and covariance $TS_I\Sigma S_I^T$, the same quantities obtained by direct selection. $\square$

A Marginal Can Be Singular

A positive definite joint covariance guarantees that the covariance of any nonempty coordinate subset is positive definite. Indeed, for any nonzero $a\in\mathbb{R}^k$, the vector $S_I^T a$ is nonzero, and $$ a^T(S_I\Sigma S_I^T)a=(S_I^T a)^T\Sigma(S_I^T a)>0. $$ If the joint covariance is only positive semidefinite, however, a selected covariance may be singular. This is a valid Gaussian marginal, not a failure of the selection rule.

Worked Example: A Singular Selected Marginal

Let $Z$ and $W$ be independent $N(0,1)$ random variables, and define $$ X_1=Z,\qquad X_2=1+2Z,\qquad X_3=2+W. $$ Then $X=(X_1,X_2,X_3)^T$ is jointly Gaussian, with $$ \mu= \begin{pmatrix}0\\1\\2\end{pmatrix}, \qquad \Sigma= \begin{pmatrix} 1&2&0\\ 2&4&0\\ 0&0&1 \end{pmatrix}. $$ Selecting coordinates 1 and 2 gives $$ \begin{pmatrix}X_1\\X_2\end{pmatrix} \sim N_2\left( \begin{pmatrix}0\\1\end{pmatrix}, \begin{pmatrix}1&2\\2&4\end{pmatrix} \right). $$ The selected covariance has determinant $1(4)-2(2)=0$, and the selected variables satisfy $X_2=1+2X_1$ with probability one. In particular, $\operatorname{Var}(X_2-2X_1)=4+4(1)-4(2)=0$. The marginal is concentrated on a line, so it has no ordinary density over the plane; the stated Gaussian law is still correct.

Common Pitfall

A frequent error is to treat marginalization as conditioning. The marginal law of $U$ describes $U$ without fixing $V$; the conditional law of $U$ given $V=v$ describes $U$ after a value of $V$ has been specified. These are different questions. For the marginal, use $\mu_U$ and $\Sigma_{UU}$; do not alter them by inserting cross-covariance blocks from the joint model. Conditioning requires a separate calculation.

Another tempting but incorrect shortcut is to delete the coordinates of the mean and covariance without respecting their order. For indices $I=(3,1)$, for instance, the selected vector is $(X_3,X_1)^T$, so both the mean entries and covariance rows and columns must appear in the order 3, then 1.

Key takeaway: For any ordered subset of coordinates of a jointly Gaussian vector, the marginal is Gaussian. Select the matching entries of the mean and the matching rows and columns of the covariance; no conditioning calculation is involved.

Check Your Understanding

Use coordinate selection to answer each question.

If $X\sim N_4(\mu,\Sigma)$ and the selected order is $I=(4,2)$, which entries of $\mu$ and $\Sigma$ determine the selected mean and covariance?
For a partition $X=(U^T,V^T)^T$, state the mean and covariance of the marginal law of $U$.
Why can a selected Gaussian marginal be singular even though its distribution is well defined?
Explain why the marginal law of $U$ is not the same object as the conditional law of $U$ given $V=v$.
If a vector is selected first and then some of those selected coordinates are selected again, why does this agree with selecting those coordinates directly from the original vector?