Marginal Laws of Gaussian Subvectors

Why Marginal Laws Are Simple to Extract

A multivariate Gaussian model may describe many measurements at once, while an analysis uses only a few of them. The central task is to obtain the joint law of those chosen coordinates, keeping their original dependence intact. For a Gaussian vector, the answer is determined directly by the corresponding entries of its mean vector and covariance matrix.

The selection can be in any specified order, and the original covariance matrix may be singular. Neither circumstance requires a density or a matrix inverse. The key is to treat selection as a linear map that retains coordinates and discards the rest.

Coordinate Selection and Partitioned Parameters

Definition: Let $X=(X_1,\ldots,X_p)^\mathsf{T}$ be a random vector, and let $I=(i_1,\ldots,i_k)$ be an ordered list of distinct indices from $\{1,\ldots,p\}$. The selected subvector is $X_I=(X_{i_1},\ldots,X_{i_k})^\mathsf{T}$. Its selection matrix $S_I$ is the $k\times p$ matrix whose $a$th row has a 1 in column $i_a$ and zeros elsewhere. Thus $X_I=S_I X$.

The ordering in $I$ matters: the first selected coordinate is $X_{i_1}$, the second is $X_{i_2}$, and so on. If $X$ has mean vector $\mu$ and covariance matrix $\Sigma$, write $\mu_I$ for the entries of $\mu$ in that order and $\Sigma_{II}$ for the $k\times k$ matrix with entry $(a,b)$ equal to $\Sigma_{i_a i_b}$. This is a principal submatrix, with its rows and columns in the selected order.

For example, if the selected indices are $I=(3,1)$, then $\mu_I=(\mu_3,\mu_1)^\mathsf{T}$ and $\Sigma_{II}$ has rows and columns ordered as 3, then 1. It is not enough to extract the right entries; their arrangement must match the arrangement of $X_I$.

Gaussian marginal theorem: If $X\sim N_p(\mu,\Sigma)$, where $\Sigma$ is positive semidefinite and may be singular, then for any ordered list $I$ of $k$ distinct coordinate indices, $$ X_I=S_I X\sim N_k(\mu_I,\Sigma_{II}). $$ In particular, $E[X_I]=\mu_I$ and $\operatorname{Cov}(X_I)=\Sigma_{II}$.

Deriving the Marginal Law

The characteristic function of a Gaussian vector $X\sim N_p(\mu,\Sigma)$, including a possibly singular Gaussian vector, is $\phi_X(t)=\exp\{it^\mathsf{T}\mu-\tfrac12t^\mathsf{T}\Sigma t\}$ for $t\in\mathbb{R}^p$. A characteristic function determines a probability distribution uniquely. We can therefore derive the selected vector's law by evaluating its characteristic function.

For any $t\in\mathbb{R}^k$, the definition $X_I=S_I X$ gives $\phi_{X_I}(t)=E[\exp\{it^\mathsf{T}S_I X\}]=\phi_X(S_I^\mathsf{T}t)$. Substituting the Gaussian characteristic function yields the following complete calculation.

\begin{aligned} \phi_{X_I}(t) &=\exp\left\{i(S_I^\mathsf{T}t)^\mathsf{T}\mu -\tfrac12(S_I^\mathsf{T}t)^\mathsf{T}\Sigma(S_I^\mathsf{T}t)\right\}\\ &=\exp\left\{it^\mathsf{T}(S_I\mu) -\tfrac12t^\mathsf{T}(S_I\Sigma S_I^\mathsf{T})t\right\}. \end{aligned}

By the definitions of the selected parameters, $S_I\mu=\mu_I$ and $S_I\Sigma S_I^\mathsf{T}=\Sigma_{II}$. The displayed function is the characteristic function of $N_k(\mu_I,\Sigma_{II})$. Uniqueness of characteristic functions proves the distributional assertion. The mean and covariance also follow entry by entry: the $a$th component is $X_{i_a}$, so its mean is $\mu_{i_a}$, and the covariance of components $a$ and $b$ is $\operatorname{Cov}(X_{i_a},X_{i_b})=\Sigma_{i_a i_b}$. $\square$

This derivation does not integrate a joint density. That matters when the original law is singular, since a singular Gaussian vector has no density with respect to ordinary $p$-dimensional volume. It also shows why the result holds for every valid covariance matrix, not only for positive definite ones.

Worked Example: Selecting Nonconsecutive Coordinates

Suppose $X\sim N_4(\mu,\Sigma)$, with $\mu=(-1,0,2,5)^\mathsf{T}$ and $\Sigma=LL^\mathsf{T}+D$, where $$ L=\begin{pmatrix} 1&0\\ 0&1\\ 1&1\\ 0&1 \end{pmatrix}, \qquad D=\begin{pmatrix} 3&0&0&0\\ 0&2&0&0\\ 0&0&3&0\\ 0&0&0&1 \end{pmatrix}. $$ Find the law of $X_I=(X_1,X_3)^\mathsf{T}$.

First, the specified covariance is valid: for any $v\in\mathbb{R}^4$, $v^\mathsf{T}\Sigma v=\|L^\mathsf{T}v\|^2+v^\mathsf{T}Dv\geq 0$, since $D$ has positive diagonal entries. The matrix and selected parameters are $$ \Sigma= \begin{pmatrix} 4&0&1&0\\ 0&3&1&1\\ 1&1&5&1\\ 0&1&1&2 \end{pmatrix}, \qquad \mu_I= \begin{pmatrix}-1\\2\end{pmatrix}, \qquad \Sigma_{II}= \begin{pmatrix}4&1\\1&5\end{pmatrix}. $$

The off-diagonal entry 1 is $\operatorname{Cov}(X_1,X_3)$, and the diagonal entries are $\operatorname{Var}(X_1)=4$ and $\operatorname{Var}(X_3)=5$. Thus $X_I\sim N_2\bigl((-1,2)^\mathsf{T},\begin{psmallmatrix}4&1\\1&5\end{psmallmatrix}\bigr)$. The order $(1,3)$ is the order used in both the mean and covariance.

Selected Covariance Blocks Remain Valid

A covariance matrix must be positive semidefinite. The marginal theorem gives a selected covariance block, and the following direct argument verifies that this block is always valid, even if the original matrix or the selected block is singular.

Result: If $\Sigma$ is positive semidefinite, then $\Sigma_{II}=S_I\Sigma S_I^\mathsf{T}$ is positive semidefinite.

Proof. For any $u\in\mathbb{R}^k$, the definition of positive semidefiniteness and the fact that $S_I^\mathsf{T}u\in\mathbb{R}^p$ give $$ u^\mathsf{T}\Sigma_{II}u =u^\mathsf{T}S_I\Sigma S_I^\mathsf{T}u =(S_I^\mathsf{T}u)^\mathsf{T}\Sigma(S_I^\mathsf{T}u)\geq 0. $$ Therefore $\Sigma_{II}$ is positive semidefinite. $\square$

Positive semidefinite does not mean positive definite. Some selected coordinates can be exactly linearly related, in which case the marginal Gaussian law is itself singular.

Worked Example: A Singular Marginal Gaussian Law

Let $Z\sim N(0,1)$ and $W\sim N(0,1)$ be independent, and define $X=(1+Z,\;2+2Z,\;-1+W,\;Z+W)^\mathsf{T}$. Find the marginal law of $X_I=(X_1,X_2)^\mathsf{T}$.

The two selected coordinates are $X_1=1+Z$ and $X_2=2+2Z$. Their mean vector is $(1,2)^\mathsf{T}$. Since $\operatorname{Var}(Z)=1$, $$ \operatorname{Var}(X_1)=1,\qquad \operatorname{Cov}(X_1,X_2)=2,\qquad \operatorname{Var}(X_2)=4. $$

Consequently, $$ X_I\sim N_2\left( \begin{pmatrix}1\\2\end{pmatrix}, \begin{pmatrix}1&2\\2&4\end{pmatrix} \right). $$ The covariance matrix is singular: its determinant is $1\cdot4-2\cdot2=0$. This is also visible directly from $X_2=2X_1$ with probability one. The marginal theorem still applies; no inverse covariance matrix or ordinary two-dimensional density is needed.

Worked Example: Keeping Dependence in a Larger Selection

Let $U,\varepsilon_1,\ldots,\varepsilon_5$ be independent, with $U\sim N(0,1)$ and $\varepsilon_i\sim N(0,2)$. Set $X_i=\mu_i+U+\varepsilon_i$, where $\mu=(0,1,-1,2,4)^\mathsf{T}$. Find the law of $(X_2,X_4,X_5)^\mathsf{T}$.

For each coordinate, $\operatorname{Var}(X_i)=\operatorname{Var}(U)+\operatorname{Var}(\varepsilon_i)=1+2=3$. For distinct $i$ and $j$, the only shared random term is $U$, so $\operatorname{Cov}(X_i,X_j)=\operatorname{Var}(U)=1$. The selected mean entries, in the requested order, are $(1,2,4)^\mathsf{T}$. The selected covariance block is therefore $$ \operatorname{Cov}\begin{pmatrix}X_2\\X_4\\X_5\end{pmatrix} = \begin{pmatrix} 3&1&1\\ 1&3&1\\ 1&1&3 \end{pmatrix}. $$

Hence $(X_2,X_4,X_5)^\mathsf{T}\sim N_3\bigl((1,2,4)^\mathsf{T},\begin{psmallmatrix}3&1&1\\1&3&1\\1&1&3\end{psmallmatrix}\bigr)$. The nonzero off-diagonal entries remain because the selected measurements share the common random term $U$. Keeping only the three variances would discard their dependence and would not give the marginal covariance matrix.

Marginalizing in Stages

Selection can be performed directly or in stages. If $I=(i_1,\ldots,i_k)$ is an ordered selection and $J=(i_{r_1},\ldots,i_{r_m})$ selects some of those coordinates in their existing order, then selecting $J$ from $X_I$ gives exactly $X_J$. The same consistency holds for the parameters: the final mean and covariance are the entries of $\mu$ and $\Sigma$ at the indices in $J$, in that order. This follows because both routes retain the same coordinates, and the marginal theorem assigns each retained pair the same covariance entry $\Sigma_{i_{r_a}i_{r_b}}$.

This property is useful when a model is first reduced to a block of measurements and then to a smaller subset. It also provides a simple check on calculations: repeated selection must give the same result as selecting the final coordinates directly.

Common Mistake: Confusing Marginalization With Conditioning

A frequent error is to use a conditional-Gaussian formula when the question asks only for selected coordinates. For a marginal law, retain the chosen entries of $\mu$ and the corresponding principal submatrix of $\Sigma$. Do not subtract a covariance adjustment or invert the covariance block of the discarded coordinates. Such operations are not part of marginalization.

Another error is to treat a selected coordinate as independent of the discarded coordinates, or to drop off-diagonal entries in the retained covariance block. Selection discards variables from the description; it does not remove dependence among the variables that remain. The covariance between two retained coordinates is still their original covariance.

Key takeaway: For an ordered coordinate set $I$, the Gaussian marginal is $N_k(\mu_I,\Sigma_{II})$. Extract the mean entries and covariance rows and columns in exactly the same order. The result remains valid for singular Gaussian laws.

Check Your Understanding

Answer each question using the coordinate order stated.

If $X\sim N_5(\mu,\Sigma)$ and $I=(4,2)$, which entries form $\mu_I$, and which rows and columns form $\Sigma_{II}$?
Suppose two selected coordinates have covariance block $\begin{psmallmatrix}1&2\\2&4\end{psmallmatrix}$. Is the marginal Gaussian law nonsingular? Give a reason.
Why is $S_I\Sigma S_I^\mathsf{T}$ positive semidefinite whenever $\Sigma$ is positive semidefinite?
In the common-factor example, why are the covariances between distinct selected coordinates equal to 1 rather than 0?
State the distinction between extracting a marginal covariance block and applying a conditional-Gaussian covariance formula.