Independent Sums of Gaussian Vectors

Why Independent Gaussian Sums Are Gaussian

Sums of independent Gaussian quantities appear throughout statistical theory: totals from independent sources, combined measurement errors, and sums of random vectors in sampling models. In the multivariate setting, the coordinates within any one vector may be correlated, and its covariance matrix may even be singular. The useful fact is that independence between the vectors lets their characteristic functions multiply, producing another Gaussian characteristic function.

The calculation of the mean and covariance is related but logically separate. Means add directly, while covariance matrices add because independence makes all cross-covariance terms zero. The characteristic-function argument establishes more than those two moment calculations: it identifies the entire distribution of the sum.

Gaussian Vectors and Their Characteristic Functions

For a random vector $X$ in $\mathbb{R}^d$, its characteristic function is the complex-valued function of $t\in\mathbb{R}^d$ defined by $\varphi_X(t)=E[\exp(i t^\top X)]$, where $i^2=-1$. The characteristic function exists for every probability distribution because the integrand has absolute value one.

Definition: A random vector $X\in\mathbb{R}^d$ has a multivariate Gaussian distribution with mean vector $\mu\in\mathbb{R}^d$ and covariance matrix $\Sigma$ if its characteristic function is $$ \varphi_X(t)=\exp\left(i t^\top\mu-\frac{1}{2}t^\top\Sigma t\right),\qquad t\in\mathbb{R}^d, $$ where $\Sigma$ is symmetric and positive semidefinite. We write $X\sim N_d(\mu,\Sigma)$. Positive semidefinite matrices are permitted, so this definition includes singular Gaussian distributions.

This characteristic function follows from the Gaussian construction. If $Z$ has independent standard normal coordinates and $\Sigma=AA^\top$, then $X=\mu+AZ$ has characteristic function $E[\exp(i t^\top\mu+i t^\top AZ)]$. Independence of the coordinates of $Z$, and the standard normal characteristic function $\exp(-s^2/2)$, give $\exp(i t^\top\mu-\frac12 t^\top AA^\top t)$, as claimed. A positive semidefinite $\Sigma$ admits such a factor $A$, even when $A$ is rank deficient.

We will also use the uniqueness theorem for characteristic functions: if two random vectors in $\mathbb{R}^d$ have the same characteristic function at every $t\in\mathbb{R}^d$, they have the same distribution. Thus matching the displayed Gaussian form identifies a distribution, not merely its first two moments.

The Independent-Sum Theorem

Theorem: Let $X_1,\ldots,X_m$ be mutually independent random vectors in $\mathbb{R}^d$, with $X_j\sim N_d(\mu_j,\Sigma_j)$, where each $\Sigma_j$ is symmetric and positive semidefinite. Then $$ \sum_{j=1}^m X_j\sim N_d\left(\sum_{j=1}^m\mu_j,\ \sum_{j=1}^m\Sigma_j\right). $$ In particular, the mean vectors and covariance matrices add, and the sum is Gaussian even when one or more of its covariance matrices are singular.

Proof. Set $S=\sum_{j=1}^m X_j$. Mutual independence implies that the characteristic function of the sum is the product of the individual characteristic functions. Therefore, for every $t\in\mathbb{R}^d$,

\begin{aligned} \varphi_S(t) &=\prod_{j=1}^m\varphi_{X_j}(t)\\ &=\prod_{j=1}^m \exp\left(i t^\top\mu_j-\frac12 t^\top\Sigma_jt\right)\\ &=\exp\left(i t^\top\sum_{j=1}^m\mu_j -\frac12 t^\top\left(\sum_{j=1}^m\Sigma_j\right)t\right). \end{aligned}

The matrix $\sum_j\Sigma_j$ is symmetric and positive semidefinite: for every $v\in\mathbb{R}^d$, $v^\top(\sum_j\Sigma_j)v=\sum_jv^\top\Sigma_jv\geq 0$. The final expression is consequently the characteristic function of the Gaussian distribution with the stated mean and covariance. Uniqueness of characteristic functions proves the distributional claim. $\square$

For completeness, the moment calculations can be checked directly. Each vector has finite second moments, so $S$ does as well. Linearity of expectation gives $E[S]=\sum_j\mu_j$. Expanding the covariance of the sum yields

\operatorname{Var}(S) =\sum_{j=1}^m\operatorname{Var}(X_j) +\sum_{\substack{j,k=1\\j\ne k}}^m E\left[(X_j-\mu_j)(X_k-\mu_k)^\top\right].

For $j\ne k$, independence makes $X_j-\mu_j$ and $X_k-\mu_k$ independent, and each has expectation zero. Entry by entry, the expectation of their outer product is therefore zero. The cross terms vanish, leaving $\operatorname{Var}(S)=\sum_j\Sigma_j$. This also shows why independence matters to the covariance calculation.

Worked Example: Adding Two Correlated Bivariate Gaussian Vectors

Suppose $X$ and $Y$ are independent, with $X\sim N_2((1,-1)^\top,\begin{pmatrix}4&1\\1&3\end{pmatrix})$ and $Y\sim N_2((2,3)^\top,\begin{pmatrix}1&-0.5\\-0.5&2\end{pmatrix})$. Find the distribution of $S=X+Y$.

First add the mean vectors: \[ E[S]=(1,-1)^\top+(2,3)^\top=(3,2)^\top. \] Then add the covariance matrices, entry by entry:

\operatorname{Var}(S)= \begin{pmatrix}4&1\\1&3\end{pmatrix} + \begin{pmatrix}1&-0.5\\-0.5&2\end{pmatrix} = \begin{pmatrix}5&0.5\\0.5&5\end{pmatrix}.

The off-diagonal entries are covariances, not variances, so they also add. As a check, the resulting covariance matrix is positive definite: its determinant is $5\cdot5-0.5^2=24.75>0$, and its first diagonal entry is positive. By the theorem, $S\sim N_2((3,2)^\top,\begin{pmatrix}5&0.5\\0.5&5\end{pmatrix})$. The coordinates of $S$ are not independent, since their covariance is $0.5$, not zero.

Calculations With Contrasts and More Than Two Summands

Once the sum’s vector distribution is known, scalar linear combinations can be read from its mean and covariance. If $S$ has mean $\mu_S$ and covariance $\Sigma_S$, then a fixed vector $a\in\mathbb{R}^d$ gives a scalar $a^\top S$ with mean $a^\top\mu_S$ and variance $a^\top\Sigma_Sa$. This is especially convenient for checking a contrast or total without recalculating the full distribution.

Worked Example: A Contrast of a Sum of Three Vectors

Let $X_1,X_2,X_3$ be mutually independent bivariate Gaussian vectors with means $(1,0)^\top,(0,2)^\top,(-1,1)^\top$, respectively, and covariance matrices \[ \Sigma_1=\begin{pmatrix}1&0\\0&2\end{pmatrix},\quad \Sigma_2=\begin{pmatrix}2&1\\1&2\end{pmatrix},\quad \Sigma_3=\begin{pmatrix}1&-1\\-1&3\end{pmatrix}. \] Find the distribution of $S=X_1+X_2+X_3$, then find the mean and variance of $D=S_1-S_2$.

Adding means gives \[ \mu_S=(1,0)^\top+(0,2)^\top+(-1,1)^\top=(0,3)^\top. \] Adding covariance matrices gives

\Sigma_S=\Sigma_1+\Sigma_2+\Sigma_3 =\begin{pmatrix}4&0\\0&7\end{pmatrix}.

Hence $S\sim N_2((0,3)^\top,\Sigma_S)$. Let $a=(1,-1)^\top$, so $D=a^\top S$. Its mean is $a^\top\mu_S=1\cdot0+(-1)\cdot3=-3$. Its variance is

\operatorname{Var}(D)=a^\top\Sigma_Sa =\begin{pmatrix}1&-1\end{pmatrix} \begin{pmatrix}4&0\\0&7\end{pmatrix} \begin{pmatrix}1\\-1\end{pmatrix} =4+7=11.

Thus $D\sim N(-3,11)$. This scalar calculation is consistent with subtracting the two coordinates of $S$: their variances are 4 and 7 and their covariance is zero, so the variance of the difference is $4+7-2(0)=11$.

Singular Covariance Matrices Are Included

A Gaussian sum need not have a density with respect to ordinary $d$-dimensional volume. If all summands vary only along the same lower-dimensional subspace, their sum can remain confined to that subspace. The characteristic-function theorem requires positive semidefinite covariance matrices, not positive definite ones, so it applies without modification.

Worked Example: A Sum Supported on a Line

Let $Z$ and $W$ be independent standard normal random variables, and define $X=(1+2Z,1+2Z)^\top$ and $Y=(2+W,2+W)^\top$. Find the law and support of $S=X+Y$.

The mean vectors are $\mu_X=(1,1)^\top$ and $\mu_Y=(2,2)^\top$. Since the random part of $X$ is $2Z(1,1)^\top$, its covariance is \[ \Sigma_X=4\begin{pmatrix}1&1\\1&1\end{pmatrix}. \] Similarly, $\Sigma_Y=\begin{pmatrix}1&1\\1&1\end{pmatrix}$. The vectors $X$ and $Y$ are independent because they are functions of independent $Z$ and $W$. The independent-sum theorem therefore gives

S\sim N_2\left( \begin{pmatrix}3\\3\end{pmatrix}, \begin{pmatrix}5&5\\5&5\end{pmatrix}\right).

The covariance matrix has determinant $25-25=0$, so it is singular. Directly, $S=(3+2Z+W,3+2Z+W)^\top$; its coordinates are equal with probability one. Its support is the line $\{(u,u)^\top:u\in\mathbb{R}\}$, rather than the entire plane. The characteristic function confirms the same conclusion: for $t=(t_1,t_2)^\top$, \[ \varphi_S(t)=\exp\left(3i(t_1+t_2)-\frac52(t_1+t_2)^2\right), \] which is precisely the Gaussian characteristic function with the displayed mean and covariance.

Common Pitfall: Covariance Addition Is Not a Gaussianity Argument

A tempting shortcut is to calculate a candidate mean and covariance and then conclude that a sum must be Gaussian. Those two moments alone do not identify a general probability distribution. Here, Gaussianity follows because the summands are Gaussian and independent: independence gives the product formula for characteristic functions, and that product has the Gaussian form. The moment calculation checks the parameters but does not replace that distributional argument.

A second mistake is to add covariance matrices when the summands are not independent. In general, for two vectors with finite second moments, \[ \operatorname{Var}(X+Y)=\operatorname{Var}(X)+\operatorname{Var}(Y) +\operatorname{Cov}(X,Y)+\operatorname{Cov}(Y,X). \] Independence makes both cross-covariance matrices zero. Without it, the covariance calculation needs those terms, and the characteristic functions do not in general factor into a product.

Finally, independence here is between the whole random vectors $X_1,\ldots,X_m$. Coordinates within an individual vector need not be independent; their dependence is recorded in the off-diagonal entries of that vector’s covariance matrix. Replacing vector independence with a claim about coordinate-wise properties changes the assumptions and does not justify the theorem’s factorization.

Key takeaway: For mutually independent $d$-dimensional Gaussian vectors, the sum is Gaussian with mean $\sum_j\mu_j$ and covariance $\sum_j\Sigma_j$. Independence supplies both the characteristic-function factorization and the vanishing cross-covariances; positive definiteness is not required.

Check Your Understanding

Use the characteristic-function argument and covariance rules developed above.

Two independent Gaussian vectors have means $\mu_1,\mu_2$ and covariance matrices $\Sigma_1,\Sigma_2$. Write the characteristic function of their sum.
Why does the independent-sum theorem allow a singular covariance matrix?
For independent vectors with covariance matrices $\begin{pmatrix}2&-1\\-1&3\end{pmatrix}$ and $\begin{pmatrix}1&2\\2&5\end{pmatrix}$, calculate the covariance matrix of their sum.
If the summands are not independent, which terms may need to be added to the covariance of a two-vector sum?
In the line-supported example, what feature of the covariance matrix signals that the distribution cannot have a full-dimensional density?