Sampling Distribution of the Multivariate Sample Mean

Why the Distribution of the Sample Mean Matters

A sample mean is a vector summary: it estimates several population means at once. Its coordinates do not generally vary independently. When measurements on different variables are correlated, the uncertainty in the sample mean includes that same dependence, reduced by the sample size. Knowing its exact distribution lets us calculate the distribution of any linear summary and compare mean vectors from independent groups.

Here the observations are assumed to be independent and multivariate normal. This assumption gives an exact finite-sample result, not merely an approximation for large samples. The result concerns the sample mean itself; it does not require estimating a covariance matrix from the data.

Definition: Let $X_1,\ldots,X_n$ be independent random vectors, each with distribution $N_p(\mu,\Sigma)$, where $\mu$ is a $p$-dimensional mean vector and $\Sigma$ is a $p\times p$ positive-definite covariance matrix. The multivariate sample mean is $$ \bar X=\frac{1}{n}\sum_{i=1}^n X_i. $$ For a random vector $Z$, $Z\sim N_p(a,V)$ means that its characteristic function at $t\in\mathbb R^p$ is $\exp\{it^{\mathsf T}a-\tfrac12t^{\mathsf T}Vt\}$. The covariance matrix $V$ may be singular for a linear transformation of a normal vector.

Exact Distribution of the Sample Mean

The mean of $\bar X$ is $\mu$, and its covariance matrix is $\Sigma/n$. For the covariance calculation, independence makes the cross-covariance terms between distinct observations zero:

E[\bar X]=\frac{1}{n}\sum_{i=1}^n E[X_i]=\mu,\qquad \operatorname{Cov}(\bar X) =\frac{1}{n^2}\sum_{i=1}^n\operatorname{Cov}(X_i) =\frac{\Sigma}{n}.

These moments alone do not establish a distribution. Normality follows by multiplying characteristic functions, as the next result shows.

Theorem: If $X_1,\ldots,X_n$ are independent and each has distribution $N_p(\mu,\Sigma)$, then $$ \bar X\sim N_p\left(\mu,\frac{\Sigma}{n}\right). $$

Proof. For any $t\in\mathbb R^p$, independence and the characteristic function of $N_p(\mu,\Sigma)$ give

\begin{aligned} \varphi_{\bar X}(t) &=E\!\left[\exp\left\{it^{\mathsf T}\frac{1}{n}\sum_{i=1}^n X_i\right\}\right]\\ &=\prod_{i=1}^n E\!\left[\exp\left\{i(t/n)^{\mathsf T}X_i\right\}\right]\\ &=\left[ \exp\left\{i(t/n)^{\mathsf T}\mu-\frac12(t/n)^{\mathsf T}\Sigma(t/n)\right\} \right]^n\\ &=\exp\left\{it^{\mathsf T}\mu-\frac12t^{\mathsf T}\frac{\Sigma}{n}t\right\}. \end{aligned}

This is the characteristic function of $N_p(\mu,\Sigma/n)$. Since a probability distribution is uniquely determined by its characteristic function, the stated distribution follows. $\square$

Worked Example: The Mean of a Bivariate Normal Sample

Suppose $n=16$ independent observations have distribution $N_2(\mu,\Sigma)$, with $ \mu=(10,20)^{\mathsf T} $ and $ \Sigma=\begin{pmatrix}4&1.2\\1.2&9\end{pmatrix} $. Find the exact distribution of the sample mean and its coordinate standard errors.

The theorem gives $\bar X\sim N_2(\mu,\Sigma/16)$. Divide every entry of the covariance matrix by 16:

\frac{\Sigma}{16} = \begin{pmatrix} 4/16&1.2/16\\ 1.2/16&9/16 \end{pmatrix} = \begin{pmatrix} 0.25&0.075\\ 0.075&0.5625 \end{pmatrix}.

Thus $\bar X\sim N_2\left((10,20)^{\mathsf T}, \begin{pmatrix}0.25&0.075\\0.075&0.5625\end{pmatrix}\right)$. The first coordinate has standard error $\sqrt{0.25}=0.5$, and the second has standard error $\sqrt{0.5625}=0.75$. Their covariance is $0.075$, so the two sample-mean coordinates are not independent. The covariance scales by $1/n$; the standard errors scale by $1/\sqrt n$.

Linear Contrasts of the Sample Mean

Often the target is not the entire mean vector but a linear combination of its coordinates. For a fixed $q\times p$ matrix $A$, the transformed mean $A\bar X$ is a $q$-dimensional vector. A single contrast is the special case $q=1$, with $A=c^{\mathsf T}$ for a fixed vector $c$.

Theorem: Under the assumptions above, for any fixed $q\times p$ matrix $A$, $$ A\bar X\sim N_q\left(A\mu,\frac{A\Sigma A^{\mathsf T}}{n}\right). $$ In particular, for a fixed $c\in\mathbb R^p$, $$ c^{\mathsf T}\bar X\sim N\left(c^{\mathsf T}\mu,\frac{c^{\mathsf T}\Sigma c}{n}\right). $$

Proof. For $s\in\mathbb R^q$, the characteristic function of $A\bar X$ is

\begin{aligned} E[\exp\{is^{\mathsf T}A\bar X\}] &=\varphi_{\bar X}(A^{\mathsf T}s)\\ &=\exp\left\{is^{\mathsf T}A\mu -\frac12s^{\mathsf T}\frac{A\Sigma A^{\mathsf T}}{n}s\right\}. \end{aligned}

This is the characteristic function of the stated $q$-variate normal distribution, including when $A\Sigma A^{\mathsf T}$ is singular. Taking $q=1$ and $A=c^{\mathsf T}$ gives the scalar contrast formula. $\square$

Worked Example: A Weighted Contrast

Continue with the bivariate model in the previous example. Find the distribution of $2\bar X_1+\bar X_2$, where $\bar X_1$ and $\bar X_2$ are the two coordinates of $\bar X$.

Set $c=(2,1)^{\mathsf T}$. The contrast mean is $c^{\mathsf T}\mu=2(10)+1(20)=40$. For its variance, first use the population covariance:

c^{\mathsf T}\Sigma c =2^2(4)+2(2)(1)(1.2)+1^2(9) =16+4.8+9=29.8.

Divide by $n=16$, giving variance $29.8/16=1.8625$. Therefore $2\bar X_1+\bar X_2\sim N(40,1.8625)$, with standard deviation $\sqrt{1.8625}$, approximately $1.365$. The covariance term contributes $4.8$ before division by $n$; omitting it would give an incorrect variance.

Differences Between Independent Sample Means

Suppose two independent groups provide observations $X_1,\ldots,X_{n_1}$ and $Y_1,\ldots,Y_{n_2}$, with $X_i\sim N_p(\mu_1,\Sigma_1)$ and $Y_j\sim N_p(\mu_2,\Sigma_2)$. Independence is required both within each group and between the groups. Applying the sample-mean result separately gives the distributions of $\bar X$ and $\bar Y$; their independence then determines the distribution of their difference.

Theorem: For independent samples with the distributions just specified, $$ \bar X-\bar Y\sim N_p\left(\mu_1-\mu_2,\frac{\Sigma_1}{n_1}+\frac{\Sigma_2}{n_2}\right). $$ If both populations have covariance $\Sigma$, the covariance of the difference is $\left(\frac{1}{n_1}+\frac{1}{n_2}\right)\Sigma$.

Derivation. For $t\in\mathbb R^p$, the characteristic function of the difference factors because $\bar X$ and $\bar Y$ are independent. The characteristic function of $-\bar Y$ is obtained by evaluating that of $\bar Y$ at $-t$. Thus

\begin{aligned} \varphi_{\bar X-\bar Y}(t) &=\varphi_{\bar X}(t)\varphi_{\bar Y}(-t)\\ &=\exp\left\{it^{\mathsf T}\mu_1-\frac12t^{\mathsf T}\frac{\Sigma_1}{n_1}t\right\} \exp\left\{-it^{\mathsf T}\mu_2-\frac12t^{\mathsf T}\frac{\Sigma_2}{n_2}t\right\}\\ &=\exp\left\{it^{\mathsf T}(\mu_1-\mu_2) -\frac12t^{\mathsf T}\left(\frac{\Sigma_1}{n_1}+\frac{\Sigma_2}{n_2}\right)t\right\}. \end{aligned}

This is the characteristic function of the asserted multivariate normal distribution, proving the result. $\square$

Worked Example: Difference of Two Independent Group Means

Two independent groups have common covariance $ \Sigma=\begin{pmatrix}4&1\\1&9\end{pmatrix} $. The first group has $n_1=10$ and mean $\mu_1=(5,2)^{\mathsf T}$; the second has $n_2=15$ and mean $\mu_2=(3,1)^{\mathsf T}$. Find the distribution of $\bar X-\bar Y$.

The mean difference is $(5-3,2-1)^{\mathsf T}=(2,1)^{\mathsf T}$. The covariance multiplier is $1/10+1/15=3/30+2/30=1/6$, so

\operatorname{Cov}(\bar X-\bar Y) =\frac{1}{6}\begin{pmatrix}4&1\\1&9\end{pmatrix} =\begin{pmatrix}2/3&1/6\\1/6&3/2\end{pmatrix}.

Consequently, $\bar X-\bar Y\sim N_2\left((2,1)^{\mathsf T}, \begin{pmatrix}2/3&1/6\\1/6&3/2\end{pmatrix}\right)$. For example, the difference in the first coordinate has variance $2/3$, and the difference in the second has variance $3/2$. Their covariance is $1/6$. Independence of the groups is what permits adding the two sample-mean covariance matrices.

Interpretation and a Common Pitfall

The exact normal law is useful because it translates uncertainty in a vector mean into uncertainty in any fixed linear summary. The covariance matrix $\Sigma/n$ also shows how the precision changes as the sample grows: every variance is divided by $n$, while each standard error is divided by $\sqrt n$. For two independent groups, both sample sizes and both covariance matrices contribute to the uncertainty in the difference.

Common mistake: subtracting covariances or treating coordinates as independent. It may seem that the variance of a difference should be a difference of variances, or that a contrast's variance is just the sum of squared weights times coordinate variances. Neither is generally correct. For independent random vectors, covariance matrices add when forming a difference; within a contrast, covariance terms also contribute. For example, $c^{\mathsf T}\Sigma c$ includes the off-diagonal entries of $\Sigma$, as the weighted-contrast calculation showed. If the two group means are dependent, their cross-covariance terms must also be included, and the independent-sample theorem above does not apply.

Another important distinction is between a known population covariance and an estimated one. These results specify the exact distribution when the sampling covariance matrices are specified. Replacing them with estimated covariance matrices changes the inferential problem; the normal law stated here does not by itself justify treating estimated quantities as known.

Check Your Understanding

Use the definitions and results above to answer these questions.

If the sample size is multiplied by four, what happens to the covariance matrix and to each coordinate standard error of the sample mean?
For a fixed vector $c$, explain why the variance of $c^{\mathsf T}\bar X$ can depend on off-diagonal entries of $\Sigma$.
Two independent samples have covariance matrices $\Sigma_1$ and $\Sigma_2$ and sizes $n_1$ and $n_2$. State the mean and covariance of their difference in sample means.
What assumption allows the characteristic function of a difference of sample means to factor into two characteristic functions?
If $A$ is not full rank, does the linear transformation $A\bar X$ still have a multivariate normal distribution? What may happen to its covariance matrix?