Independence of the Normal Mean and Sample Covariance

Why Separate the Mean from the Residual Variation?

In a multivariate normal sample, the sample mean describes the location of the observations, while the sample covariance describes their centered spread and dependence across coordinates. These summaries are computed from the same observations, so their independence is not automatic. The key is a transformation of the sample that separates the direction in which all observations move together from the directions that describe deviations around their mean.

This independence is useful for exact conditional calculations and for sampling distributions built from both summaries. It is a special property of the normal model: the proof depends on the factorization of a Gaussian density after an orthogonal transformation.

Setup and Definitions

Definition: Let $X_1,\ldots,X_n$ be independent random column vectors with common distribution $N_p(\mu,\Sigma)$, where $\mu\in\mathbb R^p$, $\Sigma$ is positive definite, and $n\geq 2$. The sample mean vector and centered sample covariance matrix are $$ \bar X=\frac{1}{n}\sum_{i=1}^n X_i, \qquad S=\frac{1}{n-1}\sum_{i=1}^n (X_i-\bar X)(X_i-\bar X)'. $$

Positive definiteness ensures that each observation has a nonsingular multivariate normal density. The matrix $S$ is symmetric and nonnegative definite; it need not be invertible when the sample size is small relative to $p$. Independence below does not require $S$ to be invertible.

Arrange the observations as rows of an $n\times p$ matrix $X$, so row $i$ is $X_i'$. Let $\mathbf 1_n$ denote the $n$-vector of ones. Then the matrix of centered observations is $$ X-\mathbf 1_n\bar X', $$ and $S=(X-\mathbf 1_n\bar X')'(X-\mathbf 1_n\bar X')/(n-1)$.

An Orthogonal Change of Coordinates

Choose an $n\times n$ orthogonal matrix $Q$ whose first row is $\mathbf 1_n'/\sqrt n$. Such a matrix exists by extending that unit vector to an orthonormal basis of $\mathbb R^n$. Define $Z=QX$, and write its rows as $Z_1,\ldots,Z_n$, each a row vector of length $p$. The first row is $$ Z_1=\frac{1}{\sqrt n}\sum_{i=1}^n X_i'=\sqrt n\,\bar X'. $$ The remaining rows give coordinates in directions perpendicular to $\mathbf 1_n$, which is exactly the subspace containing centered sample variation.

Theorem: Under the setup above, $\bar X$ and $S$ are independent. In fact, the first transformed row $Z_1$ is independent of the entire collection $Z_2,\ldots,Z_n$, and $$ S=\frac{1}{n-1}\sum_{j=2}^n Z_j'Z_j. $$

Proof by Density Factorization

The joint density of $X$, with respect to Lebesgue measure on $\mathbb R^{n\times p}$, is proportional to $$ |\Sigma|^{-n/2} \exp\left\{-\frac{1}{2}\operatorname{tr}\left[\Sigma^{-1}(X-\mathbf 1_n\mu')'(X-\mathbf 1_n\mu')\right]\right\}. $$ Apply the transformation $Z=QX$. Since $Q$ is orthogonal, its determinant has absolute value one, and the Jacobian for this transformation on the $n\times p$ matrix is $|\det Q|^p=1$. Also, $Q\mathbf 1_n=\sqrt n\,e_1$, where $e_1$ is the first coordinate vector. Thus the transformed centered matrix is $$ Q(X-\mathbf 1_n\mu')=Z-\sqrt n\,e_1\mu'. $$

The trace in the exponent is the sum of the row-wise quadratic forms. Keeping the row-vector dimensions explicit gives $$ \operatorname{tr}\left[\Sigma^{-1}(Z-\sqrt n\,e_1\mu')'(Z-\sqrt n\,e_1\mu')\right] =(Z_1-\sqrt n\,\mu')\Sigma^{-1}(Z_1-\sqrt n\,\mu')' +\sum_{j=2}^n Z_j\Sigma^{-1}Z_j'. $$ The first term is a quadratic form in the row vector $Z_1-\sqrt n\,\mu'$; each remaining term is a quadratic form in the row vector $Z_j$. In particular, all products have compatible dimensions.

Consequently, the transformed density factors into a function of $Z_1$ and a function of $(Z_2,\ldots,Z_n)$. The change of variables has preserved the density's normalization, so this factorization proves that $Z_1$ is independent of the remaining rows. The first row is $\sqrt n\,\bar X'$, so it is equivalent to $\bar X$ for independence purposes.

It remains to express $S$ using those remaining rows. The matrix $Q'\operatorname{diag}(0,1,\ldots,1)Q$ is the orthogonal projection $I_n-\mathbf 1_n\mathbf 1_n'/n$. Therefore $$ X-\mathbf 1_n\bar X' =\left(I_n-\frac{\mathbf 1_n\mathbf 1_n'}{n}\right)X =Q'\operatorname{diag}(0,1,\ldots,1)Z. $$ Because $Q$ is orthogonal, taking the cross-product gives $$ (X-\mathbf 1_n\bar X')'(X-\mathbf 1_n\bar X') =Z'\operatorname{diag}(0,1,\ldots,1)Z =\sum_{j=2}^n Z_j'Z_j. $$ Hence $S$ is a function of $Z_2,\ldots,Z_n$. A random variable independent of a collection is independent of every measurable function of that collection, so $\bar X$ and $S$ are independent. $\square$

A Conditional Consequence

Result: For any Borel set $A\subseteq\mathbb R^p$, $$ P(\bar X\in A\mid S)=P(\bar X\in A)\quad\text{almost surely}. $$ More generally, for any integrable measurable function $g$, $E[g(\bar X)\mid S]=E[g(\bar X)]$ almost surely.

To verify the probability statement, let $B$ be any measurable set of symmetric matrices. Independence gives $$ P(\{\bar X\in A\}\cap\{S\in B\})=P(\bar X\in A)P(S\in B). $$ This is the defining integral identity for the constant $P(\bar X\in A)$ to be a version of $P(\bar X\in A\mid S)$. The conditional-expectation statement follows by the same argument, first for indicator functions, then for simple functions, and then for integrable $g$ by the defining limit property of conditional expectation. Thus observing the sample covariance does not change the conditional distribution of the sample mean.

Worked Examples

Worked Example: Three Bivariate Observations

Suppose $X_1,X_2,X_3$ are independent $N_2(\mu,\Sigma)$ vectors, with $$ \mu=\begin{pmatrix}1\\-1\end{pmatrix}, \qquad \Sigma=\begin{pmatrix}4&1\\1&2\end{pmatrix}. $$ The determinant of $\Sigma$ is $4(2)-1(1)=7>0$, and its diagonal entries are positive, so $\Sigma$ is positive definite. Use the orthogonal matrix $$ Q=\begin{pmatrix} 1/\sqrt3&1/\sqrt3&1/\sqrt3\\ 1/\sqrt2&-1/\sqrt2&0\\ 1/\sqrt6&1/\sqrt6&-2/\sqrt6 \end{pmatrix}. $$ Its rows are orthonormal, and the first row is $\mathbf 1_3'/\sqrt3$. The transformed rows are $$ Z_1=\frac{X_1'+X_2'+X_3'}{\sqrt3}=\sqrt3\,\bar X', \quad Z_2=\frac{X_1'-X_2'}{\sqrt2}, \quad Z_3=\frac{X_1'+X_2'-2X_3'}{\sqrt6}. $$ The density factorization shows that $Z_1$ is independent of $Z_2,Z_3$, with $Z_1$ having mean $\sqrt3(1,-1)$ and covariance $\Sigma$, while each of $Z_2,Z_3$ has mean zero and covariance $\Sigma$. The sample mean therefore has distribution $N_2(\mu,\Sigma/3)$. The centered sample covariance is $$ S=\frac{1}{2}\left(Z_2'Z_2+Z_3'Z_3\right). $$ It is a function only of the two residual-coordinate rows, so it is independent of $\bar X$. This remains true even when a particular observed $S$ happens to be singular.

Worked Example: An Independence Calculation with Two Observations

Let $X_1,X_2$ be independent $N_2(0,I_2)$ vectors. Set $$ Z_1=\frac{X_1'+X_2'}{\sqrt2}, \qquad Z_2=\frac{X_1'-X_2'}{\sqrt2}. $$ These are the rows of an orthogonal transformation. They are independent $N_2(0,I_2)$ vectors. Since $n=2$, $$ \bar X'=\frac{Z_1}{\sqrt2}, \qquad S=Z_2'Z_2. $$ In particular, $\bar X_1=Z_{11}/\sqrt2$ and $S_{11}=Z_{21}^2$, where $Z_{11}$ and $Z_{21}$ are independent standard normal random variables. Therefore $$ P(\bar X_1>0,\ S_{11}>1) =P(Z_{11}>0)P(Z_{21}^2>1) =\frac12\cdot 2\{1-\Phi(1)\} =1-\Phi(1)\approx 0.1587. $$ The factorization is justified by the theorem, not merely by the fact that the two displayed summaries have zero covariance.

Worked Example: Conditioning on a Sample Covariance

Suppose $n=4$, $p=2$, $\mu=(2,-1)'$, and $$ \Sigma=\begin{pmatrix}1&1/2\\1/2&4\end{pmatrix}. $$ This matrix is positive definite because its first diagonal entry is positive and its determinant is $4-1/4=15/4>0$. The first coordinate of $\bar X$ has distribution $N(2,1/4)$, since the corresponding variance is $\Sigma_{11}/n=1/4$. Thus $P(\bar X_1>2)=1/2$. The independence result gives $$ P(\bar X_1>2\mid S)=\frac12\quad\text{almost surely}. $$ For another check, the transformed residual rows have first coordinates that are three independent $N(0,1)$ variables. Hence $S_{11}$ is their squared sum divided by $3$, so $$ P(\bar X_1>2,\ S_{11}>1) =\frac12\,P\left(\chi^2_3>3\right). $$ The first factor comes from the sample mean and the second from the centered residual coordinates.

Scope and a Common Pitfall

The theorem separates the mean direction from the centered residual directions. It does not assert that the sample mean is independent of every observation, nor that the sample covariance is independent of the original sample. Both summaries are functions of the data, and the result is a particular independence induced by the Gaussian density and the orthogonal decomposition.

Incorrect reasoning: “The mean and sample covariance have zero covariance, so they are independent.” Zero covariance alone does not imply independence for general random variables or random matrices.
Correction: The proof establishes independence by factoring the joint density of the transformed Gaussian coordinates. Gaussian structure is essential to that argument.
Another scope error: Applying the conclusion to a nonnormal sample just because the observations are independent and have finite covariance is not justified. The density factorization used here need not hold outside the multivariate normal model.

This result is a foundation for exact multivariate sampling theory. It allows calculations involving a mean statistic to be separated from calculations involving centered covariance, even though both are computed from the same observations. That separation is especially useful when deriving distributions of statistics that combine these quantities.

Check Your Understanding

Try these questions using the orthogonal-coordinate argument.

Why must the first row of $Q$ be $\mathbf 1_n'/\sqrt n$ to make the first transformed row a scaled sample mean?
Explain why the density factorization proves independence of $Z_1$ and $(Z_2,\ldots,Z_n)$.
For $n=2$, express $S$ in terms of the single residual-coordinate row $Z_2$.
Does the theorem require the sample covariance matrix to be invertible? Explain.
Why is zero covariance alone not an adequate proof of independence in this setting?