Gaussian Sample Means And Centered Scatter

Why the Mean and Scatter Must Be Studied Together

A multivariate sample contains two different kinds of information. Its mean describes the sample's location, while its centered scatter describes how observations vary around that location. In a Gaussian model, these quantities have especially useful sampling laws: the mean is Gaussian, the scatter is Wishart, and the two are independent. The reduction from $n$ to $n-1$ degrees of freedom in the scatter is a consequence of estimating the mean from the same observations.

Let $X_1,\ldots,X_n$ be independent and identically distributed $p$-dimensional random vectors with $X_i\sim N_p(\mu,\Sigma)$, where $\mu\in\mathbb R^p$ and $\Sigma$ is positive definite. Define the sample mean $\bar X$, the centered scatter matrix $A$, and the sample covariance matrix $S$ as follows.

\bar X=\frac{1}{n}\sum_{i=1}^n X_i,\qquad A=\sum_{i=1}^n(X_i-\bar X)(X_i-\bar X)^T,\qquad S=\frac{A}{n-1}.

The matrix $A$ is the centered scatter; $S$ is its usual unbiased sample-covariance version. The Wishart distribution can be defined by the construction $W=\sum_{j=1}^{\nu} Z_jZ_j^T$, where the $Z_j$ are independent $N_p(0,\Sigma)$ vectors. In that case, $W\sim W_p(\Sigma,\nu)$. This construction also permits a singular Wishart matrix when the degrees of freedom are below $p$.

The Joint Sampling Law

Theorem: If $X_1,\ldots,X_n$ are independent $N_p(\mu,\Sigma)$ vectors with $\Sigma$ positive definite and $n\geq2$, then $$ \bar X\sim N_p\!\left(\mu,\frac{\Sigma}{n}\right), \qquad A\sim W_p(\Sigma,n-1), \qquad \bar X\ \text{and}\ A\ \text{are independent}. $$ Equivalently, the joint law of $(\bar X,A)$ is the product of these two marginal laws. Therefore $S=A/(n-1)$ is also independent of $\bar X$, and $E[S]=\Sigma$.

The theorem specifies the degrees of freedom as well as the distributional family. The covariance matrix of the mean is $\Sigma/n$, whereas the scatter has the Wishart construction with $n-1$ independent Gaussian vectors. Independence is a statement about the two random quantities across repeated samples; it does not mean that the individual centered observations are independent of one another.

Proof by a Change of Gaussian Coordinates

Write the centered observations relative to the population mean as $Y_i=X_i-\mu$, and place their transposes in the rows of an $n\times p$ matrix $Y$. Choose an orthogonal $n\times n$ matrix $Q$ whose first row is $n^{-1/2}(1,\ldots,1)$. Such a matrix exists by completing this unit vector to an orthonormal basis. Let $V=QY$, and write its rows as $V_1^T,\ldots,V_n^T$, where each $V_j\in\mathbb R^p$.

The joint density of the rows of $Y$ is

(2\pi)^{-np/2}|\Sigma|^{-n/2} \exp\!\left\{-\frac12\operatorname{tr}(\Sigma^{-1}Y^TY)\right\}.

The transformation $Y\mapsto QY$ has absolute Jacobian one: it applies an orthogonal transformation to each of the $p$ columns. Also, $V^TV=Y^TQ^TQY=Y^TY$. Thus the transformed density has the same expression with $V$ in place of $Y$. Since $V^TV=\sum_{j=1}^n V_jV_j^T$, the density factors into $n$ $N_p(0,\Sigma)$ densities. In particular, $V_1,\ldots,V_n$ are independent $N_p(0,\Sigma)$ vectors.

The first transformed row is tied to the sample mean:

V_1=\frac{1}{\sqrt n}\sum_{i=1}^n(X_i-\mu) =\sqrt n(\bar X-\mu).

It follows that $\bar X\sim N_p(\mu,\Sigma/n)$. To identify the scatter, let $\mathbf 1$ be the $n$-vector of ones and $H=I_n-\mathbf 1\mathbf 1^T/n$. This matrix subtracts the sample mean from each row. The rows of $Q$ after the first are orthogonal to $\mathbf 1$, so $H=Q^T\operatorname{diag}(0,1,\ldots,1)Q$. Consequently,

A=Y^THY =V^T\operatorname{diag}(0,1,\ldots,1)V =\sum_{j=2}^n V_jV_j^T.

The equality $A=Y^THY$ holds because subtracting $\mu$ first does not change deviations from the sample mean. The final sum is a Wishart construction with $n-1$ independent $N_p(0,\Sigma)$ vectors. It depends only on $V_2,\ldots,V_n$, whereas $\bar X$ depends only on $V_1$. Those rows are independent, proving the asserted independence.

Finally, $E[V_jV_j^T]=\Sigma$ for every $j$, so the representation gives $E[A]=(n-1)\Sigma$. Dividing by $n-1$ yields $E[S]=\Sigma$. This establishes the stated unbiasedness as well as the joint law. $\square$

Worked Sampling Calculations

Worked Example: A Two-Variable Sample

Suppose $n=6$ observations are independently drawn from $N_2(\mu,\Sigma)$, with \[ \mu=\begin{pmatrix}12\\50\end{pmatrix}, \qquad \Sigma=\begin{pmatrix}4&1\\1&9\end{pmatrix}. \] Find the sampling distribution of the mean and scatter, and the expectation of the scatter.

The mean has covariance $\Sigma/n$, so \[ \bar X\sim N_2\!\left( \begin{pmatrix}12\\50\end{pmatrix}, \begin{pmatrix}4/6&1/6\\1/6&9/6\end{pmatrix} \right) = N_2\!\left( \begin{pmatrix}12\\50\end{pmatrix}, \begin{pmatrix}2/3&1/6\\1/6&3/2\end{pmatrix} \right). \]

The scatter has $n-1=5$ degrees of freedom, so $A\sim W_2(\Sigma,5)$, independently of $\bar X$. From the Wishart construction, $E[A]=5\Sigma$, hence \[ E[A]=\begin{pmatrix}20&5\\5&45\end{pmatrix}, \qquad E[S]=E[A/5]=\begin{pmatrix}4&1\\1&9\end{pmatrix}=\Sigma. \] The off-diagonal expectation is the population covariance, not zero.

Worked Example: Recovering the Univariate Chi-Square Result

Let $X_1,\ldots,X_4$ be independent $N(3,4)$ observations. Determine the distributions of their mean and centered sum of squares, and state whether they are independent.

Here $p=1$, $\mu=3$, $\Sigma=4$, and $n=4$. Thus \[ \bar X\sim N(3,4/4)=N(3,1). \] The centered scatter $A=\sum_{i=1}^4(X_i-\bar X)^2$ is $W_1(4,3)$. To express it in familiar scalar terms, the Wishart construction gives $A$ the same law as $4\sum_{j=1}^3 Z_j^2$, with independent $Z_j\sim N(0,1)$. Therefore \[ \frac{A}{4}\sim\chi^2_3, \qquad \frac{A}{3}\text{ is the sample variance}. \]

The mean and $A$ are independent by the theorem. As a check on the scale, $E[A]=4\cdot3=12$, so the sample variance has expectation $12/3=4$, the population variance.

Worked Example: Matching an Observed Scatter to Its Sampling Law

Consider three recorded two-dimensional observations $(1,0)^T,(0,1)^T,(-1,-1)^T$. For illustration, suppose the sampling model is $X_i\overset{\text{iid}}{\sim}N_2(0,I_2)$. Compute the sample mean and centered scatter, then give their sampling laws under the model.

The observed mean is \[ \bar x=\frac13\begin{pmatrix}(1+0-1)\\(0+1-1)\end{pmatrix} =\begin{pmatrix}0\\0\end{pmatrix}. \] Since the mean is zero in this observed sample, the centered scatter is the sum of the three outer products: \[ A=\begin{pmatrix}1&0\\0&0\end{pmatrix} +\begin{pmatrix}0&0\\0&1\end{pmatrix} +\begin{pmatrix}1&1\\1&1\end{pmatrix} =\begin{pmatrix}2&1\\1&2\end{pmatrix}. \]

The sampling law concerns repetitions of the experiment, not just this one observed matrix. With $n=3$, $\mu=0$, and $\Sigma=I_2$, it is \[ \bar X\sim N_2(0,I_2/3), \qquad A\sim W_2(I_2,2), \] and these random quantities are independent. The sample covariance for the displayed data is $S=A/2$. The observed value of $A$ is one realization; it does not alter the degrees of freedom.

Why Gaussianity Matters

Independence here is not a consequence of merely having independent observations. The proof uses the fact that an orthogonal change of coordinates preserves the Gaussian density and makes the transformed rows independent. For non-Gaussian observations, the mean and centered scatter need not be independent.

For example, take two independent Bernoulli observations with success probability $1/2$. If their sample mean is $0$ or $1$, their centered scatter is $0$; if their sample mean is $1/2$, their centered scatter is $1/2$. Thus the scatter is determined by the mean in this example, so they cannot be independent.

Common pitfall: “The observations are independent, so their sample mean and sample scatter are independent.” This reasoning is false. Independence of the sample mean and centered scatter is guaranteed by the Gaussian sampling model, not by independence of the observations alone.

The result is useful because it separates inference about location from inference about dispersion under the Gaussian model. It also explains why the sample covariance uses $n-1$: centering removes the component of the sample in the all-ones direction, leaving $n-1$ independent Gaussian coordinates to generate the scatter. This argument gives the sampling law directly without requiring a density for the Wishart matrix, which may be singular when its degrees of freedom are too small.

Check Your Understanding

Use the Gaussian sampling assumptions and distinguish the observed matrices from their repeated-sampling laws.

For an independent $N_p(\mu,\Sigma)$ sample of size $n$, state the law and covariance matrix of $\bar X$.
Why does the centered scatter have $n-1$, rather than $n$, Wishart degrees of freedom?
What feature of the transformed Gaussian rows establishes independence of the sample mean and scatter?
If $A\sim W_p(\Sigma,n-1)$, what is the expectation of $A/(n-1)$?
Does independence of observations alone guarantee independence of their sample mean and centered scatter? Explain.