Sampling Laws of the Sample Mean and Covariance

Why These Sampling Laws Matter

In a one-sample multivariate analysis, the sample mean estimates a population mean vector, while the sample covariance describes variation around that estimate. To make exact statements about either statistic, we need their sampling distributions: how they vary across repeated samples from the same population. Under Gaussian sampling, these distributions have especially useful forms, and the sample mean and sample covariance are independent. That independence is a specific consequence of the Gaussian model, not a general property of sample averages and variances.

Let \(\mathbf X_1,\ldots,\mathbf X_n\) be independent and identically distributed \(p\)-dimensional random vectors with distribution \(N_p(\boldsymbol\mu,\boldsymbol\Sigma)\), where \(\boldsymbol\mu\in\mathbb R^p\) and \(\boldsymbol\Sigma\) is positive definite. Assume \(n\geq 2\). Define the sample mean and unbiased sample covariance by

The denominator \(n-1\) in \(\mathbf S\) is part of the definition used here. It makes \(\mathbf S\) unbiased for \(\boldsymbol\Sigma\), and it determines which multiple of \(\mathbf S\) has a Wishart distribution.

The Exact Gaussian Sampling Laws

With this convention, the exact sampling result is:

Deriving the Mean Law and the Wishart Law

The sample mean is a linear combination of independent Gaussian vectors. Its expectation is \(E[\bar{\mathbf X}]=\boldsymbol\mu\). Independence gives its covariance:

A linear transformation of a jointly Gaussian vector is Gaussian, so these moments identify its distribution. This proves the sample-mean part of the theorem.

For the covariance law and the independence claim, arrange the centered observations as the rows of an \(n\times p\) matrix \(\mathbf A\), with row \(i\) equal to \((\mathbf X_i-\boldsymbol\mu)^T\). Choose an orthogonal \(n\times n\) matrix \(\mathbf Q\) whose first row is \((1/\sqrt n,\ldots,1/\sqrt n)\), and write the rows of \(\mathbf Q\mathbf A\) as \(\mathbf Y_1^T,\ldots,\mathbf Y_n^T\). Orthogonality preserves the sum of outer products:

The first transformed row is \(\mathbf Y_1=\sqrt n(\bar{\mathbf X}-\boldsymbol\mu)\). The rows \(\mathbf Y_j^T\) of \(\mathbf Q\mathbf A\) are independent \(N_p(\mathbf 0,\boldsymbol\Sigma)\) vectors. To verify this independence, the joint density of \(\mathbf A\) is proportional to \(\exp\{-\tfrac12\operatorname{tr}(\boldsymbol\Sigma^{-1}\mathbf A^T\mathbf A)\}\). The transformation \(\mathbf A\mapsto\mathbf Q\mathbf A\) has absolute Jacobian one and preserves \(\mathbf A^T\mathbf A\). Consequently the transformed joint density is proportional to

which factors into the \(n\) Gaussian densities. Thus the transformed rows are independent. This argument also shows that \(\mathbf Y_1\) is independent of the collection \(\mathbf Y_2,\ldots,\mathbf Y_n\).

Let \(\mathbf 1\) denote the \(n\)-vector of ones. The matrix that subtracts the sample mean from each observation is \(\mathbf H=\mathbf I_n-\mathbf 1\mathbf 1^T/n\). Since \(\mathbf Q\mathbf 1=\sqrt n\,\mathbf e_1\), orthogonality gives \(\mathbf Q\mathbf H\mathbf Q^T=\operatorname{diag}(0,1,\ldots,1)\). It follows that

The right-hand side is a sum of \(n-1\) independent outer products of \(N_p(\mathbf 0,\boldsymbol\Sigma)\) vectors. By the definition of the Wishart distribution, it has law \(W_p(\boldsymbol\Sigma,n-1)\). Dividing by \(n-1\) gives the stated law for \(\mathbf S\). Because \(\bar{\mathbf X}\) is a function of \(\mathbf Y_1\), while \(\mathbf S\) is a function of \(\mathbf Y_2,\ldots,\mathbf Y_n\), their independence follows from the independence of the transformed rows. \(\square\)

The rank condition follows from the same representation. The matrix \(\mathbf S\) is formed from \(n-1\) centered Gaussian vectors in \(\mathbb R^p\), so its rank is at most \(\min(p,n-1)\). When \(n-1\geq p\), these Gaussian vectors span \(\mathbb R^p\) with probability one: successively, a Gaussian vector with a nonsingular covariance has probability zero of lying in any fixed proper subspace. Thus \(\mathbf S\) is positive definite with probability one in that case. When \(n-1<p\), its rank is necessarily less than \(p\).

Worked Examples

Interpretation and Common Pitfalls

The two distributions answer different questions. The mean law quantifies estimation uncertainty: its covariance shrinks by a factor of \(n\). The Wishart law describes the random scatter matrix before division by \(n-1\); its degrees of freedom are \(n-1\) because estimating the mean imposes one linear constraint on the centered observations. Together with independence, these laws are foundational for exact one-sample inference with Gaussian data.

• Using the wrong scaling. With the unbiased definition of \(\mathbf S\), it is \((n-1)\mathbf S\), not \(\mathbf S\), that has distribution \(W_p(\boldsymbol\Sigma,n-1)\).
• Using \(n\) degrees of freedom. Centering at the sample mean leaves \(n-1\) independent Gaussian residual directions, so the Wishart degrees of freedom are \(n-1\).
• Assuming independence without Gaussian sampling. The proof depends on the Gaussian density factoring after an orthogonal transformation. For general distributions, the sample mean and sample covariance need not be independent.
• Assuming the covariance estimate is always invertible. Its rank is at most \(n-1\). If \(n-1<p\), it must be singular, even when the population covariance is positive definite.