Why Covariance Invertibility Matters
The unknown-covariance Hotelling procedure compares a sample mean vector with a hypothesized mean after scaling differences by the estimated covariance. That scaling uses the inverse sample covariance. If the covariance estimate is singular, the usual statistic is not defined, and its standard exact calibration cannot be applied. The issue is not merely computational: centering the observations limits how many independent directions the sample can contain.
Let \(X_1,\ldots,X_n\) be observations in \(p\) dimensions, with sample mean \(\bar X\). The sample covariance is based on the centered vectors \(X_i-\bar X\). These vectors always sum to zero, so no more than \(n-1\) of them can be linearly independent. Thus, if \(n-1<p\), they cannot span all \(p\) dimensions, regardless of the observed values.
A \(p\times p\) matrix has an inverse exactly when its rank is \(p\). The rank bound above therefore gives a necessary condition for invertibility: \(n-1\geq p\), or \(n>p\). Under a continuous, full-rank model such as a nonsingular multivariate normal distribution, this condition is also sufficient with probability one. The distinction between “necessary” and “sufficient” matters: even with \(n>p\), special dependence or exact linear constraints in the data can make \(S\) singular.
The Rank Under a Nonsingular Gaussian Model
The following result makes the sample-size threshold precise. The assumption \(n\geq2\) is essential because \(S\) is not defined for one observation.
To prove the rank statement, arrange the observations as rows of an \(n\times p\) matrix \(X\). Let \(\mathbf 1\) be the \(n\)-vector of ones and set \(H=I_n-\mathbf 1\mathbf 1^\mathsf{T}/n\), the centering matrix. If \(Z\) is the matrix whose rows are independent \(N_p(0,I_p)\) vectors, then the Gaussian model can be represented as \(X=\mathbf 1\mu^\mathsf{T}+Z\Sigma^{1/2}\). Since \(H\mathbf 1=0\), centering removes the mean and gives $$ (n-1)S =\Sigma^{1/2}Z^\mathsf{T}HZ\Sigma^{1/2}. $$ Because \(\Sigma^{1/2}\) is invertible, multiplication on both sides by it does not change rank. Also, \(Z^\mathsf{T}HZ=(HZ)^\mathsf{T}(HZ)\), and a matrix \(A\) and \(A^\mathsf{T}A\) have the same rank. It remains to find the rank of \(HZ\).
The matrix \(H\) is the orthogonal projection onto the \((n-1)\)-dimensional space perpendicular to \(\mathbf 1\). Choose an \(n\times(n-1)\) matrix \(Q\) with orthonormal columns spanning that space. Then \(H=QQ^\mathsf{T}\), and \(\operatorname{rank}(HZ)=\operatorname{rank}(Q^\mathsf{T}Z)\). Orthogonal transformation of each standard Gaussian column of \(Z\) shows that \(Q^\mathsf{T}Z\) is an \((n-1)\times p\) matrix with independent standard normal entries. Such a matrix has rank \(\min(n-1,p)\) with probability one: a maximal square minor is a polynomial in continuously distributed entries, and at least one such minor is not identically zero, so its zero set has probability zero. This proves the rank formula. Since \(\min(n-1,p)=p\) exactly when \(n>p\), the invertibility conclusion follows. \(\square\)
Connection to the Ordinary Hotelling Procedure
For a hypothesized mean vector \(\mu_0\), the ordinary one-sample Hotelling statistic with unknown covariance is $$ T^2=n(\bar X-\mu_0)^\mathsf{T}S^{-1}(\bar X-\mu_0). $$ Its definition requires \(S^{-1}\). Under independent observations from \(N_p(\mu,\Sigma)\) with positive definite \(\Sigma\), the exact null calibration is $$ \frac{n-p}{p(n-1)}T^2\sim F_{p,n-p} \qquad\text{when }H_0:\mu=\mu_0\text{ and }n>p. $$ Both the inverse and the positive denominator degrees of freedom \(n-p\) require \(n>p\). When \(n=p\), centering leaves at most \(p-1\) directions; when \(n<p\), it leaves at most \(n-1<p\). In either case \(S^{-1}\) does not exist, so the displayed statistic and its stated \(F\) calibration are unavailable.
Worked Example: The Boundary Case \(n=p\)
Take three observations in three dimensions: \(X_1=(1,0,0)^\mathsf{T}\), \(X_2=(0,1,0)^\mathsf{T}\), and \(X_3=(0,0,1)^\mathsf{T}\). Here \(n=p=3\). The sample mean is \(\bar X=(1/3,1/3,1/3)^\mathsf{T}\).
The sum of the centered outer products is \(I_3-\mathbf 1\mathbf 1^\mathsf{T}/3\). Since \(n-1=2\), $$ S=\frac12\left(I_3-\frac{\mathbf 1\mathbf 1^\mathsf{T}}{3}\right) =\begin{pmatrix} 1/3&-1/6&-1/6\\ -1/6&1/3&-1/6\\ -1/6&-1/6&1/3 \end{pmatrix}. $$ Multiplying this matrix by \(\mathbf 1=(1,1,1)^\mathsf{T}\) gives zero, so it is singular. Its other two eigenvalues are \(1/2\), and hence its rank is \(2=p-1\). The sample has three observations, but their centered versions can span at most two directions. The ordinary Hotelling statistic cannot be computed using \(S^{-1}\).
Worked Example: More Observations Than Dimensions
Consider the four two-dimensional observations \((0,0),(1,0),(0,1),(1,1)\). Here \(n=4\), \(p=2\), and \(\bar X=(1/2,1/2)^\mathsf{T}\). The sums of squared deviations in each coordinate are both \(1\), and the cross-product sum is \(0\). Thus $$ S=\frac{1}{3} \begin{pmatrix}1&0\\0&1\end{pmatrix}, \qquad S^{-1}= \begin{pmatrix}3&0\\0&3\end{pmatrix}. $$
For illustration, set \(\mu_0=(0,0)^\mathsf{T}\). Then $$ T^2 =4 \begin{pmatrix}1/2&1/2\end{pmatrix} \begin{pmatrix}3&0\\0&3\end{pmatrix} \begin{pmatrix}1/2\\1/2\end{pmatrix} =4\left(\frac34+\frac34\right)=6. $$ The exact-calibration multiplier is \((n-p)/(p(n-1))=2/(2\cdot3)=1/3\), so the transformed statistic is \(6/3=2\). The arithmetic shows what is possible when \(n>p\): \(S\) has full rank and the ordinary statistic is defined. The example is a calculation, not a claim that these four fixed points were generated by a Gaussian model.
Worked Example: Fewer Observations Than Dimensions
Let \(p=3\), \(n=2\), \(X_1=(2,1,0)^\mathsf{T}\), and \(X_2=(0,1,0)^\mathsf{T}\). The mean is \(\bar X=(1,1,0)^\mathsf{T}\), and the centered observations are \((1,0,0)^\mathsf{T}\) and \((-1,0,0)^\mathsf{T}\). Therefore $$ S= \begin{pmatrix} 2&0&0\\ 0&0&0\\ 0&0&0 \end{pmatrix}. $$ Its rank is \(1=n-1\), and its null space includes the second and third coordinate directions. No inverse exists.
The zero estimated variance in those directions is a statement about these two observed points: they do not vary in those coordinates. It does not establish that the population covariance has zero variance there. Replacing \(S^{-1}\) with a generalized inverse would define a different procedure; it would not make the ordinary Hotelling statistic or its stated exact calibration valid.
What Singularity Does—and Does Not—Say
The rank argument explains why the number of independent centered directions is at most \(n-1\), but it does not mean every singular covariance has the same cause. If \(n\leq p\), singularity is unavoidable from the dimension bound. If \(n>p\), singularity can still occur when variables are exact linear combinations of one another, when the data lie in a lower-dimensional subspace, or through other structural constraints. Under the nonsingular Gaussian model in the theorem, those extra failures occur with probability zero; outside that model, they may be present by design.
A singular \(S\) means there is a nonzero vector \(a\) such that \(Sa=0\). Equivalently, the observed centered data have no component in direction \(a\). The quadratic form in Hotelling’s statistic uses \(S^{-1}\) to measure discrepancies in all \(p\) directions, so it cannot be evaluated as written when any such direction is missing. This is why the issue affects the inferential procedure, not just a numerical algorithm.
There is also a boundary case often hidden by formulas: for \(n=1\), the sample covariance with denominator \(n-1\) is not defined at all. For every \(n\geq2\), it is defined, but its rank is at most \(\min(p,n-1)\). Keeping these two statements separate avoids treating an undefined covariance as a singular one.
Check Your Understanding
Use the rank bound and the theorem’s assumptions to answer these questions.
- For \(n=5\) observations in \(p=5\) dimensions, what is the largest possible rank of \(S\), and can the ordinary Hotelling statistic use \(S^{-1}\)?
- Why is the sample covariance undefined rather than merely singular when \(n=1\)?
- Under the nonsingular Gaussian assumptions, what is the almost-sure rank of \(S\) when \(n=4\) and \(p=2\)?
- Why does \(n>p\) not guarantee invertibility for every possible data set?
- What does a nonzero vector in the null space of \(S\) say about the centered observations?