Ridge-Regularized Hotelling Statistics

Why Regularize a Hotelling Statistic?

A quadratic mean statistic measures a mean shift after accounting for variability in different directions. In the classical one-sample Hotelling statistic, this adjustment uses the inverse sample covariance matrix. In high dimensions, however, that matrix may be singular because the sample covariance has rank at most \(n-1\). Even when it is invertible, a very small eigenvalue can make its inverse highly sensitive to small changes in the data.

Ridge regularization replaces direct inversion with inversion after adding a positive multiple of the identity matrix. This gives a well-defined statistic in every direction, including directions in which the sample covariance has zero variance. It also changes how directions are weighted, so the resulting statistic is not simply the classical Hotelling statistic made computable: its interpretation and null calibration require care.

Definition and Basic Setup

Let \(X_1,\ldots,X_n\) be independent and identically distributed \(p\)-dimensional random vectors with mean \(\mu\) and finite covariance matrix \(\Sigma\). Write \(\bar X=n^{-1}\sum_{i=1}^n X_i\), and define the sample covariance matrix by

For a one-sample test of \(H_0:\mu=\mu_0\), let \(d=\bar X-\mu_0\). The classical statistic is \(n d^\mathsf{T}S^{-1}d\) when \(S\) is invertible. For a specified ridge parameter \(\lambda>0\), the ridge-regularized statistic is

The definition does not by itself specify a rejection threshold or a \(p\)-value. A test also needs a null calibration appropriate to the model and the way \(\lambda\) is set. In particular, the classical Hotelling \(T^2\) calibration does not automatically transfer to \(T_\lambda\).

What the Ridge Does in Each Direction

Because \(S\) is symmetric and positive semidefinite, it has an orthonormal eigenbasis. Write its eigenvalues as \(s_1,\ldots,s_p\), all nonnegative, and express \(d\) in the same basis with coordinates \(d_1,\ldots,d_p\). In these coordinates, adding \(\lambda I_p\) changes each eigenvalue from \(s_j\) to \(s_j+\lambda\), while leaving the eigenvectors unchanged. Thus

A direction with large sample variance receives a smaller inverse-variance weight. A direction with small sample variance would receive an extremely large weight under direct inversion; ridge regularization limits that weight to at most \(1/\lambda\). The same mechanism also means that even a genuine mean shift in a low-variance direction is downweighted relative to the unregularized statistic.

Proof. For any nonzero vector \(v\),

So \(S+\lambda I_p\) is positive definite. In an orthonormal eigenbasis of \(S\), the statistic is \(n\sum_j d_j^2/(s_j+\lambda)\). Its derivative with respect to \(\lambda\) is

If \(d\ne 0\), at least one \(d_j^2\) is positive, so the derivative is strictly negative. \(\square\)

Singularity, Limits, and Interpretation

When \(p\geq n\), the centered observations sum to zero, so their span has dimension at most \(n-1\). Consequently, \(S\) has rank at most \(n-1\) and is singular. Ridge regularization handles this algebraically: every zero eigenvalue becomes \(\lambda\). But a finite value is not the same as a reliable estimate of variation in that direction. Ridge supplies a stabilizing amount to the matrix being inverted; it does not create additional information in the sample.

The eigenvalue expression also shows what happens as \(\lambda\) approaches zero. Terms with \(s_j>0\) approach their unregularized values. If \(s_j=0\) and \(d_j\ne0\), the corresponding term is \(n d_j^2/\lambda\), which diverges. If \(s_j=0\) and \(d_j=0\), that direction contributes zero for every positive \(\lambda\). This distinction matters: the limiting behavior depends on the observed mean difference as well as on the covariance matrix.

For \(s_{\max}\), the largest eigenvalue of \(S\), the statistic also obeys the useful bounds

To see this, each eigenvalue satisfies \(0\leq s_j\leq s_{\max}\), so \(1/(s_{\max}+\lambda)\leq1/(s_j+\lambda)\leq1/\lambda\). Multiply by \(n d_j^2\), sum over \(j\), and use \(\sum_j d_j^2=\lVert d\rVert^2\). In particular, the upper bound stays finite even if \(S\) is singular.

Null Calibration and Practical Meaning

Under independent multivariate normal sampling with \(n>p\), the classical one-sample Hotelling statistic has its familiar \(F\) calibration when the covariance is unregularized. That result relies on the ordinary inverse sample covariance and its distributional properties. Replacing \(S^{-1}\) by \((S+\lambda I_p)^{-1}\) changes the quadratic form, so the classical \(F\) reference distribution should not be used without a separate justification.

For example, in the normal model with known \(\Sigma\), conditional on \(S\), the null distribution of \(T_\lambda\) is a weighted sum of independent squared standard normal variables. The weights are the eigenvalues of \(\Sigma^{1/2}(S+\lambda I_p)^{-1}\Sigma^{1/2}\). This follows by writing \(\sqrt n(\bar X-\mu_0)=\Sigma^{1/2}Z\), where \(Z\sim N_p(0,I_p)\), and diagonalizing the resulting symmetric quadratic form. With unknown \(\Sigma\), this conditional expression is not itself a generally available calibration rule; further assumptions or a justified calibration procedure are needed.

Ridge statistics are useful when direct covariance inversion is unstable or impossible, but they trade variance-direction sensitivity for stabilization. The statistic is also invariant under orthogonal changes of coordinates when the same \(\lambda I_p\) is used: rotations preserve lengths and transform \(S\) and \(d\) together. It is not generally invariant under arbitrary rescalings of variables, because adding the same \(\lambda\) to every covariance eigenvalue depends on measurement units. A ridge value has covariance units, so changing scales without corresponding adjustment changes the statistic.

Common Mistakes

• Treating finiteness as validity. The ridge matrix is invertible for \(\lambda>0\), but this alone does not prove that a chosen null distribution or cutoff is valid.
• Assuming the ridge statistic has the classical Hotelling calibration. The matrix being inverted has changed, so the classical result does not automatically apply.
• Thinking ridge deletes zero-variance directions. A direction with zero sample variance receives weight \(1/\lambda\); it can contribute substantially if the mean difference has a component there.
• Ignoring measurement scale. A common ridge value is applied on the covariance scale. Rescaling variables can change the statistic unless the regularization is transformed appropriately.