Selecting Covariance Regularization by Held-Out Likelihood

Why Select Regularization by Held-Out Likelihood?

Covariance regularization controls a trade-off. Strong shrinkage can stabilize noisy estimates but miss real dependence; weak shrinkage or entrywise thresholding can preserve structure but also retain sampling noise. A tuning parameter should therefore be judged by how well its covariance estimate describes observations not used to construct it. Held-out Gaussian log-likelihood makes that comparison directly, provided the candidate covariance matrices are positive definite.

The procedure is simple to state: estimate several covariance matrices using a training sample, score each with an independent validation sample, and choose the candidate with the largest validation log-likelihood. The score is a data-dependent selection rule, not a guarantee that the selected matrix is closest under every loss. Its interpretation also depends on the Gaussian model used for scoring.

A common candidate set is a finite grid of estimates \(\hat\Sigma_\lambda\) indexed by a shrinkage parameter \(\lambda\), a thresholding parameter \(\tau\), or both. Each candidate must be positive definite, because the Gaussian density requires an inverse and a positive determinant. If thresholding produces a matrix that is not positive definite, use a specified positive-definite repair before scoring it; otherwise its Gaussian log-likelihood is not defined.

Deriving the Validation Criterion

Define the validation scatter matrix around the known mean by $$ S_{\mathrm{val}}=\frac{1}{m}\sum_{i=1}^m (X_i-\mu)(X_i-\mu)^\mathsf{T}. $$ For any square matrix \(A\) and vector \(z\), the scalar \(z^\mathsf{T}Az\) equals \(\operatorname{tr}(Azz^\mathsf{T})\). Applying this identity to each validation residual gives $$ \sum_{i=1}^m (X_i-\mu)^\mathsf{T}C^{-1}(X_i-\mu) =m\,\operatorname{tr}(C^{-1}S_{\mathrm{val}}). $$ Consequently, $$ \frac{1}{m}\ell_{\mathrm{val}}(C) =-\frac{1}{2}\left[p\log(2\pi)+\log\det(C) +\operatorname{tr}(C^{-1}S_{\mathrm{val}})\right]. $$ The term \(p\log(2\pi)\) is identical for every candidate. Thus maximizing held-out log-likelihood is equivalent to minimizing $$ Q_{\mathrm{val}}(C)=\log\det(C)+\operatorname{tr}(C^{-1}S_{\mathrm{val}}). $$ This matrix expression summarizes both parts of the score: the determinant term penalizes diffuse covariance estimates, while the trace term assesses how well the candidate's inverse covariance fits the validation scatter.

The equivalence follows directly from the derived score: the omitted factor \(-1/2\), the factor \(m\), and the constant involving \(p\) do not depend on the candidate, so none can change its ranking. This also means the full score can be calculated from \(S_{\mathrm{val}}\) without retaining the validation observations individually, as long as the chosen centering is the same.

What Population Quantity Does the Score Target?

Conditional on the training data, a trained candidate \(C\) is fixed. If the validation observations are independent draws from \(N_p(\mu,\Sigma)\), then \(E[S_{\mathrm{val}}]=\Sigma\). Therefore the conditional expected criterion, apart from a candidate-independent constant, is $$ E[Q_{\mathrm{val}}(C)\mid \text{training data}] =\log\det(C)+\operatorname{tr}(C^{-1}\Sigma). $$ This expectation gives a precise population interpretation of the selection rule.

Proof. For \(X\sim N_p(\mu,\Sigma)\), the expected negative log-likelihood under covariance \(C\) is $$ E_\Sigma[-\log f_C(X)] =\frac{1}{2}\left[p\log(2\pi)+\log\det(C) +E_\Sigma\{(X-\mu)^\mathsf{T}C^{-1}(X-\mu)\}\right]. $$ The quadratic expectation equals \(\operatorname{tr}(C^{-1}\Sigma)\), by the same trace identity used for validation scatter. Subtract the expected negative log-likelihood under covariance \(\Sigma\). The result is one half of the expression in braces in the theorem.

To establish its sign, let \(A=\Sigma^{1/2}C^{-1}\Sigma^{1/2}\), which is positive definite, with eigenvalues \(a_1,\ldots,a_p>0\). Then \(\operatorname{tr}(C^{-1}\Sigma)=\sum_{j=1}^p a_j\), and \(\log\det(C)-\log\det(\Sigma)=-\log\det(A)=-\sum_{j=1}^p\log(a_j)\). The expression in braces is therefore $$ \sum_{j=1}^p \left(a_j-\log(a_j)-1\right)\geq 0, $$ since \(a-\log(a)-1\geq 0\) for \(a>0\), with equality only at \(a=1\). Equality for the sum requires every eigenvalue of \(A\) to equal one, so \(A=I\), which implies \(C=\Sigma\). Thus the expected log score is maximized uniquely at the true covariance. \(\square\)

This theorem identifies the population target of the Gaussian score, not a guarantee about a particular finite validation sample or a restricted candidate set. In practice, the candidate closest to \(\Sigma\) under this likelihood criterion may differ from the one closest under Frobenius loss or operator-norm loss.

Comparing Shrinkage and Thresholding

For shrinkage and thresholding, the same calculation applies once each tuning value produces a positive-definite matrix. For example, one can form candidates by shrinking off-diagonal entries toward zero or by thresholding them and then applying a fixed positive-definite repair. The validation set chooses among the resulting matrices; it does not determine which regularization family is appropriate in every setting.

Conditions, Interpretation, and Common Pitfalls

The derivations assume that validation observations are independent of the data used to fit the candidates and follow the Gaussian model being scored. Independence ensures that, conditional on the training data, the candidate matrices are fixed while the validation observations still have the stated distribution. If the mean is unknown and estimated from the training sample, one can score validation observations using that training-only estimate. The criterion formula still holds with residuals centered at the estimate, but conditional on training, their expected scatter includes a contribution from mean-estimation error. The clean population result above uses the known common mean.

A frequent mistake is to select the parameter with the smallest validation error under one criterion and then claim it must be best under all covariance losses. The correction is to name the target: held-out Gaussian likelihood ranks candidates by \(\log\det(C)+\operatorname{tr}(C^{-1}S_{\mathrm{val}})\), whereas another loss can rank them differently. Another pitfall is to report the maximized validation score as an unbiased estimate of the selected model's future performance. Because the same validation observations were used to choose the winner, its score is subject to selection optimism. Use an independent test sample, or an appropriate resampling design, for a less selection-biased final assessment.