Cholesky Parameterization of Gaussian Covariance

Why Parameterize a Gaussian Covariance?

For a nondegenerate Gaussian random vector with a density, the covariance matrix must be symmetric and positive definite. In general, a Gaussian covariance matrix may instead be singular and positive semidefinite; this tutorial concerns the nondegenerate model. Directly optimizing over the entries of a symmetric covariance matrix requires maintaining a positive-definiteness constraint throughout the optimization. The Cholesky parameterization replaces that constraint with a simple triangular structure and positive diagonal entries.

The parameterization is useful both conceptually and computationally. It covers every positive-definite covariance matrix exactly once, makes the log-determinant easy to calculate, and lets quadratic forms be evaluated by solving triangular systems rather than explicitly forming a matrix inverse.

The free entries of \(L\) are its \(p(p+1)/2\) entries on and below the diagonal. The entries above the diagonal are fixed at zero, and each diagonal entry must be strictly positive. These conditions matter: they make \(LL'\) positive definite and ensure that the factor is unique.

Why the Parameterization Covers Exactly the Positive-Definite Matrices

First, if \(L\) has positive diagonal entries, then it is invertible. For nonzero \(v\), \(L'v\) is nonzero, so \(v'LL'v=(L'v)'(L'v)>0\). Thus \(LL'\) is positive definite.

For existence, use induction on the dimension. In dimension one, a positive-definite matrix is \([a]\) with \(a>0\), and its factor is \([\sqrt{a}]\). For the induction step, write a positive-definite matrix in block form:

The matrix \(C-bb'/a\) is positive definite. Indeed, for every nonzero \(y\), use the nonzero vector \((-b'y/a,y')'\) in the quadratic form for \(\Sigma\). Direct multiplication gives

By the induction hypothesis, \(C-bb'/a=RR'\) for a lower-triangular \(R\) with positive diagonal. Then

For uniqueness, the first diagonal entry of any such factor must be \(\sqrt a\), and its first column below that entry must be \(b/\sqrt a\), as is seen by multiplying out \(LL'=\Sigma\). The remaining lower-right block must factor \(C-bb'/a\). Inductive uniqueness of that factor proves uniqueness of \(L\). \(\square\)

Computing the Gaussian Log-Likelihood Through the Factor

Suppose \(x_1,\ldots,x_n\) are independent observations from \(N_p(\mu,\Sigma)\), where \(\Sigma\) is positive definite. The Gaussian log-likelihood, expressed using \(L\) with \(\Sigma=LL'\), has a convenient form. For each observation define the residual \(r_i=x_i-\mu\). Since \(L\) is invertible, the quadratic form in the Gaussian density is \(r_i'\Sigma^{-1}r_i=\|L^{-1}r_i\|^2\), where \(\|u\|^2=u'u\). Also, \(\det(\Sigma)=\det(L)^2=\prod_{j=1}^p L_{jj}^2\). Hence:

This formula is equivalent to the usual expression involving \(\log\det(\Sigma)\) and \(\Sigma^{-1}\), not a different likelihood. To verify the equivalence, \(L^{-1}\) exists and \(\Sigma^{-1}=(LL')^{-1}=L'^{-1}L^{-1}\). Thus \(r_i'\Sigma^{-1}r_i=(L^{-1}r_i)'(L^{-1}r_i)\). The determinant identity gives \(\log\det(\Sigma)=2\sum_j\log L_{jj}\). Substitution into the Gaussian log-likelihood yields the displayed formula.

For computation, \(L^{-1}r_i\) should normally be obtained by solving \(Lu_i=r_i\), rather than by explicitly calculating \(L^{-1}\). This is a triangular system, so it can be solved by forward substitution. The diagonal log terms are also straightforward to evaluate. Together, these features make the factor a useful optimization variable.

Unconstrained Coordinates for Optimization

The Cholesky factor still has positive diagonal constraints. A common way to enforce them automatically is to write each diagonal entry as an exponential. Let \(A\) be lower triangular with unrestricted real entries and set

Every real choice of the free entries of \(A\) produces a lower-triangular \(L\) with positive diagonal, and hence a positive-definite covariance \(LL'\). Conversely, any permitted \(L\) is represented by \(A_{jj}=\log L_{jj}\) and \(A_{jk}=L_{jk}\) for \(j>k\). Thus this is a one-to-one reparameterization from unrestricted real coordinates to the set of Cholesky factors.

Common Mistakes and Practical Interpretation

• Allowing zero or negative diagonal entries in the Cholesky factor. The usual Cholesky factor is defined with strictly positive diagonal entries. A zero diagonal makes \(L\) singular, while using unrestricted diagonal entries loses the stipulated unique factor representation. Exponentiating unconstrained diagonal parameters avoids both problems.
• Assuming every Gaussian covariance must be positive definite. That is true for the nondegenerate Gaussian density used in the likelihood above, but not for every Gaussian random vector. A singular Gaussian can have a positive-semidefinite covariance and does not have this ordinary full-dimensional density.
• Confusing a positive diagonal with a diagonal covariance. The factor \(L\) is triangular, not generally diagonal. Its off-diagonal entries encode covariance; in the first example \(L\) is triangular while \(LL'\) has nonzero off-diagonal entries.
• Computing the likelihood with the wrong determinant term. Since \(\det(\Sigma)=(\prod_j L_{jj})^2\), the log-likelihood contribution is \(-n\sum_j\log L_{jj}\). Omitting the factor of two in \(\log\det(\Sigma)=2\sum_j\log L_{jj}\) gives an incorrect likelihood.

In optimization, this parameterization ensures that every finite parameter value corresponds to an admissible nondegenerate covariance. It does not guarantee that an optimizer finds a global maximum, nor does it remove the need to inspect numerical convergence. It does, however, prevent iterations from proposing an invalid covariance simply because a symmetric matrix of entries has become indefinite.