Covariance Likelihoods And Parameter Counting

Why Scatter and Parameter Counts Matter

A covariance model is a probability model for how several measurements vary together. Its likelihood depends on the observations through a compact summary: their scatter around the mean. Writing the likelihood in that form makes the unrestricted covariance estimate visible and provides a consistent way to count how many parameters a structured covariance model contains.

Here the mean vector is treated as fixed and specified. This keeps the focus on the covariance likelihood itself. In a Gaussian model, covariance matrices are symmetric and positive definite, so their diagonal entries and distinct off-diagonal entries determine all their free coordinates.

Definition: Let $X_1,\ldots,X_n$ be independent $p$-dimensional observations with $X_i\sim N_p(m,\Sigma)$, where the mean vector $m$ is fixed and $\Sigma$ is an unknown positive-definite covariance matrix. The scatter matrix about $m$ is $$ S_m=\sum_{i=1}^n (X_i-m)(X_i-m)^\mathsf{T}. $$ For observed vectors $x_1,\ldots,x_n$, $S_m$ denotes the corresponding observed matrix. It is symmetric and positive semidefinite.

The Gaussian Likelihood in Scatter Form

The density of one observation at $x_i$ is $$ (2\pi)^{-p/2}|\Sigma|^{-1/2} \exp\!\left\{-\frac12(x_i-m)^\mathsf{T}\Sigma^{-1}(x_i-m)\right\}. $$ Independence makes the joint likelihood the product of these densities. Taking its logarithm turns the product into a sum. The quadratic terms can be combined using the identity $a^\mathsf{T}Ba=\operatorname{tr}(Baa^\mathsf{T})$, which holds for a vector $a$ and a square matrix $B$ of matching dimension.

Result: For independent $N_p(m,\Sigma)$ observations with fixed $m$, the log-likelihood as a function of $\Sigma$ is $$ \ell(\Sigma) =-\frac{np}{2}\log(2\pi)-\frac n2\log|\Sigma| -\frac12\operatorname{tr}(\Sigma^{-1}S_m). $$

To see the scatter term explicitly, apply the trace identity to each observation and add: $$ \sum_{i=1}^n (x_i-m)^\mathsf{T}\Sigma^{-1}(x_i-m) =\sum_{i=1}^n\operatorname{tr}\!\left(\Sigma^{-1}(x_i-m)(x_i-m)^\mathsf{T}\right) =\operatorname{tr}(\Sigma^{-1}S_m). $$ The last equality uses linearity of the trace. Thus the observations affect the covariance-dependent part of the likelihood only through $S_m$.

Worked Example: A One-Dimensional Scatter Likelihood

Take $p=1$, $n=2$, fixed mean $m=0$, and observations $x_1=-1$, $x_2=1$. Their scatter is $$ S_m=(-1-0)^2+(1-0)^2=2. $$ Writing the variance as $\sigma^2>0$, the log-likelihood is $$ \ell(\sigma^2)=-\log(2\pi)-\log(\sigma^2)-\frac{1}{\sigma^2}. $$ The scatter divided by the sample size is $S_m/n=2/2=1$, suggesting the variance estimate $1$. At this value, $\ell(1)=-\log(2\pi)-1$. For the candidate variance $4$, $$ \ell(4)-\ell(1)=-\log(4)-\frac14+1=\frac34-\log(4)\approx -0.6363. $$ The candidate has lower log-likelihood. These observations have a genuine positive scatter; in particular, two observations are used rather than a single observation whose scatter about itself would be zero.

The Unrestricted Covariance Maximum

The unrestricted covariance model allows any positive-definite $p\times p$ matrix. If $S_m$ is positive definite, define $T=S_m/n$. Ignoring terms that do not depend on $\Sigma$, maximizing the log-likelihood is equivalent to minimizing $$ \log|\Sigma|+\operatorname{tr}(\Sigma^{-1}T). $$ The positive-definiteness condition on $S_m$ matters: it ensures the likelihood has a finite, unique maximum in the positive-definite parameter space.

Theorem: If the observed scatter matrix $S_m$ is positive definite, the unrestricted maximum-likelihood estimate of $\Sigma$ is $$ \widehat{\Sigma}=\frac{S_m}{n}. $$ It is the unique maximizer over all positive-definite covariance matrices.

Proof. Let $T=S_m/n$, which is positive definite, and write $$ B=T^{-1/2}\Sigma T^{-1/2}. $$ As $\Sigma$ ranges over positive-definite matrices, so does $B$, and $\Sigma=T^{1/2}BT^{1/2}$. Determinant multiplication gives $\log|\Sigma|=\log|T|+\log|B|$. Also, using invariance of trace under cyclic rearrangement of matrix products, $$ \operatorname{tr}(\Sigma^{-1}T)=\operatorname{tr}(B^{-1}). $$ Therefore the quantity to minimize is $\log|T|+\log|B|+\operatorname{tr}(B^{-1})$. The first term is constant in $B$. Let $\lambda_1,\ldots,\lambda_p$ be the positive eigenvalues of $B$. Since $B$ is positive definite, the spectral theorem yields $$ \log|B|+\operatorname{tr}(B^{-1}) =\sum_{j=1}^p\left(\log\lambda_j+\frac1{\lambda_j}\right). $$ For every $\lambda>0$, set $g(\lambda)=\log\lambda+1/\lambda$. Its derivative is $$ g'(\lambda)=\frac{\lambda-1}{\lambda^2}. $$ Thus $g$ decreases on $(0,1)$, increases on $(1,\infty)$, and has its unique minimum $g(1)=1$. The sum is consequently minimized only when every $\lambda_j=1$. A symmetric positive-definite matrix with all eigenvalues equal to one is the identity, so the unique minimizing matrix is $B=I$. This means $\Sigma=T=S_m/n$, proving the result. $\square$

If the scatter is singular, the theorem’s condition fails. For example, if there is a nonzero direction $v$ with $S_m v=0$, the data have no scatter in that direction. A covariance can be made arbitrarily small along $v$ without increasing the quadratic scatter penalty, while the determinant term drives the likelihood upward. Thus there is no finite unrestricted maximum in the positive-definite space. For scatter about a fixed mean, the rank of $S_m$ is at most $n$, so $n\geq p$ is necessary for positive definiteness; it does not by itself guarantee it, since the observed vectors must span all $p$ directions.

Worked Example: A Two-Variable Unrestricted Estimate

Suppose $p=2$, $n=4$, the mean is fixed at zero, and the observed scatter is $$ S_m=\begin{pmatrix}8&2\\2&4\end{pmatrix}. $$ Its leading diagonal entry is $8>0$, and its determinant is $8(4)-2(2)=28>0$; hence this symmetric matrix is positive definite. The unrestricted estimate is $$ \widehat{\Sigma}=\frac14S_m =\begin{pmatrix}2&0.5\\0.5&1\end{pmatrix}. $$ Its determinant is $2(1)-(0.5)^2=1.75>0$, and its first diagonal entry is positive, confirming that the estimate is positive definite. The off-diagonal estimate $0.5$ is retained: the unrestricted model does not impose zero covariance.

Counting Free Covariance Parameters

A parameter count is a count of freely variable coordinates within a model, not a count of the entries one happens to print in a matrix. A symmetric $p\times p$ covariance matrix has $p$ diagonal entries and $p(p-1)/2$ distinct off-diagonal entries. It therefore has $$ p+\frac{p(p-1)}2=\frac{p(p+1)}2 $$ free covariance parameters. Positive definiteness restricts which values these coordinates may take, but it does not reduce the local dimension: positive-definite matrices form an open subset of the space of symmetric matrices.

Parameter-count result: An unrestricted $p\times p$ covariance matrix has $p(p+1)/2$ free parameters. For common covariance structures, the counts are:

A specified positive-definite covariance matrix has $0$ free parameters.
A diagonal covariance matrix has $p$ free parameters, one variance per variable.
A spherical covariance matrix $\sigma^2 I_p$ has $1$ free parameter.
A block-diagonal covariance matrix with fixed block sizes $p_1,\ldots,p_g$, where $\sum_{k=1}^g p_k=p$, has $\sum_{k=1}^g p_k(p_k+1)/2$ free parameters.

These counts follow by identifying which entries may vary. A specified matrix has no unknown coordinates. A diagonal matrix has only its $p$ diagonal entries free. In a spherical matrix all diagonal entries equal the same scalar and all off-diagonal entries are zero, leaving one scalar. In a block-diagonal matrix, each diagonal block is a symmetric covariance matrix; block entries outside those blocks are fixed at zero. Counting the symmetric entries separately within each block gives the stated sum.

When a structured model is compared with the unrestricted model, the difference in dimensions counts the number of covariance restrictions in this basic parameter-counting sense. For a specified matrix this difference is $p(p+1)/2$. For a diagonal model it is $p(p-1)/2$. For a spherical model it is $p(p+1)/2-1$. For block sizes $p_1,\ldots,p_g$, it is $$ \frac{p(p+1)}2-\sum_{k=1}^g\frac{p_k(p_k+1)}2. $$ These arithmetic counts describe the dimensions of the models; they are not, by themselves, a test statistic or a guarantee about a test’s calibration.

Worked Example: Counting Parameters for Three Variables

For $p=3$, an unrestricted covariance matrix has $3(4)/2=6$ parameters: three variances and three distinct covariances. Under a diagonal restriction, only the three variances are free, so there are $3$ parameters and $6-3=3$ entries are constrained to zero. Under a spherical restriction, the matrix is $\sigma^2 I_3$, so only one parameter is free and the dimension difference is $6-1=5$. The counts distinguish the structures: diagonal covariance allows three different variances, while spherical covariance requires all three to be equal.

Worked Example: A Block-Diagonal Covariance Structure

Let $p=5$, divided into two fixed blocks of sizes $p_1=2$ and $p_2=3$. The first block contributes $2(3)/2=3$ parameters; the second contributes $3(4)/2=6$. Thus the block-diagonal model has $3+6=9$ free parameters. The unrestricted model has $5(6)/2=15$, so the block structure fixes $15-9=6$ cross-block covariance entries to zero. There are $2\times3=6$ such entries, which verifies the count directly.

Common Pitfalls and Practical Interpretation

A frequent error is to use $S_m/(n-1)$ as the likelihood maximizer. The divisor depends on the estimation target: in this fixed-mean Gaussian likelihood, the maximizer is $S_m/n$. The familiar unbiased covariance estimator uses a different divisor in the setting where the mean is estimated; unbiasedness and maximum likelihood are not the same criterion. Do not substitute one divisor for the other without identifying the model and the estimator being requested.

Another error is to count both $\Sigma_{ij}$ and $\Sigma_{ji}$ as independent parameters. Covariance matrices are symmetric, so those entries are equal and count only once. Likewise, a block-diagonal matrix does not have $\sum_k p_k^2$ free entries: each block is itself symmetric, and entries outside the blocks are fixed, not free.

The scatter representation is useful because it separates the observed information from the covariance model: the data enter through $S_m$, while the model determines which covariance matrices are allowed. Parameter counts then make the structural restrictions explicit. These are the ingredients needed to formulate covariance likelihood comparisons, while the comparison’s statistic and its sampling calibration require additional steps.

Check Your Understanding

Use the definitions and counts above to answer each question.

For $p=4$, how many free parameters are in an unrestricted covariance matrix and in a diagonal covariance matrix?
Why is positive definiteness of $S_m$ relevant to the existence of the unrestricted covariance maximum?
A block-diagonal covariance matrix has block sizes $1,2,3$. How many free parameters does it have, and how many fewer than the unrestricted model?
For a fixed-mean Gaussian likelihood with scatter $S_m$, what divisor appears in the unrestricted maximum-likelihood estimate?
Explain why a spherical covariance matrix has one free parameter rather than $p$.