Why Impose Covariance Structure?
A covariance matrix for \(p\) variables contains information about \(p\) variances and the relationships among every pair of variables. When \(p\) is large relative to the available information, estimating every entry separately can be unstable. A structural covariance family restricts the possible matrices, for example by requiring certain entries to be zero, several entries to be equal, or the matrix to have a particular factorization. Such restrictions reduce the number of free quantities, but the count must reflect the actual family of matrices—not merely the number of parameters written in one formula.
The central task here is to specify a family precisely and compare its dimension with that of an unrestricted covariance matrix. The restrictions may be linear equalities on entries, zero constraints, or nonlinear relationships induced by a factorization. In every case, positive definiteness remains a requirement when the model describes a nonsingular covariance matrix.
The Unrestricted Benchmark
A covariance matrix \(\Sigma\) is symmetric: \(\sigma_{ij}=\sigma_{ji}\). Thus the diagonal has \(p\) distinct variances, and each of the \(p(p-1)/2\) pairs above the diagonal contributes one distinct covariance. Positive definiteness restricts which values are allowed, but does not lower the dimension: positive-definite matrices form an open subset of the vector space of symmetric matrices.
To verify the count, list the diagonal entries and one entry from each off-diagonal pair. The total is
The positive-definiteness condition is an inequality condition, not an equality that removes a degree of freedom. For example, a sufficiently small perturbation of any positive-definite matrix remains positive definite. Consequently, around each such matrix there is an open neighborhood in the space of symmetric matrices, which has dimension \(p(p+1)/2\). \(\square\)
Worked Example: Counting an Unrestricted and a Diagonal Covariance
Suppose \(p=4\). An unrestricted covariance matrix has \(4\) variances and \(4(3)/2=6\) distinct covariances, for a total of \(4+6=10\) free parameters.
Now require all off-diagonal entries to be zero. The covariance matrix is diagonal, and only its four variances can vary. The family therefore has dimension \(4\), provided each variance is positive. The restriction sets six distinct entries to zero, reducing the count from \(10\) to \(4\).
This count concerns covariance parameters. If a model also estimates means, those mean parameters are separate and are not included here.
Equality and Zero Constraints
A prescribed zero removes a free entry when it fixes an otherwise free covariance to zero. An equality restriction reduces the count when it links entries that could otherwise vary independently. The reduction is the number of independent constraints, not necessarily the number of equations written down. For instance, an equation that follows from other restrictions adds no further reduction.
A useful way to count a family defined by linear equality constraints is to begin with the unrestricted symmetric entries and determine the rank of the constraint system. If there are \(m\) independent equality constraints and the constraints leave a nonempty positive-definite portion of the family, its local dimension is \(p(p+1)/2-m\). This simple count does not automatically apply to nonlinear constraints or to parameterizations with redundancies.
Worked Example: Equal Variances and Compound Symmetry
For \(p=4\), impose two requirements: every variance equals \(a\), and every covariance between distinct variables equals \(c\). The matrix must then be
where \(I_4\) is the \(4\times4\) identity matrix and \(\mathbf{1}\) is the four-vector of ones. The family has two parameters, \(a\) and \(c\), compared with \(10\) in the unrestricted family. It is commonly called compound symmetry.
For positive definiteness, consider vectors parallel and perpendicular to \(\mathbf{1}\). Since \(\mathbf{1}\mathbf{1}^{T}\mathbf{1}=4\mathbf{1}\), the eigenvalue in the \(\mathbf{1}\) direction is \(a+3c\). If \(x\) satisfies \(\mathbf{1}^{T}x=0\), then \(\mathbf{1}\mathbf{1}^{T}x=0\), so the other three eigenvalues are \(a-c\). A symmetric matrix is positive definite exactly when all its eigenvalues are positive. Therefore the necessary and sufficient conditions are
These inequalities also imply \(a>0\): adding \(3(a-c)>0\) and \(a+3c>0\) gives \(4a>0\). The valid parameter region is thus an open region in the \((a,c)\) plane, so the family has dimension \(2\). Counting its defining equalities alone is less direct: the parameterization by \(a,c\) makes the two remaining degrees of freedom explicit.
Factorization Constraints and Redundant Parameters
A factorization can impose structure without setting individual covariance entries to zero or equal. A useful general example is a factor covariance representation
where \(L\) is a \(p\times q\) loading matrix and \(D\) is a diagonal matrix of positive residual variances. This representation requires the shared component \(LL^{T}\) to be positive semidefinite and to have rank at most \(q\). If \(D\) has strictly positive diagonal entries, then \(\Sigma\) is positive definite.
There are \(pq+p\) entries in \(L\) and \(D\), but this is not generally the dimension of the covariance family. For every orthogonal \(q\times q\) matrix \(Q\), \(QQ^{T}=I_q\), and
Thus distinct loading matrices related by an orthogonal rotation represent the same covariance matrix. When \(L\) has full column rank, the orthogonal group has \(q(q-1)/2\) local degrees of freedom: an infinitesimal rotation is represented by a skew-symmetric matrix, which has that many free entries. At regular points where these rotations are the only continuous local nonidentifiability, the generic local dimension of the covariance family is therefore
This is a generic local count under the stated regularity and identifiability conditions. If the loading matrix is rank-deficient, or if the model has additional nonidentifiability, the count may change. A parameter count based on a factorization must therefore distinguish raw coordinates from the dimension of the represented matrices.
Worked Example: Counting a Two-Factor Covariance Family
Take \(p=5\) and \(q=2\). The raw factorization has \(5(2)=10\) loading entries and \(5\) diagonal residual variances, for \(15\) coordinates. A full-rank \(5\times2\) loading matrix can be rotated by any \(2\times2\) orthogonal matrix. The rotation group has \(2(2-1)/2=1\) local degree of freedom. Under generic local identifiability apart from this rotation, the covariance family has
free parameters, compared with \(5(6)/2=15\) for an unrestricted \(5\times5\) covariance matrix.
For a concrete member, let the rows of \(L\) be \((1,0),(0,1),(1,1),(1,-1),(2,0)\), and let \(D=\operatorname{diag}(1,2,1,2,3)\). The row dot products give
Adding \(D\) gives
For any nonzero vector \(x\in\mathbb{R}^{5}\), \(x^{T}\Sigma x=x^{T}LL^{T}x+x^{T}Dx=\lVert L^{T}x\rVert^{2}+x^{T}Dx>0\), because \(D\) has strictly positive diagonal entries and \(x\ne0\). Hence this matrix is positive definite. Its displayed zeros do not mean the entire model is a zero-constrained family: the model restriction is the factorization, and those particular zeros arise from the chosen numerical loading values.
A Reliable Counting Procedure
Before comparing a structured family with an unrestricted covariance, identify what the model regards as a distinct covariance matrix. The following sequence prevents the most common counting errors.
For \(p\) variables, start from \(p(p+1)/2\) distinct symmetric entries.
List required zeros and equalities, or specify the factorization defining membership in the family.
For linear equalities, use their rank; for a factorization, check whether multiple parameter values represent the same matrix.
State positive-definiteness conditions and note whether the count is a generic local dimension or only a raw parameter count.
Why the Count Matters—and a Common Pitfall
Parameter counts describe how much flexibility a covariance model has. A smaller family can be easier to estimate from limited data, while a restriction that is inappropriate for the application can rule out meaningful covariance patterns. The count also clarifies how strong the restrictions are relative to an unrestricted model. It does not, by itself, establish that the structure is scientifically appropriate or that an estimator will perform well.
Common mistake: “The factorization has \(pq+p\) entries, so the covariance family has \(pq+p\) parameters.” This counts coordinates, not distinct covariance matrices. In \(\Sigma=LL^{T}+D\), replacing \(L\) by \(LQ\) for an orthogonal \(Q\) leaves \(\Sigma\) unchanged. The correction is to account for continuous redundancy when it is present and to state the conditions under which the resulting dimension count is valid.
A related error is to subtract one degree of freedom for every equation written as a restriction. Some equations may be dependent, and nonlinear factorizations require more than simple equation counting. A sound count starts with the represented set of covariance matrices, checks independent restrictions or local identifiability, and keeps inequality conditions such as positive definiteness separate from equality constraints.
Check Your Understanding
For each question, distinguish the number of written parameters from the dimension of the represented covariance family when necessary.
- How many free entries does an unrestricted covariance matrix have when \(p=6\)?
- A \(5\times5\) covariance matrix is constrained to be diagonal. How many covariance parameters remain, and why?
- For a \(4\times4\) compound-symmetry matrix with diagonal \(a\) and common off-diagonal entry \(c\), what conditions ensure positive definiteness?
- Why can the number of entries in a loading matrix fail to equal the dimension of the covariance family it represents?
- What additional qualification is needed before using the generic dimension formula for a factorized covariance family?