Identifiability of Overlapping Covariance Components

Why Component Overlap Raises an Identification Question

Structured covariance models build a covariance matrix from known component matrices and unknown coefficients. A component may contribute to entries that another component also affects. Overlap by itself does not make parameters nonidentifiable: what matters is whether the component matrices are linearly independent. If one component can be expressed as a linear combination of others, different coefficient values may produce exactly the same covariance matrix, and therefore cannot be distinguished from the distribution of the data.

The distinction matters even when a likelihood can be computed exactly and optimized successfully. Computation can find a parameter value that fits the observed covariance, but it cannot resolve a genuine ambiguity in the parameterization. The relevant question is whether distinct admissible parameter vectors can produce the same model distribution.

Definition: Let $A_1,\ldots,A_q$ be known symmetric $p\times p$ matrices, and define the covariance map $$ \Sigma(\theta)=\sum_{j=1}^{q}\theta_j A_j,\qquad \theta\in\Theta. $$ Here $\Theta\subseteq\mathbb{R}^q$ is the admissible parameter space, restricted as needed so that $\Sigma(\theta)$ is positive definite. The parameterization is identifiable on $\Theta$ if $\Sigma(\theta)=\Sigma(\theta')$ for $\theta,\theta'\in\Theta$ implies $\theta=\theta'$. For a Gaussian model with fixed mean, this is equivalent to identifiability from the distribution, because a multivariate normal distribution is determined by its mean and covariance.

A Rank Criterion

The symmetric matrices can be represented as vectors by listing their distinct entries. Write $\operatorname{vech}(A)$ for the vector containing the diagonal and one copy of each off-diagonal entry of a symmetric matrix $A$. Form the component matrix $$ M=\begin{bmatrix} \operatorname{vech}(A_1)&\cdots&\operatorname{vech}(A_q) \end{bmatrix}. $$ Then $\operatorname{vech}(\Sigma(\theta))=M\theta$, so equality of two covariances is equivalent to $M(\theta-\theta')=0$. The rank of $M$ is the dimension of the span of the component matrices.

Theorem (rank criterion): The covariance map is identifiable on all of $\mathbb{R}^q$ if and only if $\operatorname{rank}(M)=q$. Full column rank is sufficient for identifiability on any subset $\Theta$. It is also necessary for identifiability on any parameter set containing a nonempty open subset of $\mathbb{R}^q$. On a restricted set, the exact criterion is $$ \ker(M)\cap(\Theta-\Theta)=\{0\}, $$ where $\Theta-\Theta=\{\theta-\theta':\theta,\theta'\in\Theta\}$.

Proof. For any $\theta,\theta'$, the two covariance matrices are equal precisely when $$ M\theta=M\theta' \quad\Longleftrightarrow\quad M(\theta-\theta')=0 \quad\Longleftrightarrow\quad \theta-\theta'\in\ker(M). $$ Thus the map is injective on $\Theta$ exactly when no nonzero difference of two points in $\Theta$ belongs to $\ker(M)$. This proves the stated criterion. If $\operatorname{rank}(M)=q$, then $\ker(M)=\{0\}$, so the criterion holds for every $\Theta$. Conversely, if the rank is less than $q$, choose a nonzero $d\in\ker(M)$. If $\Theta$ contains an open ball centered at $\theta_0$, then for sufficiently small $\varepsilon>0$, both $\theta_0$ and $\theta_0+\varepsilon d$ lie in that ball. They are distinct, but $$ M(\theta_0+\varepsilon d)=M\theta_0+\varepsilon Md=M\theta_0. $$ So the map is not injective on $\Theta$. $\square$

The open-set condition is important. In variance-component models, parameters are often constrained to be nonnegative, and positive definiteness may impose further restrictions. If there is an admissible point with every component strictly positive and a positive-definite covariance, these strict conditions hold in a neighborhood of that point. A rank deficiency then produces nearby admissible parameter values with the same covariance. But on a lower-dimensional parameter set, a rank-deficient matrix need not prevent injectivity: the set may exclude every nonzero null direction.

Worked Example: Overlapping Entries With Full Rank

Consider the three symmetric matrices $$ A_1=I_2,\qquad A_2=\begin{bmatrix}1&0\\0&0\end{bmatrix},\qquad A_3=\begin{bmatrix}0&1\\1&0\end{bmatrix}. $$ The first two overlap in their $(1,1)$ entry. With $\theta_j\geq 0$, the covariance is $$ \Sigma(\theta)= \begin{bmatrix} \theta_1+\theta_2&\theta_3\\ \theta_3&\theta_1 \end{bmatrix}. $$

In the order $(11,12,22)$, the component matrix is $$ M=\begin{bmatrix} 1&1&0\\ 0&0&1\\ 1&0&0 \end{bmatrix}. $$ Its determinant is $1$, so its rank is $3$, equal to the number of components. The parameters can also be recovered directly: $$ \theta_1=\Sigma_{22},\qquad \theta_2=\Sigma_{11}-\Sigma_{22},\qquad \theta_3=\Sigma_{12}. $$ For example, $\theta=(2,1,0.2)$ gives $\Sigma=\begin{bmatrix}3&0.2\\0.2&2\end{bmatrix}$. Its determinant is $3(2)-(0.2)^2=5.96>0$, and its first diagonal entry is positive, so it is positive definite. The example shows that overlapping entries do not imply nonidentifiability; linear independence is the decisive property.

What Rank Deficiency Means

If $\operatorname{rank}(M)<q$, then some nonzero vector $d$ satisfies $Md=0$, or equivalently $\sum_j d_jA_j=0$. Whenever both $\theta$ and $\theta+d$ are admissible, they produce the same covariance. The identifiable information is then about combinations of parameters that affect $\Sigma$, not necessarily about every coefficient separately.

Worked Example: A Null Direction Creates Nonidentifiability

Let $$ A_1=I_2,\qquad A_2=\begin{bmatrix}1&0\\0&0\end{bmatrix},\qquad A_3=\begin{bmatrix}0&0\\0&1\end{bmatrix}. $$ These matrices satisfy $A_1=A_2+A_3$. Consequently, $d=(1,-1,-1)^\mathsf{T}$ is a null vector of the component map. The covariance is $$ \Sigma(\theta)= \begin{bmatrix} \theta_1+\theta_2&0\\ 0&\theta_1+\theta_3 \end{bmatrix}. $$

Take $\theta=(2,1,1)$ and $\theta'=(1,2,2)$. Both parameter vectors are strictly positive, and direct substitution gives $$ \Sigma(2,1,1)= \begin{bmatrix}3&0\\0&3\end{bmatrix} =\Sigma(1,2,2). $$ Thus both give the same positive-definite covariance, despite being distinct. In this model the observable combinations are $\theta_1+\theta_2$ and $\theta_1+\theta_3$; the individual coefficients cannot be recovered uniquely from the covariance.

This ambiguity carries directly into likelihood inference. In a centered Gaussian model, the likelihood depends on the parameter vector through $\Sigma(\theta)$. If two admissible vectors give the same covariance, they give the same likelihood for every possible data set. Optimization may return one point on the set of equivalent parameter values, but that choice does not establish that the individual parameters are identified.

Constraints Can Change the Answer

The full-rank test is a simple, powerful criterion, but it is important not to overstate its necessity. A rank-deficient component matrix guarantees a nonzero null direction in the ambient parameter space. It does not guarantee that two points on a particular constrained parameter set differ by that direction. Identifiability must be assessed on the parameter space actually used by the model.

Worked Example: A Rank-Deficient Map on a Restricted Parameter Set

Let $A_1=A_2=I_2$, so the component matrix has two identical columns and rank $1$. The covariance map on unrestricted parameter vectors is $\Sigma(\theta)=(\theta_1+\theta_2)I_2$, and is not injective there. Now restrict the model to $$ \Theta=\{(t,0):t>0\}. $$ For $\theta=(t,0)$, the covariance is $tI_2$. If $tI_2=sI_2$, comparison of the diagonal entries gives $t=s$. Hence the map is injective on this restricted set, even though the component matrix does not have full column rank. The restriction has removed the ambiguity by fixing the second coefficient at zero.

Common Mistakes and Practical Checks

Assuming overlap alone causes nonidentifiability. The first worked example has overlapping entries and full rank. Check linear dependence, not merely whether two matrices affect the same entry.
Treating rank deficiency as automatic failure under every constraint. A null direction shows failure only if it connects two admissible parameter values. Use the exact criterion involving $\Theta-\Theta$, or check for an admissible interior point.
Confusing covariance rank with component-matrix rank. The relevant rank is that of $M$, whose columns are the vectorized component matrices. It is not the rank of a particular covariance $\Sigma(\theta)$.
Assuming an optimizer resolves an ambiguity. If two parameter values produce the same covariance, their Gaussian likelihoods agree for all data. Numerical optimization cannot identify a distinction absent from the model distribution.

For an actual model, form $M$ by listing each distinct symmetric entry consistently across all component matrices, then determine its rank and inspect any null vectors. Exact rank is a mathematical property; in numerical work, nearly dependent columns can also make estimation unstable even when the exact rank is full. That is a conditioning concern, distinct from exact nonidentifiability.

Key takeaway: Linear independence of the covariance-component matrices gives a transparent rank-based guarantee of identifiability. When rank is deficient, null vectors reveal covariance-preserving parameter changes; whether those changes imply nonidentifiability depends on the admissible parameter space.

Check Your Understanding

Use the covariance map and its admissible parameter space to answer each question.

For $q$ symmetric component matrices, how do you construct a matrix whose column rank tests linear independence of the components?
If the component matrix has full column rank, can two distinct parameter vectors produce the same covariance? Explain.
What does a nonzero vector in the null space of the component matrix imply about the covariance map on the unrestricted parameter space?
Why does rank deficiency not, by itself, prove nonidentifiability on every constrained parameter space?
In a Gaussian covariance model, why can an optimizer not distinguish two admissible parameter vectors that yield exactly the same covariance?