Separable Covariance and Scale Nonidentifiability

Why Mode Scales Are Not Individually Determined

A separable covariance describes dependence across a tensor’s modes using a covariance matrix for each mode. The full covariance is determined by the combined contribution of those matrices, not by an independently observable scale for each one. Consequently, different collections of mode covariance matrices can describe exactly the same covariance for the tensor.

This is a parameterization ambiguity: it does not mean that the tensor’s covariance changes or that the data contain less information about that covariance. It means that the factors’ scales cannot all be interpreted separately without a convention. We will derive the ambiguity and show how a trace constraint chooses one scale convention.

Definition: Let $X$ be a $K$-way random tensor with mode dimensions $p_1,\ldots,p_K$. A separable covariance has mode covariance factors $\Sigma_k$, where each $\Sigma_k$ is a symmetric positive-definite $p_k\times p_k$ matrix, and, under the fixed vectorization convention, $$ \operatorname{Cov}(\operatorname{vec}X)=\Sigma_K\otimes\cdots\otimes\Sigma_1. $$ The factor $\Sigma_k$ describes covariance along mode $k$. The Kronecker product combines the factors into the covariance of the vectorized tensor.

For tensor entries indexed by $i=(i_1,\ldots,i_K)$ and $j=(j_1,\ldots,j_K)$, this convention also gives $$ \operatorname{Cov}(X_{i_1,\ldots,i_K},X_{j_1,\ldots,j_K}) =\prod_{k=1}^K(\Sigma_k)_{i_kj_k}. $$ Thus every entry of the full covariance is a product of one entry from each mode factor. This product structure is the source of the scale ambiguity.

The Scale-Transfer Invariance

Suppose each mode factor is multiplied by a positive number $a_k$. The full Kronecker product is multiplied by the product of those numbers. If that product is one, the full covariance is unchanged.

Theorem: Let $\Sigma_1,\ldots,\Sigma_K$ be symmetric positive-definite mode covariance factors. For any positive numbers $a_1,\ldots,a_K$ satisfying $\prod_{k=1}^K a_k=1$, define $\widetilde{\Sigma}_k=a_k\Sigma_k$. Then $$ \widetilde{\Sigma}_K\otimes\cdots\otimes\widetilde{\Sigma}_1 =\Sigma_K\otimes\cdots\otimes\Sigma_1. $$ Each $\widetilde{\Sigma}_k$ remains symmetric positive definite. Hence the rescaled factors give exactly the same separable covariance.

Proof. A positive scalar multiple of a symmetric positive-definite matrix is symmetric positive definite, so every transformed factor remains admissible. The Kronecker product is multilinear in its factors, which gives $$ \widetilde{\Sigma}_K\otimes\cdots\otimes\widetilde{\Sigma}_1 =\left(\prod_{k=1}^K a_k\right) (\Sigma_K\otimes\cdots\otimes\Sigma_1) =\Sigma_K\otimes\cdots\otimes\Sigma_1. $$ The final equality uses $\prod_k a_k=1$. $\square$

The same conclusion follows entry by entry: each covariance entry is multiplied by $\prod_k a_k=1$. In particular, for two modes, multiplying $\Sigma_1$ by $c>0$ and $\Sigma_2$ by $1/c$ changes neither the vectorized covariance nor any covariance between tensor entries.

Worked Example: Two Different Factor Pairs, One Covariance

Consider a two-way tensor with two rows and two columns, and factors $$ \Sigma_1=A=\begin{pmatrix}2&1\\1&2\end{pmatrix}, \qquad \Sigma_2=B=\begin{pmatrix}3&0\\0&1\end{pmatrix}. $$ Both matrices are positive definite: their eigenvalues are $1,3$ for $A$, and $1,3$ for $B$. Under the stated vectorization convention the covariance is $B\otimes A$.

Take $c=4$, and set $A'=4A$ and $B'=B/4$. The new factors are still positive definite, and $$ B'\otimes A'=(B/4)\otimes(4A)=B\otimes A. $$ For a direct check, $$ B\otimes A= \begin{pmatrix} 6&3&0&0\\ 3&6&0&0\\ 0&0&2&1\\ 0&0&1&2 \end{pmatrix}. $$ The transformed factors are $A'=\begin{pmatrix}8&4\\4&8\end{pmatrix}$ and $B'=\begin{pmatrix}3/4&0\\0&1/4\end{pmatrix}$. Their Kronecker product has diagonal blocks $(3/4)A'=\begin{pmatrix}6&3\\3&6\end{pmatrix}$ and $(1/4)A'=\begin{pmatrix}2&1\\1&2\end{pmatrix}$, exactly as before. The factor entries changed, but the tensor covariance did not.

How Many Scale Choices Are There?

With $K$ factors, the positive rescalings $a_1,\ldots,a_K$ are subject to the single constraint $\prod_k a_k=1$. One can choose the scales of $K-1$ factors freely; the constraint then determines the remaining scale. For example, in a three-mode model, multiplying the first factor by $2$, the second by $1/3$, and the third by $3/2$ preserves the covariance because $2(1/3)(3/2)=1$.

This redistribution does not change the overall covariance scale. Multiplying every factor by a constant $c$, for instance, multiplies the full covariance by $c^K$. The ambiguity instead concerns how a fixed combined scale is divided among the factors. It is important not to confuse that ambiguity with uncertainty about the covariance matrix itself.

Worked Example: Checking a Three-Mode Scale Transfer

Let three mode factors be diagonal: $$ \Sigma_1=\operatorname{diag}(1,4),\quad \Sigma_2=\operatorname{diag}(2,5),\quad \Sigma_3=\operatorname{diag}(3,7). $$ Rescale them by $a_1=2$, $a_2=1/3$, and $a_3=3/2$. Their product is $2(1/3)(3/2)=1$, so the theorem says the full covariance is unchanged.

For a componentwise check, consider the variance of the tensor entry with mode indices $(2,1,2)$. Its original variance is the product of the corresponding diagonal entries: $$ (\Sigma_1)_{22}(\Sigma_2)_{11}(\Sigma_3)_{22} =4\cdot2\cdot7=56. $$ The transformed factors have diagonal entries $(2,8)$, $(2/3,5/3)$, and $(9/2,21/2)$, respectively. The same tensor entry therefore has variance $$ 8\cdot\frac{2}{3}\cdot\frac{21}{2}=56. $$ In fact, every tensor-entry covariance is unchanged: its three factor entries acquire multipliers $2$, $1/3$, and $3/2$, whose product is one.

A Trace Convention for the Factors

To interpret or estimate individual mode factors, one can impose a normalization that selects one representative from each collection of scale-equivalent factors. A convenient convention is to set the trace of each factor except the last to its dimension. The last factor absorbs the scale needed to preserve the full covariance.

Theorem: For any symmetric positive-definite factors $\Sigma_1,\ldots,\Sigma_K$, there is exactly one rescaling within their scale-equivalence class satisfying $$ \operatorname{tr}(\widetilde{\Sigma}_k)=p_k,\qquad k=1,\ldots,K-1. $$ Here “within their scale-equivalence class” means $\widetilde{\Sigma}_k=a_k\Sigma_k$ for positive $a_k$ with $\prod_k a_k=1$. This convention removes the scale-transfer ambiguity within that class.

Proof. To satisfy the trace constraint for mode $k<K$, the multiplier must be $$ a_k=\frac{p_k}{\operatorname{tr}(\Sigma_k)}. $$ These values are positive because $\Sigma_k$ is positive definite and thus has positive trace. Once $a_1,\ldots,a_{K-1}$ are fixed, the product-one constraint forces $$ a_K=\left(\prod_{k=1}^{K-1}a_k\right)^{-1}. $$ This choice is positive and completes a valid scale transfer, so it proves existence. For uniqueness, any rescaling satisfying the trace constraint must have $a_k\operatorname{tr}(\Sigma_k)=p_k$ for every $k<K$, forcing exactly the displayed values of those multipliers. The product-one constraint then forces the same $a_K$. Thus there is exactly one such rescaling within the class. $\square$

Worked Example: Applying the Trace Convention

Return to $A$ and $B$ from the two-mode example. The convention sets $\operatorname{tr}(\widetilde{\Sigma}_1)=p_1=2$. Since $\operatorname{tr}(A)=4$, the first multiplier must be $a_1=2/4=1/2$. The product-one condition then gives $a_2=2$.

The normalized factors are therefore $$ \widetilde{\Sigma}_1=A/2= \begin{pmatrix}1&1/2\\1/2&1\end{pmatrix}, \qquad \widetilde{\Sigma}_2=2B= \begin{pmatrix}6&0\\0&2\end{pmatrix}. $$ The first factor has trace $2$, as required, and $$ \widetilde{\Sigma}_2\otimes\widetilde{\Sigma}_1 =(2B)\otimes(A/2)=B\otimes A. $$ The covariance is unchanged, while the convention specifies which scale is assigned to the first factor.

Why Not Normalize Every Factor?

It may seem natural to set $\operatorname{tr}(\Sigma_k)=p_k$ for all $K$ factors. But scale transfers cannot generally achieve all these constraints at once. Indeed, under any scale transfer, $$ \prod_{k=1}^K\operatorname{tr}(\widetilde{\Sigma}_k) =\prod_{k=1}^K\left(a_k\operatorname{tr}(\Sigma_k)\right) =\left(\prod_{k=1}^K a_k\right)\prod_{k=1}^K\operatorname{tr}(\Sigma_k) =\prod_{k=1}^K\operatorname{tr}(\Sigma_k). $$ The product of the traces is invariant within the scale-equivalence class. Setting every trace to its dimension is possible only if the original invariant already equals $\prod_k p_k$. Normalizing $K-1$ factors avoids this unnecessary compatibility requirement.

Common Mistakes and Practical Meaning

Rescaling only one factor and claiming the covariance is unchanged. Scaling one factor by $c$ multiplies the full covariance by $c$. To preserve it, other factor multipliers must have product $1/c$.
Interpreting each factor’s scale as separately determined. The same separable covariance has multiple factor representations. A scale convention is needed before assigning a unique scale to each factor.
Thinking the ambiguity changes the tensor’s covariance. A scale transfer with product one leaves the full covariance exactly fixed; it changes the representation, not the model covariance.
Assuming a trace constraint proves every kind of identifiability. The normalization selects a unique representative among scale-equivalent tuples. By itself, it does not establish that no other kind of factor ambiguity is possible.

A scale convention is useful whenever mode factors are reported, compared, or used as parameters in a model. Without one, two analyses can produce different-looking factor scales while agreeing on the same separable covariance. The convention must therefore be stated when presenting the factors. It is a parameterization choice, not an additional claim about the data or a change to the covariance model.

Check Your Understanding

Use the scale-transfer condition and the trace convention to check your reasoning.

For a two-mode covariance, what multiplier must accompany multiplying the first factor by $c>0$ if the full covariance is to remain fixed?
In a three-mode model, do multipliers $3$, $1/2$, and $2/3$ preserve the covariance? Verify the product.
Why does multiplying every factor by $c$ generally change the full covariance, even though reciprocal rescaling between two factors does not?
For $K$ factors, which $K-1$ trace constraints are imposed by the convention described here, and how is the remaining multiplier determined?
What invariant explains why it may be impossible to set every factor’s trace equal to its mode dimension?