Combining Block, Toeplitz, and Kronecker Restrictions

Why Combine Covariance Restrictions?

A covariance model often needs to express more than one feature of the data. Measurements may be ordered in time, grouped into blocks, and arranged in a multidimensional array. A block restriction can encode zero covariance between groups; a Toeplitz restriction can encode covariance depending only on lag; a Kronecker restriction can encode separability across array dimensions. Imposing these features simultaneously can yield a more parsimonious model, but the restrictions may overlap, conflict, or force a covariance matrix to the boundary of positive definiteness.

The central task is to describe the combined model as an intersection, then count the parameters of that intersection—not the sum of the parameter counts for its pieces. For linear restrictions, this count is governed by the rank of all equality constraints taken together. For Kronecker restrictions, the product parameterization must also account for its scale ambiguity.

Definition: A block restriction specifies that selected covariance blocks are zero, equal, or otherwise constrained. A symmetric matrix \(\Sigma\) of order \(p\) is Toeplitz if \(\Sigma_{ij}\) depends only on the lag \(|i-j|\). A Kronecker covariance model represents a covariance matrix as a Kronecker product of smaller factor matrices, for example \(\Sigma=A\otimes B\). A combined covariance model is the set of matrices satisfying all specified restrictions simultaneously, together with symmetry and positive definiteness.

Linear Restrictions and Their Intersection

A symmetric \(p\times p\) matrix has \(p(p+1)/2\) distinct entries. Collect these entries into a vector \(s(\Sigma)\), using any fixed ordering of the diagonal and upper-triangular entries. Block-zero restrictions, equality of selected entries, and Toeplitz restrictions are linear equations in this vector. Put all such equations into one system \(Cs(\Sigma)=d\). A homogeneous restriction has \(d=0\); fixed covariance values can make \(d\) nonzero.

The number of constraints is not the number of rows written in \(C\). Some equations may be repetitions or linear combinations of others. The effective number is \(\operatorname{rank}(C)\). If the system is consistent, its solution set has dimension \(p(p+1)/2-\operatorname{rank}(C)\). This counts the affine equality-constrained space; positive definiteness must still be checked.

Theorem: Let \(C\) be a real matrix with \(m=p(p+1)/2\) columns. If \(Cs=d\) is consistent, its solution set in \(\mathbb{R}^m\) has dimension \(m-\operatorname{rank}(C)\). If this solution set contains a positive-definite symmetric matrix, the positive-definite part has the same dimension. If it contains none, the covariance model is empty, despite the consistency of the linear equations.

Proof. By consistency, choose one solution \(s_0\). Every solution is \(s_0+v\), where \(Cv=0\); conversely, every such vector is a solution. Thus the solution set is a translate of the null space of \(C\). The rank-nullity theorem gives that null space dimension as \(m-\operatorname{rank}(C)\). Now suppose the affine set contains a positive-definite matrix \(\Sigma_0\). Positive definiteness is an open condition on symmetric matrices: if the smallest eigenvalue of \(\Sigma_0\) is \(\lambda_{\min}>0\), any symmetric perturbation \(H\) with operator norm less than \(\lambda_{\min}\) leaves \(\Sigma_0+H\) positive definite. Hence a neighborhood of \(\Sigma_0\) within the affine set remains positive definite, so restricting to positive-definite matrices does not reduce its dimension. If there is no positive-definite solution, the covariance model has no members. \(\square\)

Worked Example: Toeplitz and Block-Diagonal Restrictions

Let \(\Sigma\) be a \(4\times4\) covariance matrix. Require it to be Toeplitz and block diagonal for the partition \(\{1,2\}\mid\{3,4\}\). Find the resulting form and its number of free parameters.

A symmetric Toeplitz matrix has the form \(\Sigma=\begin{pmatrix}t_0&t_1&t_2&t_3\\t_1&t_0&t_1&t_2\\t_2&t_1&t_0&t_1\\t_3&t_2&t_1&t_0\end{pmatrix}\). Block diagonality requires the cross-block entries \(\Sigma_{13},\Sigma_{14},\Sigma_{23},\Sigma_{24}\) to vanish. These are respectively \(t_2,t_3,t_1,t_2\), so the constraints imply \(t_1=t_2=t_3=0\). Therefore \(\Sigma=t_0I_4\). It is positive definite exactly when \(t_0>0\), leaving one free parameter.

The two structures do not contribute their separate parameter counts additively: the intersection is much smaller than either unrestricted structure. In this case the equality constraints are compatible and leave a nonempty positive-definite model.

Compatibility Has Two Parts

For linear restrictions, algebraic compatibility means that the equations have at least one solution. Equivalently, the rank of the coefficient matrix equals the rank of the augmented matrix \([C\ d]\). But a solution need not be a valid covariance matrix. A covariance model also requires symmetry (built into the distinct-entry representation) and positive definiteness. Thus a reliable compatibility check has two stages: solve the equalities, then determine whether any solution is positive definite.

A practical way to check the second stage is to parameterize the equality-constrained solutions and test the eigenvalues of candidate matrices. In a proof, a constructed positive-definite member establishes nonemptiness. If restrictions force a zero variance or a singular matrix, they do not define a positive-definite covariance model, even if all equalities are mutually consistent.

Worked Example: A Consistent System That Is Not a Covariance Model

Take the block and Toeplitz restrictions from the preceding example, but additionally require the lag-two covariance to equal \(t_2=0.3\). Determine whether the combined restrictions are compatible.

Block diagonality requires \(\Sigma_{13}=0\). Toeplitz structure identifies \(\Sigma_{13}=t_2\), so the first two restrictions imply \(t_2=0\). The added requirement says \(t_2=0.3\). Since \(0\ne0.3\), no matrix satisfies all the equalities. The system is algebraically inconsistent, so there is no covariance model to test for positive definiteness.

A different issue arises if the extra requirement were \(t_0=0\): the equalities would be consistent, but the resulting matrix would be \(0\), which is not positive definite. This distinction prevents “the equations have a solution” from being mistaken for “the covariance model is valid.”

Counting Kronecker Parameters Under Additional Structure

For a Kronecker model, counting the entries of the factors overcounts the degrees of freedom because a scale can be moved from one factor to another without changing the product. For nonzero scalar \(c\), \((cA)\otimes(B/c)=A\otimes B\). With positive-definite factors, one can remove this ambiguity by fixing a normalization, such as \(\operatorname{tr}(B)=1\), or by fixing one specified diagonal entry of \(B\) to one. A normalization is valid only when it can be imposed on every factor allowed by the model.

Additional Toeplitz or block restrictions should be applied to the factors or to the product exactly where the scientific model places them. For example, requiring both factors to be Toeplitz is not the same statement as requiring the full Kronecker product to be Toeplitz. Once a valid normalization and factor restrictions are specified, count the remaining factor parameters, while checking that the parameterization has no further redundancies in the region being considered.

Result: If \(A\) and \(B\) range over identifiable parameterizations of positive-definite factors with dimensions \(r\) and \(q\), respectively, and the Kronecker product has only the usual one-dimensional scale ambiguity, the Kronecker covariance family has dimension \(r+q-1\). This count applies when the factor restrictions preserve that scale transformation and no additional parameter redundancies occur.

Derivation. Before normalization, the factor parameter pair has \(r+q\) coordinates. For any admissible \(c>0\), the transformation \((A,B)\mapsto(cA,B/c)\) leaves \(A\otimes B\) unchanged, giving a one-dimensional family of parameter pairs for the same covariance. A normalization such as fixing \(\operatorname{tr}(B)\) removes this freedom: for any positive-definite \(B\), its trace is positive, so rescaling can set the trace to a chosen positive constant. The normalized parameterization therefore has \(r+q-1\) dimensions, provided there are no additional redundancies. \(\square\)

Worked Example: Toeplitz Factors and a Block Restriction

Consider a \(2\times2\) array with covariance \(\Sigma=A\otimes B\), where \(A\) and \(B\) are both \(2\times2\) Toeplitz positive-definite matrices. Also require the \(2\times2\) block matrix \(\Sigma\) to be block diagonal across the two levels indexing \(A\). Find the combined form and count its parameters.

Write \(A=\begin{pmatrix}a&c\\c&a\end{pmatrix}\) and \(B=\begin{pmatrix}b&d\\d&b\end{pmatrix}\). In block form, \(A\otimes B=\begin{pmatrix}aB&cB\\cB&aB\end{pmatrix}\). Block diagonality requires \(cB=0\). Since \(B\) is positive definite, it is nonsingular, so \(c=0\). Thus \(A=aI_2\), and the product is \(\Sigma=I_2\otimes(aB)\).

The resulting covariance consists of two identical diagonal blocks \(aB\), with zero cross-block covariance. The product depends on \(u=ab\) and \(v=ad\), so it can be written directly as \(\Sigma=I_2\otimes\begin{pmatrix}u&v\\v&u\end{pmatrix}\). Positive definiteness requires \(u>0\) and \(u^2-v^2>0\), equivalently \(u>|v|\). The combined covariance model therefore has two free parameters, \(u\) and \(v\).

The same count follows by normalizing \(b=1\): then \(a>0\), \(d\in(-1,1)\), and the product has parameters \(a\) and \(d\), again two. Without a normalization, counting \(a,b,d\) as three covariance parameters would count the scale ambiguity twice.

A Reliable Workflow for Combined Restrictions

State the matrix indexing and restrictions.
Specify the variable ordering, block partition, lag definition, and which factor or full matrix is required to have each structure.

Write all equalities together.
For linear restrictions, express them in one system \(Cs(\Sigma)=d\). For a Kronecker model, write the factorization and impose restrictions on the stated factors or product.

Check consistency and remove redundant constraints.
Use the rank of the combined constraints, not a sum of separate constraint counts. For nonlinear product restrictions, inspect the actual joint parameterization.

Check positive definiteness and count parameters.
Establish that at least one allowed matrix is positive definite. Then count the dimension of the joint model, accounting for Kronecker scale normalization where needed.

Why the Combined Count Matters

The number of free parameters describes the size of the covariance model and affects how much the model constrains estimation. A count that ignores overlapping equalities can exaggerate flexibility; a count that ignores a Kronecker scale ambiguity can mistake different factor values for different covariances. Conversely, a formally consistent set of restrictions can leave no positive-definite covariance at all. Keeping the equality constraints, parameterization, and positive-definiteness check separate makes these issues visible.

Common mistake: “The Toeplitz model has \(p\) parameters and the block model has \(k\), so their intersection has \(p+k\) parameters.” This adds parameter counts rather than describing the intersection; in general, the structures impose overlapping constraints, and the intersection can be smaller than either model. For linear restrictions, combine the equations and use their joint rank. For Kronecker restrictions, parameterize the joint factors, remove scale redundancy, and verify that the resulting matrices are positive definite.

Key takeaway: A combined covariance model is defined by all restrictions holding at once. Check algebraic compatibility, check positive definiteness, and count the dimension of the joint model after removing redundant constraints and parameterization ambiguities.

Check Your Understanding

For each question, distinguish the equality constraints from the positive-definiteness requirement.

Why does a block-diagonal Toeplitz \(4\times4\) covariance for the partition \(\{1,2\}\mid\{3,4\}\) have only one free parameter?
What rank condition determines whether a linear system \(Cs=d\) is consistent?
Can consistent linear covariance restrictions still fail to define a positive-definite model? Give one reason.
Why is the raw count of entries in two Kronecker factors generally too large by one?
When combining restrictions, why is it incorrect to add the separate parameter counts of the component structures?