Diagnostics for Tensor-Normal and Low-Rank Fits

Why Tensor-Model Diagnostics Need Structure

A small reconstruction error does not by itself establish that a tensor model is adequate. Errors can be large in directions where the model allows high variability and troublingly systematic in directions where variability should be small. Likewise, a low-rank fit can leave residual structure that is hidden by a single overall error norm. Diagnostics should therefore examine residual size relative to the fitted covariance, patterns along each mode, and sensitivity to modeling choices.

Let an observed \(K\)-way tensor have dimensions \(d_1,\ldots,d_K\), and let \(\widehat{\mathcal M}\) denote its fitted mean or low-rank reconstruction. Write \(\mathcal R=\mathcal X-\widehat{\mathcal M}\) for the residual tensor. For a tensor-normal model, suppose the mode covariance matrices \(\Sigma_k\) are positive definite. We use the convention that mode 1 varies fastest under vectorization, so the covariance of the vectorized noise is \(\Sigma_K\otimes\cdots\otimes\Sigma_1\).

Whitening puts residual components on a common scale and removes the modeled separable covariance when the covariance factors are correct. In a real fit, however, the mean, rank, and covariance factors are usually estimated from the same data. Consequently, whitened fitted residuals should not automatically be treated as independent standard normal observations. The reference results below clarify both the ideal case and a special fitted case where the effect of fitting is known exactly.

Whitening and the Reference Residual Norm

First suppose the mean tensor \(\mathcal M\) and all covariance factors are known, and the data follow a tensor-normal distribution with these parameters. The residual \(\mathcal R=\mathcal X-\mathcal M\) is then the actual noise, not a residual formed after estimating the mean.

Proof. Under the stated vectorization convention, \(\operatorname{vec}(\mathcal U)=(\Sigma_K^{-1/2}\otimes\cdots\otimes\Sigma_1^{-1/2})\operatorname{vec}(\mathcal R)\). The covariance of this vector is \[ (\Sigma_K^{-1/2}\otimes\cdots\otimes\Sigma_1^{-1/2}) (\Sigma_K\otimes\cdots\otimes\Sigma_1) (\Sigma_K^{-1/2}\otimes\cdots\otimes\Sigma_1^{-1/2})=I_D. \] The vector is Gaussian, so identity covariance implies its coordinates are independent standard normal variables. The squared Frobenius norm is the sum of their \(D\) squares, which has a chi-square distribution with \(D\) degrees of freedom. \(\square\)

This result gives a useful calibration only when the mean and covariance are known, or when a justified procedure accounts for estimation. It is not a general claim about residuals from a fitted low-rank model. Fitting uses the observed data and typically removes some residual directions, while estimating covariance factors can also alter the residual scale.

Mode-Wise Covariance Checks

A global norm can conceal structure. After whitening, inspect residuals along each mode as well. For mode \(k\), let \(U_{(k)}\) be the mode-\(k\) unfolding of \(\mathcal U\), with \(d_k\) rows and \(m_k=\prod_{j\ne k}d_j\) columns. A basic diagnostic is \(S_k=U_{(k)}U_{(k)}^\mathsf T/m_k\). If the entries of \(\mathcal U\) are independent standard normal variables, each column of this unfolding is an independent \(N(0,I_{d_k})\) vector.

To see this, write the columns of \(U_{(k)}\) as \(u_1,\ldots,u_{m_k}\). Each has \(E[u_j u_j^\mathsf T]=I_{d_k}\). Therefore \(E[S_k]=m_k^{-1}\sum_{j=1}^{m_k}E[u_j u_j^\mathsf T]=I_{d_k}\). Large diagonal entries can signal remaining excess variance along a mode; substantial off-diagonal entries can signal residual dependence that the fitted separable covariance did not remove. These are diagnostic comparisons, not automatic hypothesis tests: with few columns, substantial sampling variation is expected.

For fitted models, these same summaries remain useful plots or descriptive checks, but their reference behavior changes. In particular, residuals from a flexible low-rank fit may be constrained by the fitted subspace. A mode covariance that looks unusually small can reflect fitting rather than a genuine reduction in noise; a residual pattern that remains strong after fitting may instead indicate a deficient rank or covariance model.

Reconstruction Error, Rank, and Covariance Scale

For complete observations, the covariance-adjusted reconstruction error is \(\|\mathcal U\|_F^2\). When only a subset of entries is observed, calculate residuals on those entries rather than replacing unobserved values by zero. The appropriate covariance metric for the observed vector is the covariance submatrix restricted to the observed coordinates. Whitening the full tensor and then treating missing entries as observed zeros changes the problem and can create artificial residual structure.

Rank sensitivity is central to interpreting reconstruction residuals. Increasing rank can only improve the best achievable unweighted reconstruction error for nested rank-constrained model classes, because the lower-rank fits remain available at the higher rank. A smaller residual norm alone therefore does not establish that a larger rank is preferable. Compare the residual patterns and covariance-adjusted errors across plausible ranks, and use the stability considerations associated with rank selection rather than treating the training residual as a stand-alone selection rule.

Separable covariance factors also have a scale nonidentifiability: multiplying one factor by a positive constant and another by its reciprocal leaves their Kronecker product unchanged. A diagnostic should depend on the covariance product, not on an arbitrary allocation of scale among factors. The following calculation shows the distinction between equivalent factor normalizations and a change in the overall covariance scale.

What Fitting Does to Residual Reference Distributions

The effect of fitting can be derived exactly for a Gaussian mean restricted to a fixed linear subspace. This result is a useful benchmark, but a general nonlinear low-rank tensor fit need not be an orthogonal projection onto a fixed subspace.

Proof. The least-squares projection matrix is \(H=A(A^\mathsf T A)^{-1}A^\mathsf T\). It is symmetric, idempotent, and has rank \(p\). The residual vector is \(r=(I_m-H)z\). Since \((I_m-H)A\beta=0\), its mean is zero, and its covariance is \((I_m-H)I_m(I_m-H)^\mathsf T=I_m-H\). Choose an orthonormal basis in which the projection \(H\) has \(p\) diagonal entries equal to 1 and \(m-p\) equal to 0. In that basis, \(r\) has \(p\) coordinates identically zero and \(m-p\) independent \(N(0,1)\) coordinates. Hence \(r^\mathsf T r\), the sum of the squared residual coordinates, has a \(\chi^2_{m-p}\) distribution. \(\square\)

If a tensor-normal model has known covariance, whitening reduces its Gaussian noise to the setting of this theorem only when the mean is fitted in a fixed linear subspace. A low-rank tensor model often estimates several interacting factors and may impose nonlinear constraints; its fitted residuals do not automatically have a chi-square distribution with “number of entries minus number of parameters” degrees of freedom. Use a suitable model-specific calibration when formal tail probabilities are needed. For exploratory assessment, examine whitened residual plots, mode-wise summaries, and error stability across ranks, and be explicit about which quantities were estimated from the same data.

Common Pitfalls and Practical Interpretation

• Using raw error as if all entries had equal noise. A large raw residual can be unsurprising in a high-variance direction. Compare the covariance-adjusted error and the residual pattern after whitening.
• Calling fitted whitened residuals independent standard normal. That is the exact reference only under the stated known-parameter setting. Fitting generally changes their dependence and scale.
• Treating one mode-wise covariance estimate as decisive. The number of columns may be small, and fitted residuals do not follow the ideal independent-entry reference exactly. Interpret the matrix alongside sample size and other diagnostics.
• Choosing the largest rank because its training residual is smallest. More flexible nested fits can reduce training error mechanically. Inspect whether the improvement resolves a meaningful pattern and whether it is stable.
• Comparing covariance factors without accounting for normalization. Equivalent scale allocations can make individual factors look different even when their Kronecker product and residual diagnostics are identical.
• Filling missing entries with zero before calculating residual norms. This treats unobserved values as real measurements. Use the observed-coordinate residual and its corresponding covariance.