Tensor Denoising Under Gaussian Noise

Gaussian Tensor Denoising and Reconstruction Risk

In tensor denoising, the observed array contains both structured signal and random error. Retaining more components can represent the signal more accurately, but it also gives more noise directions an opportunity to enter the reconstruction. The central comparison is between truncated Tucker projection, which projects onto selected mode subspaces, and penalized low-rank estimation, which trades fit against a cost for model complexity.

Let \(X\in\mathbb{R}^{d_1\times\cdots\times d_K}\) be a fixed signal tensor and observe \(Y=X+E\). Throughout, assume the entries of \(E\) are independent \(N(0,\sigma^2)\). The squared reconstruction loss is the squared Frobenius norm, \(\|\widehat X-X\|_F^2\), and risk means its expectation over the noise. The calculations below concern this isotropic noise model; correlated tensor-normal noise requires accounting for its covariance structure.

Definition: For each mode \(k\), let \(U_k\in\mathbb{R}^{d_k\times r_k}\) have orthonormal columns, and set \(P_k=U_kU_k^\mathsf{T}\). The fixed-subspace Tucker projection of \(Y\) is \(\widehat X_{\boldsymbol r}=Y\times_1P_1\times_2\cdots\times_KP_K\). Its output lies in the Tucker subspace whose mode-\(k\) factors are contained in the column space of \(U_k\). The subspaces are fixed independently of \(E\). Its total retained dimension is \(q=\prod_{k=1}^K r_k\).

The operator \(P_1\otimes\cdots\otimes P_K\) represents this projection after vectorizing the tensor. It is an orthogonal projector of rank \(q\). This fixed-subspace condition matters: the usual truncated HOSVD chooses its mode subspaces from the observed tensor, so its projection depends on the noise and does not have the same simple risk identity.

Theorem: Under the stated independent Gaussian noise model, for fixed mode subspaces, \[ E\|\widehat X_{\boldsymbol r}-X\|_F^2 = \|X-X\times_1P_1\times_2\cdots\times_KP_K\|_F^2 +\sigma^2\prod_{k=1}^K r_k. \] The first term is squared approximation bias; the second is variance from the retained noise coordinates.

To prove the formula, write \(P=P_1\otimes\cdots\otimes P_K\), \(x=\operatorname{vec}(X)\), and \(e=\operatorname{vec}(E)\). Then the vectorized error is \(Px+Pe-x=(Px-x)+Pe\). Since \(P\) is an orthogonal projector, \(Px-x\) is orthogonal to the range of \(P\), while \(Pe\) lies in that range. Thus their inner product is zero for every realization. Also, \(E\|Pe\|^2=\sigma^2\operatorname{tr}(P)=\sigma^2q\), because \(E[ee^\mathsf{T}]=\sigma^2 I\) and \(P^\mathsf{T}P=P\). Taking expectations gives the displayed identity. \(\square\)

Worked Example: Bias and Variance in a Fixed Tucker Subspace

Consider a \(3\times4\times2\) tensor. Let \(P_1\) project onto the first coordinate, \(P_2\) onto the span of the first two coordinates, and \(P_3\) onto the first coordinate. Suppose the only nonzero entries of \(X\) are \(X_{111}=3\), \(X_{121}=1\), and \(X_{211}=2\).

The projection retains entries with \(i=1\), \(j\in\{1,2\}\), and \(k=1\). It therefore retains the first two signal entries and discards \(X_{211}=2\). The squared approximation error is \(2^2=4\). The retained dimension is \(q=1\cdot2\cdot1=2\), so the projected noise contributes expected squared error \(2\sigma^2\). Hence \(E\|\widehat X_{\boldsymbol r}-X\|_F^2=4+2\sigma^2\). For \(\sigma^2=1\), the risk is \(6\).

Truncation and Data-Adaptivity

A truncated HOSVD computes leading left singular vectors of the mode unfoldings of \(Y\), keeps a prescribed number in each mode, and projects onto the resulting subspaces. Unlike the fixed-subspace estimator above, these vectors depend on the observed noise. Consequently, one cannot simply substitute the chosen ranks into the fixed-projection variance term and claim that it is the exact risk: the random projector and the noise are dependent.

The fixed-subspace calculation is still a useful benchmark. It says exactly what would happen if the mode spaces were specified in advance, and it isolates the two basic costs: signal discarded outside the spaces and noise admitted inside them. Data-adaptive truncation can align its spaces with signal, but it can also align them with unusually large noise fluctuations. Assessing that extra effect requires analysis beyond the fixed-projector identity.

Worked Example: Rank Choice in an Orthogonal Component Model

Consider a three-way \(3\times3\times3\) tensor with signal \(X=1.5\,e_1\otimes e_1\otimes e_1+0.8\,e_2\otimes e_2\otimes e_2\). Restrict attention to candidate projections that retain the first \(r\) of these mutually orthogonal rank-one components, for \(r=0,1,2\). The corresponding projection dimensions are \(0,1,2\).

With \(\sigma^2=0.25\), retaining no component has risk \(1.5^2+0.8^2=2.89\). Retaining only the first has squared bias \(0.8^2=0.64\) and variance \(0.25\), giving risk \(0.89\). Retaining both has zero approximation bias and variance \(2(0.25)=0.50\). Thus the best of these three fixed candidates retains both components.

This comparison is an oracle calculation: it uses the signal coefficients to evaluate candidate risks. In practice the signal is unknown. A penalized criterion replaces that oracle choice with a data-dependent one, and the next example shows how a noise-only component can then be retained.

Penalized Low-Rank Estimation

A general rank-penalized estimate minimizes residual error plus a cost for the fitted Tucker complexity. One convenient complexity proxy is \(C(\boldsymbol r)=\prod_k r_k+\sum_k d_k r_k\): the first term counts core coordinates and the remaining terms count factor coordinates before accounting for nonidentifiability. The penalty is a model-selection device, not an exact count of independent parameters.

Definition: For a candidate tensor \(T\), let \(\boldsymbol r(T)\) be its multilinear rank, the tuple of matrix ranks of its mode unfoldings. Given \(\lambda\geq0\), define a penalized fit as a minimizer \[ \widehat T\in\arg\min_{T:\,\boldsymbol r(T)\leq\boldsymbol d} \left\{\|Y-T\|_F^2+\lambda\sigma^2 C(\boldsymbol r(T))\right\}, \qquad C(\boldsymbol r)=\prod_k r_k+\sum_k d_k r_k. \] Here \(\boldsymbol d=(d_1,\ldots,d_K)\). The zero tensor is allowed, with zero rank and zero complexity.

The objective directly expresses the fit–complexity trade-off. A larger \(\lambda\) requires a larger reduction in residual sum of squares before a more complex rank is worthwhile. The rank-penalized problem can be computationally difficult; this definition specifies the statistical target, not a claim that a global minimizer is easy to compute.

Theorem: If \(\widehat T\) is a global minimizer of the stated penalized objective, then for every candidate tensor \(T\), \[ \|\widehat T-X\|_F^2 \leq \|T-X\|_F^2 +2\langle E,\widehat T-T\rangle +\lambda\sigma^2\{C(\boldsymbol r(T))-C(\boldsymbol r(\widehat T))\}. \]

This comparison inequality is exact but does not, by itself, bound expected risk: the fitted tensor depends on \(E\), so the inner product cannot be discarded as if it were independent and mean zero. It clarifies why complexity control is needed. A penalty can discourage models that exploit noise, while a penalty that is too large can exclude genuine signal and increase approximation bias.

Worked Example: A Noise-Only Component Inflates Rank

Consider the restricted rank-one family \(T=\alpha u\), where \(u\) is a fixed unit-norm rank-one tensor and \(\alpha\in\mathbb{R}\), together with the zero tensor. Suppose the true signal is \(X=0\) and the observed coefficient \(Y_u=\langle Y,u\rangle\) is \(N(0,\sigma^2)\). Penalize the nonzero rank-one fit by \(2\lambda\), so the objective is squared error plus \(2\lambda\) when \(\alpha\ne0\).

The best nonzero coefficient is \(\widehat\alpha=Y_u\), with residual zero along \(u\). The zero fit has residual \(Y_u^2\). The penalized estimator therefore retains the component exactly when \(Y_u^2>2\lambda\), and its estimate is \(\widehat\alpha=Y_u\,\mathbf{1}\{|Y_u|>\sqrt{2\lambda}\}\). A nonzero estimate in this noise-only setting is rank inflation.

Set \(\sigma^2=1\) and \(\lambda=2\). Then the threshold is \(2\), and the probability of retaining a spurious component is \(P(|Z|>2)=2\Phi(-2)\approx0.0455\), where \(Z\sim N(0,1)\). Its expected squared reconstruction error is \(E[Z^2\mathbf{1}\{|Z|>2\}]=2\{2\phi(2)+\Phi(-2)\}\approx0.2615\). Indeed, integration by parts gives \(\int_2^\infty z^2\phi(z)\,dz=2\phi(2)+\Phi(-2)\), and symmetry doubles this upper-tail value. The zero estimator has risk zero here; penalization reduces, but does not eliminate, noise-driven rank inflation.

Exact Threshold Risk and a Common Pitfall

The one-component example has a useful general form. If the true coefficient is \(\theta\), write \(Y_u=\theta+\sigma Z\), with \(Z\sim N(0,1)\), and retain \(Y_u\) when \(|Y_u|>t\). Define \(a=(t-\theta)/\sigma\) and \(b=(-t-\theta)/\sigma\). Since \(t\geq0\), \(b\leq a\). Retention is the event \(Z>a\) or \(Z<b\); omission is \(b\leq Z\leq a\). On retention the estimation error is \(\sigma Z\), and on omission it is \(-\theta\). Thus the risk is

E[(\widehat\theta-\theta)^2] =\sigma^2\left\{a\phi(a)+1-\Phi(a)+\Phi(b)-b\phi(b)\right\} +\theta^2\{\Phi(a)-\Phi(b)\}.

The tail terms follow from integration by parts: \(E[Z^2\mathbf{1}\{Z>a\}]=a\phi(a)+1-\Phi(a)\) and \(E[Z^2\mathbf{1}\{Z<b\}]=\Phi(b)-b\phi(b)\). The middle-interval probability is \(\Phi(a)-\Phi(b)\).

Pitfall: reversing the transformed tail events. It is wrong to transform the two retention inequalities into unrelated sign-reversed bounds. From \(\theta+\sigma Z>t\) one gets \(Z>a\); from \(\theta+\sigma Z<-t\) one gets \(Z<b\). The correct retention event is \(\{Z>a\text{ or }Z<b\}\), not \(\{Z>b\text{ or }Z<-a\}\).
Pitfall: treating a selected projector as fixed. The \(\sigma^2\prod_k r_k\) variance term is proved for subspaces fixed independently of the noise. A rank or subspace chosen after inspecting \(Y\) has selection effects that the fixed-projection identity does not capture.

Practical Interpretation

For a specified Tucker subspace, reconstruction risk has a transparent decomposition: bias is the signal energy missed by the projection, and variance is noise energy admitted through its retained coordinates. Penalized low-rank estimation extends the comparison by allowing the fit to choose its complexity, but that choice is random. The penalty should therefore be understood as protection against fitting noise, not as a guarantee that every retained component reflects signal.

A useful diagnostic principle is to ask whether a rank increase produces a substantial improvement in fit relative to the noise scale and the complexity added. In a fixed subspace, the variance cost is explicit. Under data-adaptive selection, a large observed component may be signal or an unusually large noise fluctuation; rank selection must account for that distinction rather than interpreting every improvement in residual error as evidence for additional structure.

Check Your Understanding

Use the definitions and calculations above to answer the following questions.

Why is the projected-noise contribution for a fixed Tucker subspace \(\sigma^2\prod_k r_k\)? State the role of orthogonality.
For a fixed-subspace projection, which part of the risk changes if the signal changes while the projectors and noise variance remain fixed?
In the one-component penalized example, derive the threshold condition for retaining the component by comparing the two objective values.
Explain why the fixed-subspace risk identity cannot be applied directly to mode subspaces estimated from the noisy tensor.
For the threshold rule, write the retention event in terms of \(Z\), \(a\), and \(b\), and distinguish it from the omission event.