Tensor Notation, Modes, and Unfoldings

Why Tensor Indices Need a Convention

A multiway data set may record the same variables across several dimensions: for example, measurements by participant, visit, and outcome. Its entries can be stored in a single long vector or arranged in a matrix, but either conversion needs an ordering rule. Without one, “flatten the array” does not specify which entry goes in which position. This tutorial defines tensor notation, mode fibers, matricization, and folding with one explicit convention.

The key choice is how to order the indices that become matrix columns. We use a column-major convention: among the indices other than the selected mode, the one with the smallest mode number varies fastest. Other conventions are possible, but their matrices need not have the same column order.

Definition: An order-\(d\) array \(X\) with dimensions \(I_1\times\cdots\times I_d\) is a collection of entries \(X_{i_1,\ldots,i_d}\), where \(i_m\in\{1,\ldots,I_m\}\) for each mode \(m\). The integer \(d\) is the order of the array, and each index \(i_m\) identifies a position along mode \(m\). Unless stated otherwise, entries are real numbers.

A vector is an order-1 array, and a matrix is an order-2 array. An order-3 array is often called a three-way array or a third-order tensor. Here, “tensor” refers to the array of entries and its indexing; no additional transformation rule is needed for the definitions that follow.

Fibers and Matricizations

A mode-\(n\) fiber is obtained by varying index \(i_n\) while holding every other index fixed. It is a vector of length \(I_n\). Matricization puts these fibers into the columns of a matrix, with the column locations determined by the stated ordering convention.

Definition: For mode \(n\), fix an index \(i_m\) for every \(m\ne n\). The corresponding mode-\(n\) fiber is the vector whose \(i_n\)-th entry is \(X_{i_1,\ldots,i_d}\), as \(i_n\) ranges from \(1\) to \(I_n\). The mode-\(n\) matricization, denoted \(X_{(n)}\), is an \(I_n\times\prod_{m\ne n}I_m\) matrix that places each such fiber in one column.

To specify the column precisely, list the modes other than \(n\) in increasing order as \(s_1,\ldots,s_{d-1}\). For a tensor entry with indices \(i_1,\ldots,i_d\), its row in \(X_{(n)}\) is \(i_n\), and its column is

c_n(i_1,\ldots,i_d) =1+\sum_{r=1}^{d-1}(i_{s_r}-1)\prod_{q=1}^{r-1}I_{s_q}.

An empty product is \(1\). Thus the first remaining mode, \(s_1\), varies fastest; the next one varies only after \(s_1\) has run through all its values. Equivalently, the column number is the one-based mixed-radix encoding of the remaining indices.

Worked Three-Way Indexing

Worked Example: Unfolding a Three-Way Array

Let \(X\) have dimensions \(2\times3\times2\), with entries \(X_{ijk}=100(i-1)+10(j-1)+k\). Its two slices at fixed \(k\), with rows indexed by \(i\) and columns by \(j\), are

X(:,:,1)= \begin{bmatrix} 1&11&21\\ 101&111&121 \end{bmatrix}, \qquad X(:,:,2)= \begin{bmatrix} 2&12&22\\ 102&112&122 \end{bmatrix}.

For mode 1, the remaining modes are \(2,3\), and \(j\) varies fastest. The column order is therefore \((j,k)=(1,1),(2,1),(3,1),(1,2),(2,2),(3,2)\). Row \(i=1\) contains the entries with \(i=1\); row \(i=2\) contains the entries with \(i=2\):

X_{(1)}= \begin{bmatrix} 1&11&21&2&12&22\\ 101&111&121&102&112&122 \end{bmatrix}.

For mode 2, the remaining modes are \(1,3\), so \(i\) varies fastest. The columns correspond to \((i,k)=(1,1),(2,1),(1,2),(2,2)\), while the rows correspond to \(j=1,2,3\):

X_{(2)}= \begin{bmatrix} 1&101&2&102\\ 11&111&12&112\\ 21&121&22&122 \end{bmatrix}.

For mode 3, the remaining modes are \(1,2\), again with \(i\) varying fastest. The columns correspond to \((i,j)=(1,1),(2,1),(1,2),(2,2),(1,3),(2,3)\), and the rows correspond to \(k=1,2\):

X_{(3)}= \begin{bmatrix} 1&101&11&111&21&121\\ 2&102&12&112&22&122 \end{bmatrix}.

Each matrix contains all 12 entries exactly once. The three matrices differ in which index labels the rows and in the ordering of the remaining indices in their columns.

Calculating a Column Index

Worked Example: Finding a Mode-3 Matrix Location

Suppose \(X\) has dimensions \(4\times3\times5\times2\). Find the location in \(X_{(3)}\) of \(X_{3,2,4,2}\).

The row is the mode-3 index, so the row number is \(i_3=4\). The remaining modes, in increasing order, are \(1,2,4\). Their indices are \(3,2,2\), and their dimensions are \(4,3,2\). Mode 1 varies fastest, so the column is

\begin{aligned} c_3 &=1+(i_1-1)+I_1(i_2-1)+I_1I_2(i_4-1)\\ &=1+(3-1)+4(2-1)+4\cdot3(2-1)\\ &=1+2+4+12\\ &=19. \end{aligned}

Thus \(X_{3,2,4,2}\) appears in row 4, column 19 of \(X_{(3)}\). The mode-3 matrix has \(5\) rows and \(4\cdot3\cdot2=24\) columns, so that location is within its dimensions.

Folding and the Inverse Mapping

Folding reverses a matricization: it places every matrix entry back at the tensor index that generated its row and column. To define it, use a matrix with the dimensions required for a mode-\(n\) unfolding, namely \(I_n\times\prod_{m\ne n}I_m\), and retain the same column-index formula.

Definition: The mode-\(n\) folding of a matrix \(M\) with \(I_n\) rows and \(\prod_{m\ne n}I_m\) columns is the order-\(d\) array \(X\) defined by \(X_{i_1,\ldots,i_d}=M_{i_n,c_n(i_1,\ldots,i_d)}\). It is denoted \(\operatorname{fold}_n(M)\). In particular, \(\operatorname{fold}_n(X_{(n)})=X\).

Worked Example: Folding a Mode-2 Matrix

Take the mode-2 matrix from the three-way example:

M= \begin{bmatrix} 1&101&2&102\\ 11&111&12&112\\ 21&121&22&122 \end{bmatrix}.

Here the dimensions are \(I_1=2\), \(I_2=3\), \(I_3=2\). The mode-2 columns are ordered by \((i,k)=(1,1),(2,1),(1,2),(2,2)\). For instance, \(X_{2,3,2}\) has row \(j=3\) and column \(1+(i-1)+I_1(k-1)=1+(2-1)+2(2-1)=4\). Thus its value is \(M_{3,4}=122\).

Applying the same decoding to all entries gives the slices at \(k=1\) and \(k=2\):

\operatorname{fold}_2(M)(:,:,1)= \begin{bmatrix} 1&11&21\\ 101&111&121 \end{bmatrix}, \qquad \operatorname{fold}_2(M)(:,:,2)= \begin{bmatrix} 2&12&22\\ 102&112&122 \end{bmatrix}.

These are the original slices, so folding \(X_{(2)}\) recovers \(X\) entry by entry.

Why Unfolding Is Reversible

Theorem: Under the stated column convention, mode-\(n\) matricization is a one-to-one rearrangement of the entries of \(X\). Folding is its inverse. In particular, if \(\|X\|_F^2=\sum_{i_1,\ldots,i_d}X_{i_1,\ldots,i_d}^2\), then \(\|X_{(n)}\|_F=\|X\|_F\).

Proof. Fix a row \(i_n\) and a column number \(c\) between \(1\) and \(\prod_{m\ne n}I_m\). Write \(c-1\) in mixed radix with radices \(I_{s_1},\ldots,I_{s_{d-1}}\). Successive remainders upon division by \(I_{s_1}, I_{s_2},\ldots\) uniquely determine the digits \(i_{s_1}-1,\ldots,i_{s_{d-1}}-1\), each in its allowed range. Hence every matrix location corresponds to exactly one tensor index tuple, and each tuple gives exactly one matrix location by the column formula.

By definition, matricization copies the entry at that tuple into its corresponding matrix location; folding uses the same unique correspondence in reverse. Therefore \(\operatorname{fold}_n(X_{(n)})=X\), and applying matricization after folding also returns the original matrix. Since every entry is copied once and only once, summing squared entries over the matrix is the same as summing squared entries over the array. Thus \(\|X_{(n)}\|_F^2=\|X\|_F^2\), and taking nonnegative square roots gives the norm equality. \(\square\)

Result: Each column of \(X_{(n)}\) is exactly one mode-\(n\) fiber, and every mode-\(n\) fiber occurs as one column.

Justification. Fix all indices other than \(i_n\). The column number \(c_n\) depends only on those fixed indices, not on \(i_n\). As \(i_n\) varies from \(1\) to \(I_n\), the corresponding matrix entries therefore occupy that column in rows \(1,\ldots,I_n\), in order. Conversely, the mixed-radix uniqueness in the proof above ensures that different fixed tuples give different columns and that every column corresponds to a fixed tuple. This establishes the claimed one-to-one correspondence between fibers and columns.

Common Mistakes and Practical Checks

The most frequent error is to calculate a correct set of columns in the wrong order. For example, in the mode-2 unfolding of a \(2\times3\times2\) array, ordering columns by \((k,i)\) rather than \((i,k)\) would still place every entry exactly once, but it would not follow the convention used here. Any calculation that assumes a particular column corresponds to a particular fiber could then be wrong.

A reliable check is to state the remaining modes in increasing order and identify which index varies fastest before writing the matrix. Also check that the unfolding has \(I_n\) rows, \(\prod_{m\ne n}I_m\) columns, and exactly the same number of entries as the original array. These checks detect many indexing errors, although a wrong ordering can still have the correct dimensions.

Check Your Understanding

Use the definitions and column convention above to answer the following.

What are the dimensions of the mode-2 matricization of an order-4 array with dimensions \(3\times5\times2\times4\)?
For an array with dimensions \(2\times3\times4\), which remaining index varies fastest in the mode-3 matricization?
For the same array, find the row and column of \(X_{2,3,4}\) in \(X_{(3)}\).
What information is needed to fold a matrix into an order-\(d\) array, besides the matrix entries?
Why does matricization preserve the Frobenius norm, even though it changes the shape of the data?