Tensor reshaping

In multilinear algebra, a reshaping of tensors is any bijection between the set of indices of an order-$$M$$ tensor and the set of indices of an order-$$L$$ tensor, where $$L < M $$. The use of indices presupposes tensors in coordinate representation with respect to a basis. The coordinate representation of a tensor can be regarded as a multi-dimensional array, and a bijection from one set of indices to another therefore amounts to a rearrangement of the array elements into an array of a different shape. Such a rearrangement constitutes a particular kind of linear map between the vector space of order-$$M$$ tensors and the vector space of order-$$L$$ tensors.

Definition
Given a positive integer $$M$$, the notation $$[M]$$ refers to the set $$\{1, \dots, M \}$$ of the first $M$ positive integers.

For each integer $$m$$ where $$1 \le m \le M $$ for a positive integer $$M$$, let $$V_m$$ denote an $$I_m$$-dimensional vector space over a field $$F$$. Then there are vector space isomorphisms (linear maps)

$$\begin{align} V_1 \otimes \cdots \otimes V_M & \simeq F^{I_1} \otimes \cdots \otimes F^{I_M} \\ & \simeq F^{I_{\pi_1}} \otimes \cdots \otimes F^{I_{\pi_M}} \\ & \simeq F^{I_{\pi_1} I_{\pi_2}} \otimes F^{I_{\pi_3}} \otimes \cdots \otimes F^{I_{\pi_M}} \\ & \simeq F^{I_{\pi_1} I_{\pi_3}} \otimes F^{I_{\pi_2}} \otimes F^{I_{\pi_4}} \otimes \cdots \otimes F^{I_{\pi_M}} \\ & \,\,\,\vdots \\ & \simeq F^{I_1 I_2 \ldots I_M}, \end{align} $$

where $$\pi \in \mathfrak{S}_M$$ is any permutation and $$\mathfrak{S}_M $$ is the symmetric group on $$M$$ elements. Via these (and other) vector space isomorphisms, a tensor can be interpreted in several ways as an order-$$L$$ tensor where $$L \le M$$.

Coordinate representation
The first vector space isomorphism on the list above, $$V_1 \otimes \cdots \otimes V_M \simeq F^{I_1} \otimes \cdots \otimes F^{I_M}$$, gives the coordinate representation of an abstract tensor. Assume that each of the $$M$$ vector spaces $$V_m$$ has a basis $$\{ v_1^m, v_2^m, \ldots, v_{I_m}^m \}$$. The expression of a tensor with respect to this basis has the form $$\mathcal{A} = \sum_{i_1=1}^{I_1}\ldots\sum_{i_M=1}^{I_M} a_{i_1,i_2,\ldots,i_M} v_{i_1}^1 \otimes v_{i_2}^2 \otimes \cdots \otimes v_{i_M}^{M},$$ where the coefficients $$a_{i_1,i_2,\ldots,i_M}$$ are elements of $$F$$. The coordinate representation of $$\mathcal{A}$$ is $$\sum_{i_1=1}^{I_1}\ldots\sum_{i_M=1}^{I_M} a_{i_1,i_2,\ldots,i_M} \mathbf{e}_{i_1}^1 \otimes \mathbf{e}_{i_2}^2 \otimes \cdots \otimes \mathbf{e}_{i_M}^M,$$where $$\mathbf{e}_{i}^m$$ is the $$i^\text{th}$$ standard basis vector of $$F^{I_m}$$. This can be regarded as a M-way array whose elements are the coefficients $$a_{i_1,i_2,\ldots,i_M}$$.

General flattenings
For any permutation $$\pi \in \mathfrak{S}_M$$ there is a canonical isomorphism between the two tensor products of vector spaces $$V_1 \otimes V_2 \otimes \cdots \otimes V_M$$ and $$V_{\pi(1)} \otimes V_{\pi(2)} \otimes \cdots \otimes V_{\pi(M)}$$. Parentheses are usually omitted from such products due to the natural isomorphism between $$V_i\otimes(V_j\otimes V_k)$$ and $$(V_i\otimes V_j)\otimes V_k$$, but may, of course, be reintroduced to emphasize a particular grouping of factors. In the grouping, $$(V_{\pi(1)} \otimes \cdots \otimes V_{\pi(r_1)})\otimes(V_{\pi(r_1+1)} \otimes \cdots \otimes V_{\pi(r_2)})\otimes\cdots\otimes(V_{\pi(r_{L-1}+1)} \otimes \cdots \otimes V_{\pi(r_L)}),$$ there are $$L$$ groups with $$r_l-r_{l-1}$$ factors in the $$l^\text{th}$$ group (where $$r_0=0$$ and $$r_L=M$$).

Letting $$S_l=(\pi(r_{l-1}+1),\pi(r_{l-1}+2),\ldots,\pi(r_l))$$ for each $$l$$ satisfying $$1\le l\le L$$, an $$ (S_1,S_2,\ldots,S_L)$$-flattening of a tensor $$\mathcal{A}$$, denoted $$\mathcal{A}_{(S_1,S_2,\ldots,S_L)}$$, is obtained by applying the two processes above within each of the $$L$$ groups of factors. That is, the coordinate representation of the $$l^\text{th}$$ group of factors is obtained using the isomorphism $$(V_{\pi(r_{l-1}+1)} \otimes V_{\pi(r_{l-1}+2)} \otimes \cdots \otimes V_{\pi(r_l)})\simeq(F^{I_{\pi(r_{l-1}+1)}}\otimes F^{I_{\pi(r_{l-1}+2)}}\otimes\cdots\otimes F^{I_{\pi(r_l)}})$$, which requires specifying bases for all of the vector spaces $$V_k$$. The result is then vectorized using a bijection $$\mu_l:[I_{\pi(r_{l-1}+1)}]\times[I_{\pi(r_{l-1}+2)}]\times\cdots\times[I_{\pi(r_l)}]\to[I_{S_l}]$$ to obtain an element of $$F^{I_{S_l}}$$, where $I_{S_l} := \prod_{i=r_{l-1}+1}^{r_l} I_{\pi(i)}$, the product of the dimensions of the vector spaces in the $$l^\text{th}$$ group of factors. The result of applying these isomorphisms within each group of factors is an element of $$ F^{I_{S_1}} \otimes \cdots \otimes F^{I_{S_L}}$$, which is a tensor of order $$L$$.

Vectorization
By means of a bijective map $$ \mu : [I_1] \times \cdots \times [I_M] \to [I_1\cdots I_M] $$, a vector space isomorphism between $$ F^{I_1} \otimes \cdots \otimes F^{I_M} $$ and $$ F^{I_1 \cdots I_M} $$ is constructed via the mapping $$ \mathbf{e}_{i_1}^1 \otimes \cdots \mathbf{e}_{i_m}^m \otimes \cdots \otimes \mathbf{e}_{i_M}^M \mapsto \mathbf{e}_{\mu(i_1,i_2,\ldots,i_M)}, $$ where for every natural number $$i$$ such that $$1 \le i \le I_1 \cdots I_M$$, the vector $$ \mathbf{e}_i $$ denotes the ith standard basis vector of $$ F^{i_1 \cdots i_M} $$. In such a reshaping, the tensor is simply interpreted as a vector in $$ F^{I_1 \cdots I_M}$$. This is known as vectorization, and is analogous to vectorization of matrices. A standard choice of bijection $$\mu$$ is such that

which is consistent with the way in which the colon operator in Matlab and GNU Octave reshapes a higher-order tensor into a vector. In general, the vectorization of $$\mathcal{A}$$ is the vector $$ [ a_{\mu^{-1}(i)} ]_{i=1}^{I_1 \cdots I_M} $$.

The vectorization of $$\mathcal{A}$$ denoted with $$vec(\mathcal{A})$$ or $$\mathcal{A}_{[:]}$$ is an $$[S_1,S_2]$$-reshaping where $$S_1 = (1,2,\ldots,M)$$ and $$S_2=\empty$$.

Mode-m Flattening / Mode-m Matrixization
Let $$\mathcal{A} \in F^{I_1} \otimes F^{I_2} \otimes \cdots \otimes F^{I_M}$$ be the coordinate representation of an abstract tensor with respect to a basis. Mode-m matrixizing (a.k.a. flattening) of $$\mathcal{A}$$ is an $$ [S_1, S_2]$$-reshaping in which $$ S_1 = (m)$$ and $$ S_2 = (1,2,\ldots,m-1,m+1,\ldots,M)$$. Usually, a standard matrixizing is denoted by

This reshaping is sometimes called matrixizing, matricizing, flattening or unfolding in the literature. A standard choice for the bijections $$\mu_1,\ \mu_2$$ is the one that is consistent with the reshape function in Matlab and GNU Octave, namely

Definition Mode-m Matrixizing: $$[{\mathbf A}_{[m]}]_{jk} = a_{i_1\dots i_m\dots i_M}, \;\; \text{ where } j = i_m \text{ and } k=1+\sum_{n=0\atop n\neq m}^M(i_n - 1) \prod_{l=0\atop l\neq m}^{n-1} I_l.$$ The mode-m matrixizing of a tensor $${\mathcal A} \in F^{I_1\times...I_M},$$ is defined as the matrix $${\mathbf A}_{[m]} \in F^{I_m \times (I_1 \dots I_{m-1} I_{m+1} \dots I_M)}$$. As the parenthetical ordering indicates, the mode-m column vectors are arranged by sweeping all the other mode indices through their ranges, with smaller mode indexes varying more rapidly than larger ones; thus