Multilinear multiplication

In multilinear algebra, applying a map that is the tensor product of linear maps to a tensor is called a multilinear multiplication.

Abstract definition
Let $$F $$ be a field of characteristic zero, such as $$\mathbb{R} $$ or $$\mathbb{C} $$. Let $$V_k$$ be a finite-dimensional vector space over $$F$$, and let $$\mathcal{A} \in V_1 \otimes V_2 \otimes \cdots \otimes V_d$$ be an order-d simple tensor, i.e., there exist some vectors $$\mathbf{v}_k \in V_k$$ such that $$\mathcal{A} = \mathbf{v}_1 \otimes \mathbf{v}_2 \otimes \cdots \otimes \mathbf{v}_d$$. If we are given a collection of linear maps $$A_k : V_k \to W_k$$, then the multilinear multiplication of $$\mathcal{A}$$ with $$(A_1, A_2, \ldots, A_d)$$ is defined as the action on $$\mathcal{A}$$ of the tensor product of these linear maps, namely

$$ \begin{align} A_1 \otimes A_2 \otimes \cdots \otimes A_d : V_1 \otimes V_2 \otimes \cdots \otimes V_d & \to W_1 \otimes W_2 \otimes \cdots \otimes W_d, \\ \mathbf{v}_1 \otimes \mathbf{v}_2 \otimes \cdots \otimes \mathbf{v}_d & \mapsto A_1(\mathbf{v}_1) \otimes A_2(\mathbf{v}_2) \otimes \cdots \otimes A_d(\mathbf{v}_d) \end{align} $$

Since the tensor product of linear maps is itself a linear map, and because every tensor admits a tensor rank decomposition, the above expression extends linearly to all tensors. That is, for a general tensor $$\mathcal{A} \in V_1 \otimes V_2 \otimes \cdots \otimes V_d$$, the multilinear multiplication is

$$ \begin{align} & \mathcal{B} := (A_1 \otimes A_2 \otimes \cdots \otimes A_d)(\mathcal{A}) \\[4pt] = {} & (A_1 \otimes A_2 \otimes \cdots \otimes A_d)\left(\sum_{i=1}^r \mathbf{a}_i^1 \otimes \mathbf{a}_i^2 \otimes \cdots \otimes \mathbf{a}_i^d\right) \\[5pt] = {} & \sum_{i=1}^r A_1(\mathbf{a}_i^1) \otimes A_2(\mathbf{a}_i^2) \otimes \cdots \otimes A_d( \mathbf{a}_i^d ) \end{align} $$

where $\mathcal{A} = \sum_{i=1}^r \mathbf{a}_i^1 \otimes \mathbf{a}_i^2 \otimes \cdots \otimes \mathbf{a}_i^d$ with $$\mathbf{a}_i^k \in V_k$$ is one of $$\mathcal{A}$$'s tensor rank decompositions. The validity of the above expression is not limited to a tensor rank decomposition; in fact, it is valid for any expression of $$\mathcal{A}$$ as a linear combination of pure tensors, which follows from the universal property of the tensor product.

It is standard to use the following shorthand notations in the literature for multilinear multiplications:$$ (A_1, A_2, \ldots, A_d) \cdot \mathcal{A} := (A_1 \otimes A_2 \otimes \cdots \otimes A_d)(\mathcal{A}) $$and$$ A_k \cdot_k \mathcal{A} := (\operatorname{Id}_{V_1}, \ldots, \operatorname{Id}_{V_{k-1}}, A_k, \operatorname{Id}_{V_{k+1}}, \ldots, \operatorname{Id}_{V_{d}}) \cdot \mathcal{A}, $$where $$\operatorname{Id}_{V_k} : V_k \to V_k $$ is the identity operator.

Definition in coordinates
In computational multilinear algebra it is conventional to work in coordinates. Assume that an inner product is fixed on $$V_k$$ and let $$V_k^*$$ denote the dual vector space of $$V_k $$. Let $$\{ e_1^k, \ldots, e_{n_k}^k \} $$ be a basis for $$V_k $$, let $$\{ (e_1^k)^*, \ldots, (e_{n_k}^k)^* \} $$ be the dual basis, and let $$\{f_1^k, \ldots, f_{m_k}^k \} $$ be a basis for $$W_k $$. The linear map $M_k = \sum_{i=1}^{m_k} \sum_{j=1}^{n_k} m_{i,j}^{(k)} f_i^k \otimes (e_j^k)^* $ is then represented by the matrix $$\widehat{M}_k = [m_{i,j}^{(k)}] \in F^{m_k \times n_k}$$. Likewise, with respect to the standard tensor product basis $$\{ e_{j_1}^1 \otimes e_{j_2}^2 \otimes \cdots \otimes e_{j_d}^d \}_{j_1,j_2,\ldots,j_d} $$, the abstract tensor$$\mathcal{A} = \sum_{j_1=1}^{n_1} \sum_{j_2=1}^{n_2} \cdots \sum_{j_d=1}^{n_d} a_{j_1,j_2,\ldots,j_d} e_{j_1}^1 \otimes e_{j_2}^2 \otimes \cdots \otimes e_{j_d}^d $$is represented by the multidimensional array $$\widehat{\mathcal{A}} = [a_{j_1,j_2,\ldots,j_d}] \in F^{n_1 \times n_2 \times \cdots \times n_d} $$ . Observe that $$\widehat{\mathcal{A}} = \sum_{j_1=1}^{n_1} \sum_{j_2=1}^{n_2} \cdots \sum_{j_d=1}^{n_d} a_{j_1,j_2,\ldots,j_d} \mathbf{e}_{j_1}^1 \otimes \mathbf{e}_{j_2}^2 \otimes \cdots \otimes \mathbf{e}_{j_d}^d, $$

where $$\mathbf{e}_j^k \in F^{n_k} $$ is the jth standard basis vector of $$F^{n_k} $$ and the tensor product of vectors is the affine Segre map $$\otimes : (\mathbf{v}^{(1)}, \mathbf{v}^{(2)}, \ldots, \mathbf{v}^{(d)}) \mapsto [v_{i_1}^{(1)} v_{i_2}^{(2)} \cdots v_{i_d}^{(d)}]_{i_1,i_2,\ldots,i_d} $$. It follows from the above choices of bases that the multilinear multiplication $$\mathcal{B} = (M_1, M_2, \ldots, M_d) \cdot \mathcal{A}$$ becomes

The resulting tensor $$\widehat{\mathcal{B}}$$ lives in $$F^{m_1 \times m_2 \times \cdots \times m_d}$$.

Element-wise definition
From the above expression, an element-wise definition of the multilinear multiplication is obtained. Indeed, since $$\widehat{\mathcal{B}}$$ is a multidimensional array, it may be expressed as $$\widehat{\mathcal{B}} = \sum_{j_1=1}^{n_1} \sum_{j_2=1}^{n_2} \cdots \sum_{j_d=1}^{n_d} b_{j_1,j_2,\ldots,j_d} \mathbf{e}_{j_1}^1 \otimes \mathbf{e}_{j_2}^2 \otimes \cdots \otimes \mathbf{e}_{j_d}^d, $$where $$b_{j_1,j_2,\ldots,j_d} \in F  $$ are the coefficients. Then it follows from the above formulae that

$$ \begin{align} & \left( (\mathbf{e}_{i_1}^1)^T, (\mathbf{e}_{i_2}^2)^T, \ldots, (\mathbf{e}_{i_d}^d)^T \right) \cdot \widehat{\mathcal{B}} \\ = {} & \sum_{j_1=1}^{n_1} \sum_{j_2=1}^{n_2} \cdots \sum_{j_d=1}^{n_d} b_{j_1,j_2,\ldots,j_d} \left( (\mathbf{e}_{i_1}^1)^T \mathbf{e}_{j_1}^1 \right) \otimes \left((\mathbf{e}_{i_2}^2)^T \mathbf{e}_{j_2}^2\right) \otimes \cdots \otimes \left( (\mathbf{e}_{i_d}^d)^T \mathbf{e}_{j_d}^d \right) \\ = {} & \sum_{j_1=1}^{n_1} \sum_{j_2=1}^{n_2} \cdots \sum_{j_d=1}^{n_d} b_{j_1,j_2,\ldots,j_d} \delta_{i_1, j_1} \cdot \delta_{i_2,j_2} \cdots \delta_{i_d, j_d} \\ = {} & b_{i_1,i_2,\ldots,i_d}, \end{align} $$

where $$\delta_{i,j} $$ is the Kronecker delta. Hence, if $$\mathcal{B} = (M_1, M_2, \ldots, M_d) \cdot \mathcal{A} $$, then

where the $$m_{i,j}^{(k)} $$ are the elements of $$\widehat{M}_k $$ as defined above.

Properties
Let $$\mathcal{A} \in V_1 \otimes V_2 \otimes \cdots \otimes V_d $$ be an order-d tensor over the tensor product of $$F $$-vector spaces.

Since a multilinear multiplication is the tensor product of linear maps, we have the following multilinearity property (in the construction of the map):

$$A_1 \otimes \cdots \otimes A_{k-1} \otimes (\alpha A_k + \beta B) \otimes A_{k+1} \otimes \cdots \otimes A_d = \alpha A_1 \otimes \cdots \otimes A_d + \beta A_1 \otimes \cdots \otimes A_{k-1} \otimes B \otimes A_{k+1} \otimes \cdots \otimes A_d $$

Multilinear multiplication is a linear map: $$(M_1, M_2, \ldots, M_d) \cdot (\alpha \mathcal{A} + \beta \mathcal{B}) = \alpha \; (M_1, M_2, \ldots, M_d) \cdot \mathcal{A} + \beta \; (M_1, M_2, \ldots, M_d) \cdot \mathcal{B} $$

It follows from the definition that the composition of two multilinear multiplications is also a multilinear multiplication:

$$(M_1, M_2, \ldots, M_d) \cdot \left( (K_1, K_2, \ldots, K_d) \cdot \mathcal{A} \right) = (M_1 \circ K_1, M_2 \circ K_2, \ldots, M_d \circ K_d) \cdot \mathcal{A}, $$

where $$M_k : U_k \to W_k$$ and $$K_k : V_k \to U_k$$ are linear maps.

Observe specifically that multilinear multiplications in different factors commute,

$$M_k \cdot_k \left( M_\ell \cdot_\ell \mathcal{A} \right) = M_\ell \cdot_\ell \left( M_k \cdot_k \mathcal{A} \right) = M_k \cdot_k M_\ell \cdot_\ell \mathcal{A}, $$

if $$k \ne \ell.$$

Computation
The factor-k multilinear multiplication $$M_k \cdot_k\mathcal{A}$$ can be computed in coordinates as follows. Observe first that

$$ \begin{align} M_k \cdot_k \mathcal{A} &= M_k \cdot_k \sum_{j_1=1}^{n_1} \sum_{j_2=1}^{n_2} \cdots \sum_{j_d=1}^{n_d} a_{j_1,j_2,\ldots,j_d} \mathbf{e}_{j_1}^1 \otimes \mathbf{e}_{j_2}^2 \otimes \cdots \otimes \mathbf{e}_{j_d}^d \\ &= \sum_{j_1=1}^{n_1} \cdots \sum_{j_{k-1}=1}^{n_{k-1}} \sum_{j_{k+1}=1}^{n_{k+1}} \cdots \sum_{j_d=1}^{n_d} \mathbf{e}_{j_1}^1 \otimes \cdots \otimes \mathbf{e}_{j_{k-1}}^{k-1} \otimes M_k \left ( \sum_{j_k=1}^{n_k} a_{j_1,j_2,\ldots,j_d} \mathbf{e}_{j_k}^k \right) \otimes \mathbf{e}_{j_{k+1}}^{k+1} \otimes \cdots \otimes \mathbf{e}_{j_d}^d. \end{align} $$

Next, since

$$F^{n_1} \otimes F^{n_2} \otimes \cdots \otimes F^{n_d} \simeq F^{n_k} \otimes (F^{n_1} \otimes \cdots \otimes F^{n_{k-1}} \otimes F^{n_{k+1}} \otimes \cdots \otimes F^{n_d}) \simeq F^{n_k} \otimes F^{n_1 \cdots n_{k-1} n_{k+1} \cdots n_d}, $$

there is a bijective map, called the factor-k standard flattening, denoted by $$(\cdot)_{(k)} $$, that identifies $$M_k \cdot_k \mathcal{A}$$ with an element from the latter space, namely

$$\left( M_k \cdot_k \mathcal{A} \right)_{(k)} := \sum_{j_1=1}^{n_1} \cdots \sum_{j_{k-1}=1}^{n_{k-1}} \sum_{j_{k+1}=1}^{n_{k+1}} \cdots \sum_{j_d=1}^{n_d} M_k \left ( \sum_{j_k=1}^{n_k} a_{j_1,j_2,\ldots,j_d} \mathbf{e}_{j_{k}}^{k} \right) \otimes \mathbf{e}_{\mu_k(j_1,\ldots,j_{k-1},j_{k+1},\ldots,j_d)} := M_k \mathcal{A}_{(k)}, $$

where $$\mathbf{e}_j $$is the jth standard basis vector of $$F^{N_k}$$, $$N_k = n_1 \cdots n_{k-1} n_{k+1} \cdots n_d $$, and $$\mathcal{A}_{(k)} \in F^{n_k} \otimes F^{N_k} \simeq F^{n_k \times N_k} $$ is the factor-k flattening matrix of $$\mathcal{A}$$ whose columns are the factor-k vectors $$[a_{j_1,\ldots,j_{k-1},i,j_{k+1},\ldots,j_d}]_{i=1}^{n_k} $$ in some order, determined by the particular choice of the bijective map

$$\mu_k : [1,n_1] \times \cdots \times [1,n_{k-1}] \times [1,n_{k+1}] \times \cdots \times [1,n_d] \to [1,N_k]. $$

In other words, the multilinear multiplication $$(M_1, M_2, \ldots, M_d) \cdot \mathcal{A}$$ can be computed as a sequence of d factor-k multilinear multiplications, which themselves can be implemented efficiently as classic matrix multiplications.

Applications
The higher-order singular value decomposition (HOSVD) factorizes a tensor given in coordinates $$\mathcal{A} \in F^{n_1 \times n_2 \times \cdots \times n_d} $$ as the multilinear multiplication $$\mathcal{A} = (U_1, U_2, \ldots, U_d) \cdot \mathcal{S} $$, where $$U_k \in F^{n_k \times n_k} $$ are orthogonal matrices and $$\mathcal{S} \in F^{n_1 \times n_2 \times \cdots \times n_d} $$.