Khatri–Rao product

In mathematics, the Khatri–Rao product or block Kronecker product of two partitioned matrices $$\mathbf{A}$$ and $$\mathbf{B}$$ is defined as
 * $$ \mathbf{A} \ast \mathbf{B} = \left(\mathbf{A}_{ij} \otimes \mathbf{B}_{ij}\right)_{ij}$$

in which the ij-th block is the mipi × njqj sized Kronecker product of the corresponding blocks of A and B, assuming the number of row and column partitions of both matrices is equal. The size of the product is then (Σi mipi) × (Σj njqj).

For example, if A and B both are 2 × 2 partitioned matrices e.g.:
 * $$ \mathbf{A} =

\left[ \begin{array} {c | c} \mathbf{A}_{11} & \mathbf{A}_{12} \\ \hline \mathbf{A}_{21} & \mathbf{A}_{22} \end{array} \right] = \left[ \begin{array} {c c | c} 1 & 2 & 3 \\ 4 & 5 & 6 \\ \hline 7 & 8 & 9 \end{array} \right] ,\quad \mathbf{B} = \left[ \begin{array} {c | c} \mathbf{B}_{11} & \mathbf{B}_{12} \\ \hline \mathbf{B}_{21} & \mathbf{B}_{22} \end{array} \right] = \left[ \begin{array} {c | c c} 1 & 4 & 7 \\ \hline 2 & 5 & 8 \\ 3 & 6 & 9 \end{array} \right] , $$

we obtain:

\mathbf{A} \ast \mathbf{B} = \left[ \begin{array} {c | c} \mathbf{A}_{11} \otimes \mathbf{B}_{11} & \mathbf{A}_{12} \otimes \mathbf{B}_{12} \\ \hline \mathbf{A}_{21} \otimes \mathbf{B}_{21} & \mathbf{A}_{22} \otimes \mathbf{B}_{22} \end{array} \right] = \left[ \begin{array} {c c | c c} 1 & 2 & 12 & 21 \\ 4 & 5 & 24 & 42 \\ \hline 14 & 16 & 45 & 72 \\ 21 & 24 & 54 & 81 \end{array} \right]. $$

This is a submatrix of the Tracy–Singh product of the two matrices (each partition in this example is a partition in a corner of the Tracy–Singh product).

Column-wise Kronecker product
The column-wise Kronecker product of two matrices is a special case of the Khatri-Rao product as defined above, and may also be called the Khatri–Rao product. This product assumes the partitions of the matrices are their columns. In this case m1 = m, p1 = p, n = q and for each j: nj = pj = 1. The resulting product is a mp × n matrix of which each column is the Kronecker product of the corresponding columns of A and B. Using the matrices from the previous examples with the columns partitioned:

\mathbf{C} = \left[ \begin{array} { c | c | c} \mathbf{C}_1 & \mathbf{C}_2 & \mathbf{C}_3 \end{array} \right] = \left[ \begin{array} {c | c | c} 1 & 2 & 3 \\ 4 & 5 & 6 \\ 7 & 8 & 9 \end{array} \right] ,\quad \mathbf{D} = \left[ \begin{array} { c | c | c } \mathbf{D}_1 & \mathbf{D}_2 & \mathbf{D}_3 \end{array} \right] = \left[ \begin{array} { c | c | c } 1 & 4 & 7 \\ 2 & 5 & 8 \\ 3 & 6 & 9 \end{array} \right] , $$

so that:

\mathbf{C} \ast \mathbf{D} = \left[ \begin{array} { c | c | c } \mathbf{C}_1 \otimes \mathbf{D}_1 & \mathbf{C}_2 \otimes \mathbf{D}_2 & \mathbf{C}_3 \otimes \mathbf{D}_3 \end{array} \right] = \left[ \begin{array} { c | c | c } 1 & 8 & 21 \\ 2 & 10 & 24 \\ 3 & 12 & 27 \\ 4 & 20 & 42 \\ 8 & 25 & 48 \\ 12 & 30 & 54 \\ 7 & 32 & 63 \\ 14 & 40 & 72 \\ 21 & 48 & 81 \end{array} \right]. $$

This column-wise version of the Khatri–Rao product is useful in linear algebra approaches to data analytical processing and in optimizing the solution of inverse problems dealing with a diagonal matrix.

In 1996 the column-wise Khatri–Rao product was proposed to estimate the angles of arrival (AOAs) and delays of multipath signals and four coordinates of signals sources at a digital antenna array.

Face-splitting product
An alternative concept of the matrix product, which uses row-wise splitting of matrices with a given quantity of rows, was proposed by V. Slyusar in 1996. This matrix operation was named the "face-splitting product" of matrices or the "transposed Khatri–Rao product". This type of operation is based on row-by-row Kronecker products of two matrices. Using the matrices from the previous examples with the rows partitioned:

\mathbf{C} = \begin{bmatrix} \mathbf{C}_1 \\\hline \mathbf{C}_2 \\\hline \mathbf{C}_3\\ \end{bmatrix} = \begin{bmatrix} 1 & 2 & 3 \\\hline 4 & 5 & 6 \\\hline 7 & 8 & 9 \end{bmatrix} ,\quad \mathbf{D} = \begin{bmatrix} \mathbf{D}_1\\\hline \mathbf{D}_2\\\hline \mathbf{D}_3\\ \end{bmatrix} = \begin{bmatrix} 1 & 4 & 7 \\\hline 2 & 5 & 8 \\\hline 3 & 6 & 9 \end{bmatrix} , $$

the result can be obtained:

\mathbf{C} \bull \mathbf{D} = \begin{bmatrix} \mathbf{C}_1 \otimes \mathbf{D}_1\\\hline \mathbf{C}_2 \otimes \mathbf{D}_2\\\hline \mathbf{C}_3 \otimes \mathbf{D}_3\\ \end{bmatrix} = \begin{bmatrix} 1 & 4 & 7 & 2 & 8 & 14 & 3 & 12 & 21 \\\hline 8 & 20 & 32 & 10 & 25 & 40 & 12 & 30 & 48 \\\hline 21 & 42 & 63 & 24 & 48 & 72 & 27 & 54 & 81 \end{bmatrix}. $$

Examples

 * $$\begin{align}

&\left(     \begin{bmatrix}         1 & 0 \\         0 & 1 \\         1 & 0      \end{bmatrix}      \bullet      \begin{bmatrix}         1 & 0 \\         1 & 0 \\         0 & 1      \end{bmatrix}      \right) \left(     \begin{bmatrix}         1 & 1 \\         1 & -1      \end{bmatrix}      \otimes      \begin{bmatrix}         1 & 1 \\         1 & -1      \end{bmatrix}      \right) \left(     \begin{bmatrix}         \sigma_1 & 0 \\         0 & \sigma_2 \\      \end{bmatrix}      \otimes      \begin{bmatrix}         \rho_1 & 0 \\         0 & \rho_2 \\      \end{bmatrix}      \right) \left(     \begin{bmatrix}         x_1 \\         x_2      \end{bmatrix}      \ast      \begin{bmatrix}         y_1 \\         y_2      \end{bmatrix}      \right) \\[5pt] {}={} &\left(     \begin{bmatrix}         1 & 0 \\         0 & 1 \\         1 & 0      \end{bmatrix}      \bullet      \begin{bmatrix}         1 & 0 \\         1 & 0 \\         0 & 1      \end{bmatrix}      \right) \left(     \begin{bmatrix}         1 & 1 \\         1 & -1      \end{bmatrix}      \begin{bmatrix}         \sigma_1 & 0 \\         0 & \sigma_2 \\      \end{bmatrix}      \begin{bmatrix}         x_1 \\         x_2      \end{bmatrix}      \,\otimes\,      \begin{bmatrix}         1 & 1 \\         1 & -1      \end{bmatrix}      \begin{bmatrix}         \rho_1 & 0 \\         0 & \rho_2 \\      \end{bmatrix}      \begin{bmatrix}         y_1 \\         y_2      \end{bmatrix}      \right) \\[5pt] {}={} &     \begin{bmatrix} 1 & 0 \\        0 & 1 \\         1 & 0      \end{bmatrix} \begin{bmatrix} 1 & 1 \\        1 & -1      \end{bmatrix} \begin{bmatrix} \sigma_1 & 0 \\ 0 & \sigma_2 \\ \end{bmatrix} \begin{bmatrix} x_1 \\ x_2 \end{bmatrix} \,\circ\, \begin{bmatrix} 1 & 0 \\        1 & 0 \\         0 & 1      \end{bmatrix} \begin{bmatrix} 1 & 1 \\        1 & -1      \end{bmatrix} \begin{bmatrix} \rho_1 & 0 \\ 0 & \rho_2 \\ \end{bmatrix} \begin{bmatrix} y_1 \\ y_2 \end{bmatrix} . \end{align}$$

Theorem
If $$M = T^{(1)} \bullet \dots \bullet T^{(c)}$$, where $$T^{(1)}, \dots, T^{(c)}$$ are independent components a random matrix $$T$$ with independent identically distributed rows $$T_1, \dots, T_m\in \mathbb R^d$$, such that
 * $$E\left[(T_1x)^2\right] = \left\|x\right\|_2^2$$ and $$E\left[(T_1 x)^p\right]^\frac{1}{p} \le \sqrt{ap}\|x\|_2$$,

then for any vector $$x$$
 * $$\left| \left\|Mx\right\|_2 - \left\|x\right\|_2 \right| < \varepsilon \left\|x\right\|_2$$

with probability $$1 - \delta$$ if the quantity of rows
 * $$m = (4a)^{2c} \varepsilon^{-2} \log 1/\delta + (2ae)\varepsilon^{-1}(\log 1/\delta)^c.$$

In particular, if the entries of $$T$$ are $$\pm 1$$ can get
 * $$m = O\left(\varepsilon^{-2}\log1/\delta + \varepsilon^{-1}\left(\frac{1}{c}\log1/\delta\right)^c\right)$$

which matches the Johnson–Lindenstrauss lemma of $$m = O\left(\varepsilon^{-2}\log1/\delta\right)$$ when $$\varepsilon$$ is small.

Block face-splitting product
According to the definition of V. Slyusar the block face-splitting product of two partitioned matrices with a given quantity of rows in blocks


 * $$ \mathbf{A} =

\left[ \begin{array} {c | c} \mathbf{A}_{11} & \mathbf{A}_{12} \\ \hline \mathbf{A}_{21} & \mathbf{A}_{22} \end{array} \right]

,\quad \mathbf{B} = \left[ \begin{array} {c | c} \mathbf{B}_{11} & \mathbf{B}_{12} \\ \hline \mathbf{B}_{21} & \mathbf{B}_{22} \end{array} \right] , $$

can be written as :



\mathbf{A} [\bull] \mathbf{B} = \left[ \begin{array} {c | c} \mathbf{A}_{11} \bull \mathbf{B}_{11} & \mathbf{A}_{12} \bull \mathbf{B}_{12} \\ \hline \mathbf{A}_{21} \bull \mathbf{B}_{21} & \mathbf{A}_{22} \bull \mathbf{B}_{22} \end{array} \right]. $$

The transposed block face-splitting product (or Block column-wise version of the Khatri–Rao product) of two partitioned matrices with a given quantity of columns in blocks has a view:



\mathbf{A} [\ast] \mathbf{B} = \left[ \begin{array} {c | c} \mathbf{A}_{11} \ast \mathbf{B}_{11} & \mathbf{A}_{12} \ast \mathbf{B}_{12} \\ \hline \mathbf{A}_{21} \ast \mathbf{B}_{21} & \mathbf{A}_{22} \ast \mathbf{B}_{22} \end{array} \right]. $$

Main properties

 * 1) Transpose:
 * $$\left(\mathbf{A} [\ast] \mathbf{B} \right)^\textsf{T} = \textbf{A}^\textsf{T} [\bull] \mathbf{B}^\textsf{T}$$

Applications
The Face-splitting product and the Block Face-splitting product used in the tensor-matrix theory of digital antenna arrays. These operations are also used in:
 * Artificial Intelligence and Machine learning systems to minimization of convolution and tensor sketch operations,
 * A popular Natural Language Processing models, and hypergraph models of similarity,
 * Generalized linear array model in statistics
 * Two- and multidimensional P-spline approximation of data,
 * Studies of genotype x environment interactions.