User:Cs32en/Sandbox/matrixcalculus

Definitions
Let

$$ \mathbf X \isin \mathbb R^{m \times n} $$

$$ \mathbf Y \isin \mathbb R^{p \times q} $$,

where m and p are contravariant (vertical) indices, n and q are covariant (horizontal) indices.

Then, we define the "tiled" matrix derivative and the "vectorial" matrix derivative as follows:

Tiled matrix derivative:

$$ \mathrm D_{\mathrm{tiled}} := \begin{pmatrix} \frac{\partial\,\mathbf Y^1_1}{\partial\,\mathbf X^1_1} & \frac{\partial\,\mathbf Y^1_2}{\partial\,\mathbf X^1_1} & \cdots & \frac{\partial\,\mathbf Y^1_q}{\partial\,\mathbf X^1_1} & \cdots & \frac{\partial\,\mathbf Y^1_q}{\partial\,\mathbf X^m_1} \\ \frac{\partial\,\mathbf Y^2_1}{\partial\,\mathbf X^1_1} & \frac{\partial\,\mathbf Y^2_2}{\partial\,\mathbf X^1_1} & \cdots & \frac{\partial\,\mathbf Y^2_q}{\partial\,\mathbf X^1_1} & \cdots & \frac{\partial\,\mathbf Y^2_q}{\partial\,\mathbf X^m_1} \\ \vdots & \vdots & \ddots & \vdots & \ddots & \vdots \\ \frac{\partial\,\mathbf Y^p_1}{\partial\,\mathbf X^1_1} & \frac{\partial\,\mathbf Y^p_2}{\partial\,\mathbf X^1_1} & \cdots & \frac{\partial\,\mathbf Y^p_q}{\partial\,\mathbf X^1_1} & \cdots & \frac{\partial\,\mathbf Y^p_q}{\partial\,\mathbf X^m_1} \\ \vdots & \vdots & \ddots & \vdots & \ddots & \vdots \\ \frac{\partial\,\mathbf Y^p_1}{\partial\,\mathbf X^1_n} & \frac{\partial\,\mathbf Y^p_2}{\partial\,\mathbf X^1_n} & \cdots & \frac{\partial\,\mathbf Y^p_q}{\partial\,\mathbf X^1_n} & \cdots & \frac{\partial\,\mathbf Y^p_q}{\partial\,\mathbf X^m_n} \end{pmatrix} $$

$$ \frac{\partial\,\mathbf Y^p_q}{\partial\,\mathbf X^m_n} = \mathrm D_{\mathrm{tiled}} $$

$$ \mathrm D_{\mathrm{tiled}} \isin \mathbb R^{np \times mq} $$

Vectorial matrix derivative:

$$ \mathrm D_{\mathrm{vec}} := \begin{pmatrix} \frac{\partial\,\mathbf Y^1_1}{\partial\,\mathbf X^1_1} & \frac{\partial\,\mathbf Y^1_1}{\partial\,\mathbf X^2_1} & \cdots & \frac{\partial\,\mathbf Y^1_1}{\partial\,\mathbf X^m_1} & \cdots & \frac{\partial\,\mathbf Y^1_1}{\partial\,\mathbf X^m_n} \\ \frac{\partial\,\mathbf Y^2_1}{\partial\,\mathbf X^1_1} & \frac{\partial\,\mathbf Y^2_1}{\partial\,\mathbf X^2_1} & \cdots & \frac{\partial\,\mathbf Y^2_1}{\partial\,\mathbf X^m_1} & \cdots & \frac{\partial\,\mathbf Y^2_1}{\partial\,\mathbf X^m_n} \\ \vdots & \vdots & \ddots & \vdots & \ddots & \vdots \\ \frac{\partial\,\mathbf Y^p_1}{\partial\,\mathbf X^1_1} & \frac{\partial\,\mathbf Y^p_1}{\partial\,\mathbf X^2_1} & \cdots & \frac{\partial\,\mathbf Y^p_1}{\partial\,\mathbf X^m_1} & \cdots & \frac{\partial\,\mathbf Y^p_1}{\partial\,\mathbf X^m_n} \\ \vdots & \vdots & \ddots & \vdots & \ddots & \vdots \\ \frac{\partial\,\mathbf Y^p_q}{\partial\,\mathbf X^1_1} & \frac{\partial\,\mathbf Y^p_q}{\partial\,\mathbf X^2_1} & \cdots & \frac{\partial\,\mathbf Y^p_q}{\partial\,\mathbf X^m_1} & \cdots & \frac{\partial\,\mathbf Y^p_q}{\partial\,\mathbf X^m_n} \end{pmatrix} $$

$$ \mathrm D\,\mathbf Y^p_q\left(\mathbf X^m_n\right) = \frac{\partial\,\operatorname{vec}\,\mathbf Y^p_q}{\partial\,\left(\operatorname{vec}\,\mathbf X^m_n\right)^{\mathrm T}} = \mathrm D_{\mathrm{vec}} $$

$$ \mathrm D_{\mathrm{vec}} \isin \mathbb R^{pq \times mn} $$