Numerator / Denominator Layout

There are mainly two types of layout for matrix differentiation:

• Numerator Layout
• Denominator Layout

Different texts use different layouts. They are both valid and most of the times the results are the transpose of another.

Table of contents

1. Numerator Layout
2. Denominator Layout

---

Numerator Layout

In the numerator layout, the resulting matrix respects the dimensions of the numerator.

Let $\boldsymbol{x} \in \mathbb{R}^3$ be a vector.

Let $f: \mathbb{R}^3 \rightarrow \mathbb{R}$ be a scalar function.

The transformation matrix of $f$ would be a $1 \times 3$ row vector.

The gradient of $f$ in the numerator layout is:

\[\nabla_\boldsymbol{x} f = \frac{d f}{d \boldsymbol{x}} = \begin{bmatrix} \frac{\partial f}{\partial x_1} & \frac{\partial f}{\partial x_2} & \frac{\partial f}{\partial x_3} \end{bmatrix}_{1 \times 3}\]

If $\boldsymbol{f}: \mathbb{R}^3 \rightarrow \mathbb{R}^2$ is a vector field:

\[\boldsymbol{f}(\boldsymbol{x}) = \begin{bmatrix} f_1(\boldsymbol{x}) \\ f_2(\boldsymbol{x}) \end{bmatrix}\]

the Jacobian matrix in the numerator layout is:

\[\frac{d \boldsymbol{f}}{d \boldsymbol{x}} = \begin{bmatrix} \frac{\partial f_1}{\partial x_1} & \frac{\partial f_1}{\partial x_2} & \frac{\partial f_1}{\partial x_3} \\ \frac{\partial f_2}{\partial x_1} & \frac{\partial f_2}{\partial x_2} & \frac{\partial f_2}{\partial x_3} \end{bmatrix}\]

Notice how the layout of the numerator is preserved in the resulting matrix.

Intuitive Calculus Notation

With numerator layout,

\[\frac{\partial}{\partial \boldsymbol{x}} \boldsymbol{Ax} = \boldsymbol{A}\]

In terms of symbolic notations, numerator layout is more intuitive and aligned with regular scalar calculus.

---

Denominator Layout

In the denominator layout, the resulting matrix respects the dimensions of the denominator.

Let $\boldsymbol{x} \in \mathbb{R}^3$ be a (column) vector:

\[\boldsymbol{x} = \begin{bmatrix} x_1 \\ x_2 \\ x_3 \end{bmatrix}\]

Let $f: \mathbb{R}^3 \rightarrow \mathbb{R}$ be a scalar function.

The gradient of $f$ in the denominator layout is:

\[\nabla_\boldsymbol{x} f = \frac{d f}{d \boldsymbol{x}} = \begin{bmatrix} \frac{\partial f}{\partial x_1} \\ \frac{\partial f}{\partial x_2} \\ \frac{\partial f}{\partial x_3} \end{bmatrix}_{3 \times 1}\]

If $\boldsymbol{f}: \mathbb{R}^3 \rightarrow \mathbb{R}^2$ is a vector field:

\[\boldsymbol{f}(\boldsymbol{x}) = \begin{bmatrix} f_1(\boldsymbol{x}) \\ f_2(\boldsymbol{x}) \end{bmatrix}\]

the Jacobian matrix in the denominator layout is:

\[\frac{d \boldsymbol{f}}{d \boldsymbol{x}} = \begin{bmatrix} \frac{\partial f_1}{\partial x_1} & \frac{\partial f_2}{\partial x_1} \\ \frac{\partial f_1}{\partial x_2} & \frac{\partial f_2}{\partial x_2} \\ \frac{\partial f_1}{\partial x_3} & \frac{\partial f_2}{\partial x_3} \end{bmatrix}\]

Notice how the layout of the denominator is preserved in the resulting matrix.

Transpose of Numerator Layout

With denominator layout,

\[\frac{\partial}{\partial \boldsymbol{x}} \boldsymbol{Ax} = \boldsymbol{A}^T\]

Which is the transpose of the numerator layout.