Symmetric, Positive Definite Matrix

Relation to Inner Product

Any general inner product can be represented as a matrix multiplication.

The matrix that uniquely defines the inner product is called the symmetric, positive definite matrix.

Let $V$ be an $n$-dimensional inner product space with inner product $\langle\cdot,\cdot\rangle: V \times V \rightarrow \mathbb{R}$.

Let $B = (\mathbf{b}_1, \dots, \mathbf{b}_n)$ be an ordered basis of $V$.

Any vector $\mathbf{x}, \mathbf{y} \in V$ can be represented as a linear combination of the basis vectors:

\[\mathbf{x} = \sum_{i=1}^n \lambda_i \mathbf{b}_i \quad\quad\quad \mathbf{y} = \sum_{j=1}^n \psi_j \mathbf{b}_j\]

for some $\lambda_i, \psi_j \in \mathbb{R}$.

Also, let $\mathbf{\hat{x}} = [\lambda_1 \dots \lambda_n]^T$ and $\mathbf{\hat{y}} = [\psi_1 \dots \psi_n]^T$, which are the coordinate vectors of $\mathbf{x}$ and $\mathbf{y}$ with respect to the ordered basis $B$.

Often also denoted $[\mathbf{x}]_B$ and $[\mathbf{y}]_B$. See coordinate vectors with respect to orderded bases.

The inner product $\langle\mathbf{x},\mathbf{y}\rangle$ have three properties:

First using bilinearity:

\[\langle\mathbf{x},\mathbf{y}\rangle = \left\langle\sum_{i=1}^n \lambda_i \mathbf{b}_i, \sum_{j=1}^n \psi_j \mathbf{b}_j\right\rangle = \sum_{i=1}^n \sum_{j=1}^n \lambda_i \langle\mathbf{b}_i, \mathbf{b}_j\rangle \psi_j = \mathbf{\hat{x}}^T \mathbf{A} \mathbf{\hat{y}}\]

$\mathbf{\hat{x}}, \mathbf{\hat{y}}$ are just coordinates, so $\mathbf{A}$ is the matrix that uniquely defines the inner product operation.

Because an inner product is symmetric, $\mathbf{A}$ is also a symmetric matrix.

Now using positive definiteness:

\[\begin{equation} \label{eq:posdef} \tag{Positive Definite} \forall \mathbf{x} \in V\setminus\{\mathbf{0}\},\, \mathbf{x}^T \mathbf{A} \mathbf{x} > 0 \end{equation}\]

Such $\mathbf{A}$ is called the symmetric, positive definite matrix.

Hence, $\langle\mathbf{x},\mathbf{y}\rangle$ is a valid inner product if and only if there exists a symmetric, positive definite matrix $\mathbf{A} \in \mathbb{R}^{n \times n}$ such that

$$ \langle\mathbf{x},\mathbf{y}\rangle = \mathbf{\hat{x}}^T \mathbf{A} \mathbf{\hat{y}} $$

Again, remember that $\hat{\mathbf{x}}$ and $\hat{\mathbf{y}}$ are coordinate vectors of each respect to the ordered basis $B$.

Symmetric, Positive Semi-Definite Matrix

In equation \eqref{eq:posdef}, if the inequality is loosened to $\geq$, then $\mathbf{A}$ is called the symmetric, positive semi-definite matrix.

Nullspace of Symmetric, Positive Definite Matrix

By definition, we know that $\mathbf{x}^T \mathbf{A} \mathbf{x} > 0$ for all $\mathbf{x} \neq \mathbf{0}$. So trivially, $\mathbf{A} \mathbf{x} = \mathbf{0}$ only when $\mathbf{x} = \mathbf{0}$.

Therefore, the kernel of $\mathbf{A}$ only contains the zero vector.

In other words, the columns of $\mathbf{A}$ are linearly independent. Also, $\mathbf{A}$ is invertible.

Diagonal Elements of Symmetric, Positive Definite Matrix

Diagonal elements of $\mathbf{A}$ are positive, because $a_{ii} = \mathbf{e}_i^T \mathbf{A} \mathbf{e}_i > 0$.