Link Search Menu Expand Document
Itty Bitty

Norms and Inner Products

Table of contents
  1. Inner Product
    1. Dot Product
    2. General Inner Product
    3. Inner Product Space
    4. Existence of Symmetric, Positive Definite Matrix
  2. Norm
    1. Manhattan Norm
    2. Euclidean Norm
  3. Inner Product Induced Norm
    1. Cauchy-Schwarz Inequality
    2. Commonly Used Inner Product Induced Norms
      1. Length of a Vector
      2. Distance Between Two Vectors

Inner Product

Dot Product

$$ \mathbf{x} \cdot \mathbf{y} = \mathbf{x}^T \mathbf{y} = \sum_{i=1}^n x_i y_i $$

This special type of inner product is also called the scalar product.

General Inner Product

Review the concepts of bi-linear mapping, symmetric, and positive definite.

Let $\Omega: V \times V \rightarrow \mathbb{R}$ be a bi-linear mapping, or just use a binary operator $\langle \cdot, \cdot \rangle$

$\langle \cdot, \cdot \rangle$ is an inner product if $\Omega$ is symmetric and positive definite.

Inner Product Space

An inner product space is the tuple

$$ (V, \langle \cdot, \cdot \rangle) $$

where $V$ is a vector space and $\langle \cdot, \cdot \rangle$ is a general inner product.

When $\langle \cdot, \cdot \rangle$, is a regular dot product, this space is called the Euclidean vector space.

Existence of Symmetric, Positive Definite Matrix

$\langle \cdot, \cdot \rangle$ is an inner product if and only if there is a symmetric, positive definite matrix $A$ such that

$$ \langle \mathbf{x}, \mathbf{y} \rangle = \mathbf{x}^T A \mathbf{y} $$

See details here.

Norm

A norm on a vector space $V$ is a function that assigns each vector $\mathbf{x} \in V$ its length $\lVert\mathbf{x}\rVert \in \mathbb{R}$:

$$ \|\cdot\|: V \rightarrow \mathbb{R},\, \mathbf{x} \mapsto \|\mathbf{x}\| $$

For a function to be a norm, it must satisfy the following properties $\forall \mathbf{x}, \mathbf{y} \in V$ and $\forall \lambda \in \mathbb{R}$:

  • Absolutely homogeneous: $\lVert\lambda \mathbf{x}\rVert = |\lambda| \lVert\mathbf{x}\rVert$
  • Triangle inequality: $\lVert\mathbf{x} + \mathbf{y}\rVert \leq \lVert\mathbf{x}\rVert + \lVert\mathbf{y}\rVert$

  • Positive definite: $\lVert\mathbf{x}\rVert \geq 0$ and $\lVert\mathbf{x}\rVert = 0 \iff \mathbf{x} = \mathbf{0}$

    It is always non-negative and zero only when the vector is the zero vector.

Manhattan Norm

Also known as the $\boldsymbol{L_1}$ norm.

$\forall \mathbf{x} \in \mathbb{R}^n$:

$$ \|\mathbf{x}\|_1 = \sum_{i=1}^n |x_i| $$

Euclidean Norm

Also known as the $\boldsymbol{L_2}$ norm.

$$ \|\mathbf{x}\|_2 = \sqrt{\sum_{i=1}^n x_i^2} = \sqrt{\mathbf{x}^T \mathbf{x}} $$

Inner Product Induced Norm

Some norms can be induced by inner products, meaning:

$$ \|\mathbf{x}\| = \sqrt{\langle\mathbf{x},\mathbf{x}\rangle} $$

Not every norm is induced by an inner product. Manhattan norm is not induced by any inner product, but Euclidean norm is.

Cauchy-Schwarz Inequality

For any inner product space $(V, \langle \cdot, \cdot \rangle)$, induced norm satisfies the following inequality:

$$ |\langle\mathbf{x},\mathbf{y}\rangle| \leq \|\mathbf{x}\| \|\mathbf{y}\| $$

Commonly Used Inner Product Induced Norms

Length of a Vector

Euclidean norm gives us the length of a vector. It is induced by the dot product.

$$ \|\mathbf{x}\|_2 = \sqrt{\langle\mathbf{x},\mathbf{x}\rangle} $$

Distance Between Two Vectors

Distance between two vectors (Euclidean distance) is also induced by the dot product.

$$ d(\mathbf{x}, \mathbf{y}) = \|\mathbf{x} - \mathbf{y}\|_2 = \sqrt{\langle\mathbf{x} - \mathbf{y}, \mathbf{x} - \mathbf{y}\rangle} $$

Mapping $d: V \times V \rightarrow \mathbb{R}$, $(\mathbf{x}, \mathbf{y}) \mapsto d(\mathbf{x}, \mathbf{y})$ is called a metric.