Mutivariate Distributions

Table of contents

Multinomial Distribution

Extension of the binomial distribution to more than two categories.

Instead of have success/failure, we have $\boldsymbol{k}$ categories (e.g. dice roll, preference surveys).

We have a random vector $X = (X_1, \dots, X_k)$, where $X_i$ is the number of times category $i$ occurs.

For a given $n$ and a probability vector $p = (p_1, \dots, p_k)$,

\[n = \sum_{i=1}^k X_i\]

Then the probability mass function is:

$$ p_X(x) = \binom{n}{x_1 \dots x_k} p_1^{x_1} \dots p_k^{x_k} = \frac{n!}{x_1! \dots x_k!} p_1^{x_1} \dots p_k^{x_k} $$

And we denote:

$$ X \sim \text{Multinomial}(n, p) $$

When $X \sim \text{Multinomial}(n, p)$, the marginal distribution of $X_i$ is:

$$ X_i \sim \text{Binomial}(n, p_i) $$

The expected value of $X \sim \text{Multinomial}(n, p)$ is:

$$ n \cdot p = \begin{pmatrix} n \cdot p_1 \\ \vdots \\ n \cdot p_k \end{pmatrix} $$

The variance of $X \sim \text{Multinomial}(n, p)$ is the covariance matrix.

We have a random vector $X = (X_1, \dots, X_k)$.

The parameters are:

$\mu = (\mu_1, \dots, \mu_k)$: mean vector
$\Sigma$: $k \times k$ symmetric positive definite covariance matrix
\[\Sigma_{ij} = \text{Cov}(X_i, X_j) = \E[(X_i - \mu_i)(X_j - \mu_j)]\]

The probability density function is:

$$ f_X(x) = \frac{1}{\sqrt{(2\pi)^k \det(\Sigma)}} \exp\left( -\frac{1}{2} (x - \mu)^T \Sigma^{-1} (x - \mu) \right) $$

And we denote:

$$ X \sim \mathcal{N}(\mu, \Sigma) $$

It is similar to the univariate case, which looked like

\[Z = \frac{X - \mu}{\sigma} \iff X = \mu + \sigma Z\]

Let’s just take for granted that $\Sigma^{1/2}$ and $\Sigma^{-1/2}$ exist.

For standard multivariate normal random vector $Z \sim \mathcal{N}(0, I)$,

$$ X = \mu + \Sigma^{1/2} Z \implies X \sim \mathcal{N}(\mu, \Sigma) $$

And for multivariate normal random vector $X \sim \mathcal{N}(\mu, \Sigma)$:

$$ Z = \Sigma^{-1/2} (X - \mu) \implies Z \sim \mathcal{N}(0, I) $$