Mutivariate Distributions

Table of contents

Multinomial Distribution

Extension of the binomial distribution to more than two categories.

Instead of have success/failure, we have $k$ categories (e.g. dice roll, preference surveys).

We have a random vector $X = (X_{1}, \dots, X_{k})$ , where $X_{i}$ is the number of times category $i$ occurs.

For a given $n$ and a probability vector $p = (p_{1}, \dots, p_{k})$ ,

n = \sum_{i = 1}^{k} X_{i}

Then the probability mass function is:

$p_{X} (x) = (\binom{n}{x_{1} \dots x_{k}}) p_{1}^{x_{1}} \dots p_{k}^{x_{k}} = \frac{n!}{x_{1}! \dots x_{k}!} p_{1}^{x_{1}} \dots p_{k}^{x_{k}}$

And we denote:

$X \sim Multinomial (n, p)$

When $X \sim Multinomial (n, p)$ , the marginal distribution of $X_{i}$ is:

$X_{i} \sim Binomial (n, p_{i})$

The expected value of $X \sim Multinomial (n, p)$ is:

$n \cdot p = (\begin{matrix} n \cdot p_{1} \\ ⋮ \\ n \cdot p_{k} \end{matrix})$

The variance of $X \sim Multinomial (n, p)$ is the covariance matrix.

We have a random vector $X = (X_{1}, \dots, X_{k})$ .

The parameters are:

$μ = (μ_{1}, \dots, μ_{k})$ : mean vector
$Σ$ : $k \times k$ symmetric positive definite covariance matrix
$Σ_{i j} = Cov (X_{i}, X_{j}) = E [(X_{i} - μ_{i}) (X_{j} - μ_{j})]$

The probability density function is:

$f_{X} (x) = \frac{1}{\sqrt{(2 π)^{k} det (Σ)}} \exp (- \frac{1}{2} (x - μ)^{T} Σ^{- 1} (x - μ))$

And we denote:

$X \sim N (μ, Σ)$

It is similar to the univariate case, which looked like

Z = \frac{X - μ}{σ} ⟺ X = μ + σ Z

Let’s just take for granted that $Σ^{1 / 2}$ and $Σ^{- 1 / 2}$ exist.

For standard multivariate normal random vector $Z \sim N (0, I)$ ,

$X = μ + Σ^{1 / 2} Z ⟹ X \sim N (μ, Σ)$

And for multivariate normal random vector $X \sim N (μ, Σ)$ :

$Z = Σ^{- 1 / 2} (X - μ) ⟹ Z \sim N (0, I)$