Convergence of Random Variables

We cannot directly use the calculus definition of convergence when dealing with random vectors.

For example, suppose you have $n$ samples $X_1, X_2, \ldots, X_n$ from the same distribution of a random variable $X$.

Even if $n \to \infty$, we cannot directly say that $X_n$ converges to $X$, even though they have the same distribution, because they are random variables and $\Pr(X_n = X) = 0$.

So we introduce different types of convergence for random variables.

Table of contents
  1. Convergence in Distribution
    1. Slutsky’s Theorem
  2. Convergence in Probability
  3. Convergence in $r$-th Mean
    1. Convergence in Quadratic Mean
    2. General $r$-th Mean
  4. Chain of Implications
    1. For Point Mass Distribution

Convergence in Distribution

$X_n$ converges in distribution to $X$ (or $X_n \leadsto X$, $X_n \xrightarrow{d} X$) if:

$$ \lim_{n \to \infty} F_{n}(x) = F(x) $$

where $F_{n}$ and $F$ are the CDFs of $X_n$ and $X$ respectively.

This is also known as weak convergence.

  • If $X_n \leadsto X$ and $Y_n \leadsto c$, then $X_n + Y_n \leadsto X + c$.
  • If $X_n \leadsto X$ and $Y_n \leadsto c$, then $X_n Y_n \leadsto Xc$.
  • If $X_n \leadsto X$, then $g(X_n) \leadsto g(X)$ for any continuous function $g$.

$Y_n \leadsto c$ means $Y_n$ converges to a point mass distribution Y at $c$. Where $P(Y = c) = 1$.

Slutsky’s Theorem

The first two bullet points above are part of Slutsky’s Theorem.

Convergence in Probability

$X_n$ converges in probability to $X$ (or $X_n \xrightarrow{P} X$, $\plim_{n \to \infty} X_n = X$) if:

$$ \lim_{n \to \infty} \Pr(|X_n - X| > \epsilon) = 0 $$

for all $\epsilon > 0$.

  • If $X_n \xrightarrow{P} X$ and $Y_n \xrightarrow{P} Y$, then $X_n + Y_n \xrightarrow{P} X + Y$.
  • If $X_n \xrightarrow{P} X$ and $Y_n \xrightarrow{P} Y$, then $X_n Y_n \xrightarrow{P} XY$.
  • If $X_n \xrightarrow{P} X$, then $g(X_n) \xrightarrow{P} g(X)$ for any continuous function $g$.

Convergence in $r$-th Mean

Convergence in Quadratic Mean

$X_n$ converges in quadratic mean to $X$ (or $X_n \xrightarrow{qm} X$) if:

$$ \lim_{n \to \infty} \E[(X_n - X)^2] = 0 $$

Also called convergence in $L^2$ norm or convergence in mean square.

  • If $X_n \xrightarrow{qm} X$ and $Y_n \xrightarrow{qm} Y$, then $X_n + Y_n \xrightarrow{qm} X + Y$.

General $r$-th Mean

$X_n$ converges in $r$-th mean to $X$ if:

$$ \lim_{n \to \infty} \E[|X_n - X|^r] = 0 $$

Also called convergence in $L^r$ norm.

Chain of Implications

From the strongest to the weakest:

$$ X_n \xrightarrow{qm} X \quad \implies \quad X_n \xrightarrow{P} X \quad \implies \quad X_n \leadsto X $$

The converse is false.

For Point Mass Distribution

Only when $X$ is a point mass distribution at $c$, i.e.:

\[\Pr(X = c) = 1\]

the following holds:

$$ X_n \leadsto c \quad \implies \quad X_n \xrightarrow{P} c \quad $$