Properties of Expectation, Variance, and Covariance

This is a quick summary of the properties of expectation, variance, and covariance.

For details:

Table of contents

Expectation
Conditional Expectation
Variance
Conditional Variance
Covariance

Expectation

$\begin{matrix} (Expectated value of a constant) & E [c] = c \end{matrix}$

For the same reason

E [E [X]] = E [X]

$\begin{matrix} (Linearity of expectation) & E [a X + b Y] = a E [X] + b E [Y] \end{matrix}$ $\begin{matrix} (Expectation of joint RVs) & E [X Y] = E [X] E [Y] + Cov (X, Y) \end{matrix}$

Independence

When $X$ and $Y$ are independent,

\begin{matrix} Cov (X, Y) = 0 \\ E [X Y] = E [X] E [Y] \end{matrix}

Conditional Expectation

$\begin{matrix} (Law of total expectation) & E [X] = E [E [X | Y]] \end{matrix}$

Conditional Expectation as RV

$E [X | Y]$ is a random variable of $Y$ .

Hence $E [E [X | Y]]$ is aggregating over all possible values of $Y$ , that’s why we are left with $E [X]$ .

$\begin{matrix} E [a | X] = a \\ E [a X + b Y | Z] = a E [X | Z] + b E [Y | Z] \end{matrix}$

The above are linearity.

$\begin{matrix} E [X | X] = X \\ E [g (X) | X] = g (X) \end{matrix}$

The above are obvious. If you have $X$ , you’re expected to get $X$ .

$E [X | Y, g (Y)] = E [X | Y]$

The above is also obvious: if you already know $Y$ , knowing $g (Y)$ doesn’t change anything.

$E [X g (Y) | Y] = g (Y) E [X | Y]$ $E [E [X | Y, Z] | Y] = E [X | Y]$

These need a bit more thought.

Variance

$\begin{aligned} Var (X) & = E [(X - E [X])^{2}] \\ = E [X^{2}] - E [X]^{2} \end{aligned}$

Derivation

Just remember that $E [E [X]] = E [X]$ and $E [E [X]^{2}] = E [X]^{2}$ , because $E [X]$ is a constant.

\begin{aligned} Var (X) & = E [(X - E [X])^{2}] \\ = E [X^{2} - 2 X E [X] + E [X]^{2}] \\ = E [X^{2}] - 2 E [X] E [X] + E [X]^{2} \\ = E [X^{2}] - E [X]^{2} \end{aligned}

$\begin{matrix} Var (a X + b) = a^{2} Var (X) \\ Var (a X + b Y) = a^{2} Var (X) + b^{2} Var (Y) + 2 a b Cov (X, Y) \\ Var (a X - b Y) = a^{2} Var (X) + b^{2} Var (Y) - 2 a b Cov (X, Y) \end{matrix}$

Conditional Variance

$\begin{aligned} Var (X | Y) & = E [(X - E [X | Y])^{2} | Y] \\ = E [X^{2} | Y] - E [X | Y]^{2} \end{aligned}$

Derivation

\begin{aligned} Var (X | Y) & = E [(X - E [X | Y])^{2} | Y] \\ = E [X^{2} - 2 X E [X | Y] + E [X | Y]^{2} | Y] \\ (*) & = E [X^{2} | Y] - 2 E [X E [X | Y] | Y] + E [E [X | Y]^{2} | Y] \\ = E [X^{2} | Y] - 2 E [X | Y] E [X | Y] + E [X | Y]^{2} \\ = E [X^{2} | Y] - E [X | Y]^{2} \end{aligned}

In $(*)$ , remember that $E [X | Y]$ is a RV of $Y$ . See above:

\begin{matrix} E [X g (Y) | Y] = g (Y) E [X | Y] \\ E [g (Y) | Y] = g (Y) \end{matrix}

$\begin{matrix} (Law of total variance) & Var (X) = E [Var (X | Y)] + Var (E [X | Y]) \end{matrix}$

Derivation

\begin{aligned} E [Var (X | Y)] & = E [E [X^{2} | Y] - E [X | Y]^{2}] \\ = E [E [X^{2} | Y]] - E [E [X | Y]^{2}] \\ = E [X^{2}] - E [E [X | Y]^{2}] \end{aligned}

and,

\begin{aligned} Var (E [X | Y]) & = E [E [X | Y]^{2}] - E [E [X | Y]]^{2} \\ = E [E [X | Y]^{2}] - E [X]^{2} \end{aligned}

Then,

E [Var (X | Y)] + Var (E [X | Y]) = E [X^{2}] - E [X]^{2} = Var (X)

Covariance

$\begin{aligned} Cov (X, Y) & = E [(X - E [X]) (Y - E [Y])] \\ = E [X Y] - E [X] E [Y] \end{aligned}$