Conditional Expectation / Variance
Conditionals random variables come in handy when we construct hierarchical models where one distribution depends on another.
Summary statistics like expectation and variance are often of interest in any models.
But when we have a hierarchical model, we’d have to go through all the trouble of finding joint distributions, marginalizing them, and then finding the summary statistics.
Knowing some properties of conditional expectation and variance can save us a lot of time.
Table of contents
Conditional Expectation
Condition on a Random Variable
The conditional expectation of a random variable $X$ given $Y=y$ is:
$$ \begin{align*} \E[X|Y=y] &= \sum_x x P(X=x|Y=y) \\[1em] \E[X|Y=y] &= \int x f_{X|Y}(x|y) dx \end{align*} $$
Condition on an Event
When $A$ is an event of $\{Y=y\}$,
\[\E[X|A] = \sum_x x P(X=x|A) = \E[X|Y=y]\]Multivariate Transformation
The conditional expectation of $r(X, Y)$ given $Y=y$ is:
$$ \begin{align*} \E[r(X, Y)|Y=y] &= \sum_x r(x, y) P(X=x|Y=y) \\[1em] \E[r(X, Y)|Y=y] &= \int r(x, y) f_{X|Y}(x|y) dx \end{align*} $$
Conditional Expectation as a Random Variable
Important thing to note is that $\E[X|Y=y]$ is a function of $\boldsymbol{y}$.
So the exact value is defined only when we have a specific realization of $Y$.
Therefore, $\E[X|Y]$ is a random variable, and not a value.
$\E[X|Y]$ is a random variable of $Y$.
Law of Total Expectation
Also called rule of iterated expectation.
For any two random variables $X$ and $Y$:
$$ \E[\E[X|Y]] = \E[X] \quad \text{and} \quad \E[\E[Y|X]] = \E[Y] $$
This is a valid expectation because $\E[X|Y]$ and $\E[Y|X]$ are both random variables of $Y$ and $X$, respectively.
What?
Taking $\E[\E[X|Y]]$ as an example, first we’d have to take the expectation of $\E[X|Y]$ over $Y$:
\[\E[\E[X|Y]] = \int \E[X|Y=y] f_Y(y) dy\]Then we’d have to take the expectation of that over $X$:
\[\E[\E[X|Y]] = \int \left( \int x f_{X|Y}(x|y) dx \right) f_Y(y) dy\]Since $f_{X|Y}(x|y)f_Y(y) = f_{X, Y}(x, y)$, we can simplify:
\[\E[\E[X|Y]] = \int \int x f_{X, Y}(x, y) dx dy = \E[X]\]Conditional Variance
The conditional variance of $Y$ given $X=x$ is:
$$ \begin{align*} \Var(Y|X=x) &= \E[(Y - \E[Y|X=x])^2|X=x] \\[1em] &= \int (y - \E[Y|X=x])^2 f_{Y|X}(y|x) dy \end{align*} $$
Conditional Variance as a Random Variable
Again, just like expectation, $\Var(Y|X)$ is a random variable of $X$.
Law of Total Variance
For random variables $X$ and $Y$:
$$ \Var(Y) = \E[\Var(Y|X)] + \Var(\E[Y|X]) $$
Remember it as $EV + VE$.