Lecture #8: Joint Distributions

Multivariate Random Variables

It’s all well and good to consider random variables in isolation, but to do the actual interesting things you all want to do in probability and statistics, we need to understand how random variables relate to one another.

Bivariate Normal Distribution

Joint Probability Mass Functions

Let \(X\) and \(Y\) be two discrete random variables. The joint probability mass function (PMF) of \(X\) and \(Y\) is defined as: \[ \begin{aligned} p_{X,Y}(x, y) &= P[\{ X = x \} \cap \{ Y = y \}] \\ &= P[\{ \omega \in \Omega : X(\omega) = x \text{ and } Y(\omega) = y \}]. \end{aligned} \]

Joint PMF Example

Example: Let \(X\) be a coin toss and \(Y\) be a die roll. Find \(p_{X,Y}\).

1. What is the probability of getting a heads and an even number?
2. Are \(X\) and \(Y\) independent?

Joint Probability Density Functions

Let \(X\) and \(Y\) be two continuous random variables. The joint probability density function (PDF) of \(X\) and \(Y\) is a function \(f_{X,Y}(x,y)\) that can be integrated to yield a probability: \[ P[(X, Y) \in A] = \iint_A f_{X,Y}(x,y) \, dx \, dy, \] for any event \(A \subseteq \Omega_X \times \Omega_Y\).

Joint PDF Example

Example: Let \(X\) and \(Y\) be jointly uniformly distributed on \(A = [1,3] \times [1,4]\).

1. What is the probability that \(X \ge 2\) and \(Y \ge 2\)?
2. What is the probability that \(X\) and \(Y\) are less than 1 apart?

Marginal PMFs and Marginal PDFs

The marginal probability mass function (PMF) is defined as: \[ \begin{aligned} p_X(x) &= \sum_{y \in \Omega_Y} p_{X,Y}(x,y) & &\text{and} & p_Y(y) &= \sum_{x \in \Omega_X} p_{X,Y}(x,y). \end{aligned} \]

Independent Random Variables

Two random variables \(X\) and \(Y\) are independent if their joint PMF/PDF factors into the product of the marginals: \[ p_{X,Y}(x, y) = p_X(x) \, p_Y(y) \qquad \text{or} \qquad f_{X,Y}(x, y) = f_X(x) \, f_Y(y) \] for all \(x\) and \(y\).

Joint PMF Example

Example: Let \(X\) be a coin toss and \(Y\) be a die roll. Find \(p_{X,Y}\).

3. Find the marginal PMFs of \(X\) and \(Y\).

Joint PDF Example

Example: Let \(X\) and \(Y\) be jointly uniformly distributed on \(A = [1,3] \times [1,4]\).

\[ f_{X,Y}(x,y) = \begin{cases} \frac{1}{6}, & \text{for } 1 \le x \le 3, 1 \le y \le 4, \\ 0, & \text{otherwise}. \end{cases} \]

3. Find the marginal PDFs of \(X\) and \(Y\).

Extending to \(N\) Random Variables

A collection of random variables \(X_1, X_2, \dots, X_N\) is called independent and identically distributed (i.i.d.) if:

1. All \(X_1, X_2, \dots, X_N\) are mutually independent.

2. All \(X_1, X_2, \dots, X_N\) have the same distribution, i.e. \(F_{X_1}(x) = F_{X_2}(x) = \dots = F_{X_N}(x)\) for all \(x\).

Joint Expectation

Let \(X\) and \(Y\) be two random variables. The joint expectation is \[ E[XY] = \sum_{y \in \Omega_Y} \sum_{x \in \Omega_X} xy \, p_{X,Y}(x,y), \] if \(X\) and \(Y\) are discrete, or \[ E[XY] = \int_{\Omega_Y} \int_{\Omega_X} xy \, f_{X,Y}(x,y) \, dx \, dy, \] if \(X\) and \(Y\) are continuous.

Joint Expectation Example

Example: Let \(f_{X,Y}(x,y) = x + y\) for \(0 \le x, y \le 1\) and \(0\) otherwise. Find \(E[XY]\).

Covariance

Let \(X\) and \(Y\) be two random variables. Then the covariance of \(X\) and \(Y\) is \[ Cov(X,Y) = E[(X - \mu_X)(Y - \mu_Y)], \] where \(\mu_X = E[X]\) and \(\mu_Y = E[Y]\).

Covariance

Theorem: Let \(X\) and \(Y\) be two random variables. Then \[ Cov(X,Y) = E[XY] - E[X]E[Y]. \]

Covariance Example

Joint Expectation and Variance

Theorem: For any \(X\) and \(Y\),

1. \(E[X + Y] = E[X] + E[Y]\).
2. \(V[X + Y] = V[X] + V[Y] + 2Cov(X,Y)\).

Joint Expectation and Variance

Theorem: Let \(Y_1, Y_2, \dots, Y_n\) and \(X_1, X_2, \dots, X_m\) be random variables with \(E(Y_i) = \mu_i\) and \(E(X_j) = \xi_j\). Define: \[ U_1 = \sum_{i=1}^n a_i Y_i \qquad \text{ and } \qquad U_2 = \sum_{j=1}^m b_j X_j \] for constants \(a_1, a_2, \dots, a_n\) and \(b_1, b_2, \dots, b_m\).

Joint Expectation and Variance

Theorem: Then the following hold:

1. \(E(U_1) = \displaystyle\sum_{i = 1}^n a_i \mu_i\).
2. \(V[U_1] = \displaystyle\sum_{i=1}^n a_i^2 V[Y_i] + 2 \underset{1 \le i < j \le n}{\sum \sum} a_i a_j Cov(Y_i,Y_j)\).
3. \(Cov(U_1, U_2) = \displaystyle\sum_{i=1}^n \sum_{j=1}^m a_i b_j Cov(Y_i, X_j)\).

Covariance and Independence

Theorem: If \(X\) and \(Y\) are independent, then \[ E[XY] = E[X]E[Y]. \]

1. Does statement (a) imply statement (b)?
2. Does statement (b) imply statement (a)?

Example of Zero Covariance but Non-Independent

Example: Let \(k\) be chosen uniformly at random from \(\{1, 2, 3, 4\}\), and define \(X\) and \(Y\) by the table below.

1. What is the covariance of \(X\) and \(Y\)?
2. Are \(X\) and \(Y\) independent?

The Correlation Coefficient

Let \(X\) and \(Y\) be two random variables. The correlation coefficient is \[ \rho = \frac{Cov(X, Y)}{\sqrt{V(X) V(Y)}}. \]

Cauchy-Schwarz Inequality

Theorem: For any random variables \(X\) and \(Y\), \[ \big( E[XY] \big)^2 \le E[X^2] E[Y^2]. \]

Conditional Probability

Recall that, if we have two events \(A\) and \(B\) and \(P(B) \ne 0\), that the conditional probability of \(A\) given \(B\) is: \[ P(A \mid B) = \frac{P(A \cap B)}{P(B)}. \]

Conditional PMFs

Let \(X\) and \(Y\) be two discrete random variables. The conditional probability mass function (PMF) of \(X\) given \(Y\) is \[ p_{X \mid Y}(x \mid y) = \frac{p_{X, Y}(x, y)}{p_Y(y)}, \qquad \text{provided } p_Y(y) > 0. \]

Conditional PMFs

Theorem: Let \(X\) and \(Y\) be two discrete random variables. Let \(A\) be an event. \[ \begin{aligned} P[X \in A \mid Y = y] &= \sum_{x \in A} p_{X \mid Y}(x \mid y) \\ &= \frac{\sum_{x \in A} p_{X, Y}(x, y)}{p_Y(y)}, \end{aligned} \] and \[ \begin{aligned} P[X \in A] &= \sum_{x \in A} \sum_{y \in \Omega_Y} p_{X \mid Y}(x \mid y) p_Y(y) \\ &= \sum_{y \in \Omega_Y} P[X \in A \mid Y = y] p_Y(y). \end{aligned} \]

Conditional PMFs Example

1. Find the marginal PMF \(p_X(x)\).
2. Find the conditional PMF \(p_{X \mid Y}(x \mid 1)\).

Conditional PDFs

Let \(X\) and \(Y\) be two continuous random variables. The conditional probability density function (PDF) of \(X\) given \(Y\) is \[ f_{X \mid Y}(x \mid y) = \frac{f_{X, Y}(x, y)}{f_Y(y)}, \qquad \text{provided } f_Y(y) > 0. \]

Conditional CDFs

Let \(X\) and \(Y\) be continuous random variables. Then the conditional cumulative distribution function of \(X\) given \(Y = y\) is \[ F_{X \mid Y}(x \mid y) = \frac{\displaystyle\int_{-\infty}^x f_{X,Y}(t, y) \, dt}{f_Y(y)}. \]

Conditional Probabilities

Theorem: Let \(X\) and \(Y\) be continuous random variables, and let \(A\) be an event.

1. \(P(X \in A \mid Y = y) = \displaystyle\int_A f_{X \mid Y}(x \mid y) \, dx\).
2. \(P(X \in A) = \displaystyle\int_{\Omega_Y} P(X \in A \mid Y = y) f_Y(y) \, dy\).

Conditional CDF Example

Example: Suppose that \(f(x, y) = \frac{e^{-\frac{x}{y}} e^{-y}}{y}\) for \(0 < x, y < \infty\) and 0 otherwise. Find \(P(X > 1 \mid Y = y)\).

Conditional Expectation

The conditional expectation of \(X\) given \(Y=y\) is \[ E[X \mid Y = y] = \sum_x x \, p_{X \mid Y}(x \mid y) \] for discrete random variables, and \[ E[X \mid Y = y] = \int_{-\infty}^\infty x \, f_{X \mid Y}(x \mid y) \, dx \] for continuous random variables.

Conditional Expectation and Linear Regression

Conditional Expectation and Linear Regression

Conditional Expectation Example

1. Find \(E[X \mid Y = 10^2]\).
2. Find \(E[X \mid Y = 10^4]\).

Law of Total Expectation

Theorem: The Law of Total Expectation states that \[ E[X] = \sum_y E[X \mid Y = y] p_Y(y), \] or \[ E[X] = \int_{-\infty}^\infty E[X \mid Y = y] f_Y(y) \, dy. \]

Law of Total Expectation Example

3. Find \(E[X]\)