Lecture #9: Moment-Generating Functions

Moment-Generating Functions

For any random variable \(X\), the moment-generating function (MGF) \(M_X(t)\) of a random variable \(X\) is: \[ M_X(t) = E[e^{tX}]. \]

Moment-Generating Function Example

Example:

1. Find the moment-generating function for \(X \sim \text{Exponential}(\lambda)\).
2. Find the moment-generating function for \(X \sim \text{Bernoulli}(p)\).
3. Find the moment-generating function for \(X \sim \text{Binomial}(n, p)\).

Properties of Moment-Generating Functions

Theorem: The moment-generating function has the properties that:

1. \(M_X(0) = 1\).
2. \(\dfrac{d}{ds} M_X(s) \bigg\rvert_{s=0} = E[X]\).
3. \(\dfrac{d^2}{ds^2} M_X(s) \bigg\rvert_{s=0} = E[X^2]\).
4. \(\dfrac{d^k}{ds^k} M_X(s) \bigg\rvert_{s=0} = E[X^k]\), for any positive integer \(k\).

Properties of Moment-Generating Functions

Theorem: Let \(X\) and \(Y\) be independent random variables. Let \(Z = X + Y\). Then \[ M_Z(t) = M_X(t) M_Y(t). \]

What’s the Deal with the MGF?

The moment-generating function of a random variable uniquely defines the distribution.

Moment-Generating Functions of i.i.d. Random Variables

Theorem: Let \(M_X(t)\) and \(M_Y(t)\) denote the moment-generating functions of random variables \(X\) and \(Y\). If both moment-generating functions and \(M_X(t) = M_Y(t)\) for all \(t \in \mathbb{R}\), then \(X\) and \(Y\) have the same probability distribution.

If these independent random variables are further assumed to be identically distributed, the moment-generating function is \[ M_Z(t) = \Big(M_{X_1}(t)\Big)^N. \]

Examples of MGFs of i.i.d. Random Variables

Examples:

1. Let \(X_1, \dots, X_N\) be a sequence of i.i.d. Bernoulli random variables with parameter \(p\). Let \(Z = X_1 + \dots + X_N\). What is the distribution of \(Z\)?
2. Let \(X_1, \dots, X_N\) be a sequence of i.i.d. binomial random variables with parameters \(n\) and \(p\). Let \(Z = X_1 + \dots + X_N\). What is the distribution of \(Z\)?
3. Let \(X_1, \dots, X_N\) be a sequence of independent Poisson random variables with parameter \(\lambda_i\). Let \(Z = X_1 + \dots + X_N\). What is the distribution of \(Z\)?
4. Let \(X_1, \dots, X_N\) be a sequence of independent normal random variables with parameters \(\mu_i\) and \(\sigma_i^2\).. Let \(Z = X_1 + \dots + X_N\). What is the distribution of \(Z\)?

The Characteristic Function

There is a related function that does exist for all distributions, the characteristic function \(\varphi_X(t)\).

Properties of the Characteristic Function

Properties:

1. \(\varphi_X(-it) = M_X(t)\).
2. \(\dfrac{d^k}{ds^k} i^k \varphi_X(s) \bigg\rvert_{s=0} = E[X^k]\), for any positive integer \(k\).
3. Consider independent random variables \(X_1, \dots, X_N\). Let \(Z = \sum_{i=1}^N X_i\) be the sum of random variables. Then the MGF of \(Z\) is \[ \varphi_Z(t) = \prod_{i=1}^N \varphi_{X_i}(t). \]