Lecture #10: Inequalities, Convergence and the Central Limit Theorem
MATH 321 – Mathematical Statistics I
October 29, 2025
Inequalities
If we know a probability distribution (either the PDF or the CDF), we know, in theory, all the probabilities associated with it.
But what is we only know some summary statistics for \(X\)?
We will use some theoretical inequality results to develop this ides.
Markov’s Inequality
Theorem: Let \(X \ge 0\) be a non-negative random variable. Then, for any \(\varepsilon > 0\), we have \[ P[X \ge \varepsilon] \le \frac{E[X]}{\varepsilon}. \]
This can also be written as \[ \varepsilon P(X \ge \varepsilon) \le E[X]. \]
Example: The number of items sold in a week is a random variable with mean \(16\). Estimate the probability that next week’s sales exceed \(25\).
Tchebyshev’s Inequality
Theorem: Let \(X\) be a random variable with mean \(\mu\). Then for any \(\varepsilon > 0\), we have \[ P\big[ \lvert X - \mu \rvert \ge \varepsilon \big] \le \frac{V[X]}{\varepsilon^2}. \]
If we set \(\varepsilon = k\sigma\) for some positive integer \(k\), we get the more familiar form of Tchebyshev’s Inequality: \[ P\big[ \lvert X - \mu \rvert \ge k\sigma \big] \le \frac{1}{k^2}. \]
Example: Let \(X \sim \text{Uniform}(0,\, 2\sqrt{2})\). Find the bound of Chebyshev’s inequality for the probability \(P[ \lvert X - \mu \rvert \ge 1 \big]\).
Chernoff’s Bound
Theorem: Let \(X\) be a random variable. The, for any \(\varepsilon \ge 0\), we have that \[ P[X \ge \varepsilon] \le e^{-\varphi(\varepsilon)}, \] where \[ \varphi(\varepsilon) = \max_{s > 0} \Bigg\{ s\varepsilon - \log M_X(s) \Bigg\}, \] and \(M_X(s)\) is the moment-generating function.
\(\varphi(\varepsilon)\) is called the Fenchel-Legendre dual function of \(\log M_X\).
Hoeffding’s Inequality
Theorem: Let \(X_1, X_2, \dots, X_N\) be i.i.d. random variables with \(0 \le X_i \le 1\), and \(E[X_n] = \mu\). Then \[ P\bigg[ \lvert \overline{X}_N - \mu \rvert > \epsilon \bigg] \le 2e^{-2\epsilon^2 N}, \] where \(\overline{X}_N = \dfrac{1}{N} \displaystyle\sum_{i=1}^N X_i\).
Sample Averages
The sample average of a sequence of random varaibles \(X_1, \dots, X_N\) is \[ \overline{X}_N = \frac{1}{N} \sum_{i = 1}^N X_i. \]
This is a linear combination of random variables, so it too is a random variable.
Thus, it has an expectaion, variance, PDF, CDF and everything.
Sample Averages
If \(X_1, \dots, X_N\) are i.i.d., then the summary statistics \(E(\overline{X_N})\) and \(V(\overline{X}_N)\) are easy.
But what is the probability distribution of \(\overline{X}_N\)?
The Weak Law of Large Numbers
Theorem: Let \(X_1, \dots, X_N\) be a set of i.i.d. random variables with mean \(\mu\) and variance \(\sigma^2\). Assume \(E[X^2] < \infty\). Let \(\overline{X}_N = \dfrac{1}{N} \displaystyle\sum_{i=1}^N\). Then, for any \(\varepsilon > 0\), \[ \lim\limits_{N \to \infty} P \bigg[ \lvert \overline{X}_n - \mu \rvert > \varepsilon \bigg] = 0. \]
Convergence in Probability
A sequence of random variables \(A_1, \dots, A_N\) converges in probability to a deterministic number \(\alpha\) if for every \(\varepsilon > 0\), \[ \lim_{N \to \infty} P\Big[ \lvert A_N - \alpha \rvert > \varepsilon \Big] = 0. \]
We write \(A_N \overset{p}{\longrightarrow} \alpha\) to denote convergence in probability.
Example: Let \(X_1, \dots, X_N\) be i.i.d. random variables with \(X_i \sim \text{Uniform}(0,1)\). Define \(A_N = \min\Big\{ X_1, \dots, X_n\Big\}\). Show that \(A_N\) converges in probability to zero.
Convergence
There are many types of convergence.
A sequence of random variables \(A_1, \dots, A_N\) converges almost surely to \(\alpha\) if \[ \begin{aligned} P\Big[ \lim_{N \to \infty} A_N = \alpha \Big] &= 1, & &\text{or} & P\Big[ \lim_{N \to \infty} \lvert A_N - \alpha \rvert < \varepsilon \Big] &= 1. \end{aligned} \]
We write \(A_N \overset{a.s.}{\longrightarrow} \alpha\) to denote almost sure convergence.
The Strong Law of Large Numbers
Theorem: Let \(X_1, \dots, X_N\) be a sequence of i.i.d. random variables with common mean \(\mu\) and variance \(\sigma^2\). Assume \(E[X^4] < \infty\). Let \(\overline{X}_N = \dfrac{1}{N} \displaystyle\sum_{i=1}^N\) be the sample average. Then \[ P\bigg[ \lim_{N \to \infty} \overline{X}_N = \mu \bigg] = 1. \]
The Central Limit Theorem
Theorem: Let \(X_1, \dots, X_N\) be a sequence of i.i.d. random variables of mean \(E[X_n] = \mu\) and variance \(V[X_n] = \sigma^2\). Alsom assume that \(E\Big[ \lvert X_n^3 \rvert \Big] < \infty\). Let \(\overline{X}_N = \dfrac{1}{N} \displaystyle\sum_{i=1}^N\) be the sample average and let \(Z_N = \sqrt{N} \left( \dfrac{\overline{X}_N - \mu}{\sigma} \right)\). Then \[ \lim\limits_{N \to \infty} F_{\overline{Z}_N}(z) = F_Z(z), \] where \(Z \sim \text{Normal}(0, 1)\).
The Central Limit Theorem Examples
Examples:
- Suppose we have \(30\) i.i.d. \(X_i \overset{i.i.d.}{\sim} \text{Binomial}(n = 20, p = 0.25)\). What is the probability that the smaple mean of these \(30\) variables is above \(8\)?
- Americans males is, on average, \(69\) inches tall, with a standard deviation of \(3\) inches. If I select a sample of \(9\) American males, what is the probability that our sample mean is within \(0.5\) of an inch?
- Suppose I want my sample mean to be within the \(1\) inch of the true population mean, with \(95\%\) probability. How many people do I need to include in my sample?
