Lecture #11: The Likelihood Function & Maximum-Likelihood Estimators
MATH 321 – Mathematical Statistics I
November 2, 2025
The Likelihood Function
\[ \DeclareMathOperator*{\argmax}{arg\!max} \]
Suppose we collect data from a known distribution, but we don’t know the parameters. How can we use this data to estimate the true value of the parameters?
For example, suppose I’m polling for tomorrow’s gubernatorial election in Virginia between the Democrat Abigail Spanberger and the Republican Winsome Earle-Sears.
I can treat each voter’s as a Bernoulli random variable, but how can I estimate the true proportion of voters who will vote for Spanberger?
The Likelihood Function
Let \(\boldsymbol{X} = [X_1, \dots, X_N]^T\) be a random vector drawn from a joint PDF \(f_{\boldsymbol{X}}(\boldsymbol{x}; \boldsymbol{\theta})\), and let \(\boldsymbol{x} = [x_1, \dots, x_N]^T\) be the realizations. The likelihood functions is a function of the parameter \(\boldsymbol{\theta}\) given the realizations \(\boldsymbol{x}\): \[ \mathcal{L}(\boldsymbol{\theta} \mid \boldsymbol{x}) \overset{\text{def}}{=} f_{\boldsymbol{X}}(\boldsymbol{x}; \boldsymbol{\theta}). \]
Given i.i.d. random variables \(X_1, \dots, X_N\) that all have the same PDF \(f_{X_n}(x_n)\), the likelihood function is: \[ \mathcal{L}(\boldsymbol{\theta} \mid \boldsymbol{x}) \overset{\text{def}}{=} \prod_{n=1}^N f_{X_n}(x_n; \boldsymbol{\theta}). \]
What are the constants in these equations? What are the variables?
The Log-Likelihood Function
Given a set of i.i.d. random variables \(X_1, \dots, X_N\) with PDF \(f_{X_n}(x_n; \boldsymbol{\theta})\), the log-likelihood is defined as \[ \log \mathcal{L}(\boldsymbol{\theta} \mid \boldsymbol{x}) = \log f_{\boldsymbol{X}}(\boldsymbol{x}; \boldsymbol{\theta}) = \sum_{n=1}^N \log f_{X_n}(x_n; \boldsymbol{\theta}). \]
The log-likelihood is useful because it turns a product into a sum, which makes it much easier to differentiate, but the critical points remain the same before and after taking the logarithm.
Example: The Log-Likelihood
Example: Find the log-likelihood of a sequence of i.i.d. Bernoulli random variables \(X_1, \dots, X_N\) with parameter \(\theta\).
Example: Find the log-likelihood of a sequence of i.i.d. Normal random variables \(X_1, \dots, X_N\) with mean \(\mu\) and variance \(\sigma^2\).
The Maximum-Likelihood Estimator
Let \(\mathcal{L}(\boldsymbol{\theta})\) be the likelihood function of the parameter \(\boldsymbol{\theta}\) given the measurements \(\boldsymbol{x} = [x_1, \dots, x_N]^T\). The maximum-likelihood estimate of the parameter \(\boldsymbol{\theta}\) is a parameter that maximizes the likelihood: \[ \widehat{\boldsymbol{\theta}}_{ML} \overset{\text{def}}{=} \argmax_{\boldsymbol{\theta}} \,\mathcal{L}(\boldsymbol{\theta} \mid \boldsymbol{x}). \]
Example: Maximum-Likelihood Esitmators
Example: Find the ML estimate for a set of i.i.d. Bernoulli random variables \(\{ X_1, \dots, X_N \}\) with \(X_i \sim \text{Bernoulli}(\theta)\) for \(i = 1, \dots, N\).
Example: Find the ML estimates for a set of i.i.d. Normal random variables \(\{ X_1, \dots, X_N \}\) with \(X_i \sim \text{Normal}(\mu, \sigma^2)\) for \(i = 1, \dots, N\).
Estimate vs. Estimator
An estimate is a number, e.g. \(\widehat{\theta}_{ML} = \dfrac{1}{N} \displaystyle\sum_{i=1}^N x_i\). It is the random realization of a random variable.
An estimator is a random variable, e.g. \(\widehat{\Theta}_{ML} = \dfrac{1}{N} \displaystyle\sum_{i=1}^N X_i\). It takes a set of random variables and generates another random variable.
Unbiased Estimators
An estimator \(\widehat{\Theta}\) of \(\theta\) is unbiased if \[ E[\widehat{\Theta}] = \theta. \]
Example: Let \(X_1, \dots, X_N\) be i.i.d. Normal random variables with unknown mean \(\mu\) and unknown variance \(\sigma^2\). We have shown that the ML estimators are: \[ \begin{aligned} \widehat{\mu}_{ML} &= \frac{1}{N} \sum_{i=1}^N X_i & &\text{and} & \widehat{\sigma}^2_{ML} = \frac{1}{N} \sum_{i=1}^N (X_i - \widehat{\mu}_{ML}^2. \end{aligned} \] It is easy to show that \(E[\widehat{\mu}_ML] = \mu\). How about \(\widehat{\sigma}^2_{ML}\)? Is it an unbiased estimator?
Consistent Estimators
An estimator \(\widehat{\Theta}\) of \(\theta\) is consistent if \(\widehat{\Theta}_N \overset{p}{\rightarrow} \theta\), i.e., \[ \lim\limits_{N \to \infty} P\Big[ \Big\lvert \widehat{\Theta}_N - \theta \Big\rvert \ge \varepsilon \Big] = 0. \]
Example: The sample mean \(\overline{X} = \dfrac{1}{N} \displaystyle\sum_{i=1}^N X_i\) is a consistent estimator of \(E(X)\). Why?
Example: The sample variance is a consistent estimator of \(V(X)\). Why?
