Lecture #12: Introduction to Bayesian Statistics

We can use the conditional density as our analogue for the conditional probability: \[ \begin{aligned} f_{X \mid Y}(x \mid y) &= \frac{f_{Y \mid X}(y \mid x) f_X(x)}{f_Y(y)} \\ &= \frac{f_{Y \mid X}(y \mid x) f_X(x)}{\dint_{-\infty}^\infty f_{Y \mid X}(y \mid t) f_X(t) \, dt} \end{aligned} \]

In the standard “frequentist” worldview, probability is the long-run relative frequency of an event. Ifyou run an experiment many times, the proportion of times the event occurs out of the total number of tries will approach the “true” probability.

In the Bayesian framework, probability is instead a degree of subjective belief. We update our beliefs with new data, and adjust our belief accordingly. If I believe that an event is more likely to occur, the probability is that much higher.

In the frequentist framework, parameters of a distribution or a model are constants that describe it. It exists, but it is unknown, so we need to estimate it.

To the Bayesian, because we dont know the value of the parameter, it is a random variable.

This is called the prior distribution.

It represents the statistician’s belief about how the parameter is distributed, before we see the data.

It might be an uninformative flat prior, or it might be more specific, fitted to the situation, based on prior research or just a hunch.

This is the probability of getting the data \(\tilde{y}\) with the parameter \(\tilde{\theta}\).

Because the posterior is a function of the parameter \(\tilde{\theta}\), we can then use it as the prior of the next experiment.

If we choose a prior such that, given a certain likelihood function, the posterior follows the same distribution as the prior, we call that a conjugate prior.

Two scientists may arrive at different conclusions from the same data, .

However, it can be shown that the influence of the prior goes to zero as the sample size increases (for all but the most bone-headed priors).

So the likelihood is: \[ \begin{aligned} \mathcal{L}(x_i \mid \theta) &= \prod_{i=1}^{n} \theta^x_1 (1 - \theta)^{1 - x_i} \\ &= \theta^{\sum x_i} (1 - \theta)^{n - \sum x_i}. \end{aligned} \]

The success probability \(\theta\) can take on any value between \(0\) and \(1\), so we need a distribution with that as the support. Can you think of a good choice?

The Beta distribution should work perfect! \[ g(\theta) = \frac{\Gamma(\alpha + \beta)}{\Gamma(\alpha) \Gamma(\beta)} \theta^{\alpha - 1} (1 - \theta)^{\beta - 1}. \]

Let’s use a prior distribution of \(\text{Beta}(\alpha = 10, \beta = 10)\).

Because it is a PDF, we can get the constant part by integrating and making sure the full density function integrates to \(1\).

Notice that if we set \(\alpha^* = \sum x_i + \alpha\) and \(\beta^* = n - \sum x_i + \beta\), then this is of the form \(\theta^{\alpha^* - 1} (1 - \theta)^{\beta^* - 1}\), or matches with the Beta distribution!

Thus, the Beta-Binomial is a conjugate prior pair.

We can use this to estimate \(\widehat{\theta}\): \[ \widehat{\theta} = E[ \theta \mid \tilde{x} ] = \frac{\alpha^*}{\alpha^* + \beta^*} = \frac{72}{72+48} = \frac{72}{120} = 0.60. \]

We can also use the distribution to get a credible interval (Bayesians don’t make confidence intervals) or hypothesis tests.

If the distribution is not of a known family, it can be really hard.

We use algorithms designed to make this easier, namely Markov Chain Monte Carlo samplers.

Some of the most common are the Metropolis-Hastings algorithm or a Gibbs sampler.