Lecture #18: The Theory of Hypothesis Testing & \(p\)-values

STAT 200 – Introductory Statistics

Torin Quinlivan
tquinlivan@knox.edu

Knox College

February 10, 2026

The Theory Behind Hypothesis Testing

The theory behind hypothesis testing is:

  1. State the research hypothesis and the null hypothesis.
  2. Determine how much the data support the null hypothesis:
    • Determine the parameter tested.
    • Determine appropriate statistic.
    • Determine distribution of that statistic assuming the null hypothesis is true.
    • Determine how likely it is to observe the statistic (data) if the null hypothesis is true.
  3. Interpret that level of support.

Definition: Hypothesis

A hypothesis is a testable claim about reality.

Since it is a claim about reality, it concerns some aspect of the population (a parameter). The usual parameters hypothesized about at this level are the mean \(\mu\), proportion \(p\), and variance \(\sigma^2\). But we can hypothesize about any aspect of the population.

The “generic” parameter is \(\theta\).

Since it is a claim about reality, it separates all possible realities into those that are consistent with the hypothesis and those that are incompatible with it.

Definition: Research Hypothesis

The most important hypothesis is the one made by the researcher:

A research hypothesis is a testable claim about reality made by the scientist.

This is the claim that the statistician must eventually come to a conclusion about.

Because we are using statistics to test this hypothesis, we create two statistics-specific hypotheses. These two hypotheses divide all possible realities into two groups, those that are compatible with the research hypothesis and those that are not.

Definition: Null Hypothesis

The null hypothesis, \(H_0\), is a statement of the status quo or our prior beliefs. It is what we assume is true unless we see evidence to disprove it.

The null hypothesis is always in some way a statement of equality.

Definition: Alternative Hypothesis

The alternative hypothesis, \(H_A\), is what we are trying to prove with our sample.

The alternative hypothesis is the opposite of the null hypothesis, and if we reject the null hypothesis, we accept the sample as evidence in support of the alternative hypothesis.

Comparing the Hypotheses

Below is a table of all possible research hypotheses and their corresponding null and alternative hypotheses:

\(H_R\) \(H_0\) \(H_A\)
\(\theta < \theta_0\) \(\theta \ge \theta_0\) \(\theta < \theta_0\)
\(\theta = \theta_0\) \(\theta = \theta_0\) \(\theta \ne \theta_0\)
\(\theta > \theta_0\) \(\theta \le \theta_0\) \(\theta > \theta_0\)
\(\theta \le \theta_0\) \(\theta \le \theta_0\) \(\theta > \theta_0\)
\(\theta \ne \theta_0\) \(\theta = \theta_0\) \(\theta \ne \theta_0\)
\(\theta \ge \theta_0\) \(\theta \ge \theta_0\) \(\theta < \theta_0\)

The symbol \(\theta\) is a generic symbol for a parameter. It can represent \(\mu\), \(p\), \(\sigma^2\), or any other parameter you can imagine. The symbol \(\theta_0\) represents the value claimed by the researcher. It is a number.

Comparing the Hypotheses

\(H_R\) \(H_0\) \(H_A\)
\(\theta < \theta_0\) \(\theta \ge \theta_0\) \(\theta < \theta_0\)
\(\theta = \theta_0\) \(\theta = \theta_0\) \(\theta \ne \theta_0\)
\(\theta > \theta_0\) \(\theta \le \theta_0\) \(\theta > \theta_0\)
\(\theta \le \theta_0\) \(\theta \le \theta_0\) \(\theta > \theta_0\)
\(\theta \ne \theta_0\) \(\theta = \theta_0\) \(\theta \ne \theta_0\)
\(\theta \ge \theta_0\) \(\theta \ge \theta_0\) \(\theta < \theta_0\)

DO NOT MEMORIZE THIS TABLE! Instead learn what is says about the relationships between the research, null and alternative hypotheses.

What do all of the null hypotheses have in common? Why would that be a requirement? What is the relationship between the null and alternative hypotheses? Why does this make sense?

Definition: The \(p\)-value

The \(p\)-value is the probability of observing a sample with data this extreme, or more so, given that the null hypothesis is true.

The \(p\)-value is the probability of observing a test statistic this extreme, or more so, given that the null hypothesis is true.

The \(p\)-value is a measure of the amount of support in the data for the null hypothesis.

Definition: Decision Theory

We now have two of the three parts of the “hypothesis-testing theory” laid out earlier. The last part is making a decision about the research hypothesis based on the data.

Do the data sufficiently support the research hypothesis?

Note that the statement is binary: “Yes” or “No”.

Since the decision is binary, we need to determine a “cut-off” point between what sufficiently supports the research hypothesis and what does not. This threshold is referred to as “the alpha-value”, \(\alpha\).

If the level of support is less than \(\alpha\), we reject the null hypothesis. Otherwise, we do not, i.e. we fail to reject.

Definition: Decision Theory

The alpha value, \(\boldsymbol{\alpha}\) is the Type I error rate claimed by the statistician.

A Type I error occurs when the researcher rejects a true null hypothesis. This is also known as a false positive.

The Type I error rate is the proportion of the time the researcher commits a Type I error (rejects a true null hypothesis).

Note: The actual Type I error rate may not actually be \(\alpha\), what it is claimed to be.

Definition: Decision Theory

A Type II error occurs when the researcher fails to reject a false null hypothesis. This is also known as a false negative. It is symbolized by \(\beta\).

Both \(\alpha\) and \(\beta\) are error rates. We would like to minimize both.

  • However, decreasing \(\alpha\) results in increasing \(\beta\).
  • Furthermore, reducing either to zero results in the other being 100%.
  • Fisher suggested that we specify \(\alpha\) and let \(\beta\) be whatever it is.

Definition: Decision Theory

Of course, these aren’t necessarily the only types of errors that can happen. For instance, we could reject a false null hypothesis, but for the wrong reasons.

Possible reasons for this:

  • using the wrong test.
  • aggregation bias.
  • the ecological fallacy.
  • collinearity among predictors.

We can avoid these errors by fully understanding the tests we are using and their requirements and assumptions. The wrong test, the wrong interpretation, the wrong assumptions can all lead to errors.

Example #1: Coins

Suppose I want to determine if a certain coin is biased in favor of getting heads.

Using statistical language, my research hypothesis is

\[ \begin{aligned} H_R: p > 0.500 \end{aligned} \]

To test this, I flip the coin 100 times and get 58 heads.

Example #1: Coins

Since the research hypothesis is \(p > 0.500\), we know that the hypotheses are:

\[ \begin{aligned} H_0: p \le 0.500 \\ H_A: p > 0.500 \end{aligned} \]

If the coin is fair, we know that \(X\) is distributed as

\[ \begin{aligned} X \sim \text{Binomial}(n = 100, p = 0.500). \end{aligned} \]

So if the number of observed heads is too large, we know that the alternative hypothesis is more likely to be correct.

Example #1: Coins

The next step is to calculate the \(p\)-value:

The \(p\)-value is the probability of observing sample data this extreme — or more so — given the null hypothesis is true.

Thus, the \(p\)-value is

\[ \begin{aligned} \mathbb{P}(X \ge 58 \,\vert\, X \sim \text{Binomial}(100, 0.500)) \end{aligned} \]

Example #1: Coins

In R:

1 - pbinom(57, size=100, prob=0.500)
[1] 0.06660531

The probability of observing data this extreme (or more so), given the null hypothesis is true, is \(0.0660531\).

Example #1: Coins

The probability of observing data this extreme (or more so), given the null hypothesis is true, is \(0.0660531\).

As our \(p\)-value is greater than \(\alpha = 0.05\), we do not reject the null hypothesis, and conclude that this sample does not provide sufficient evidence that the coin is biased towards heads.

Example #2: Blackjack

I believe that my blackjack dealer is cheating. If everything is fair, then I would expect to have a blackjack 4.75% of the time. I have played \(n=132\) hands and got a blackjack only once. Is this sufficient evidence that the dealer is cheating?

Specifically, using statistical language, my research hypothesis (claim) is:

\[ H_R : p < 0.0475 \]

Example #3: Die

I believe that this die is biased against getting a 6.

Specifically, using statistical language, my research hypothesis (claim) is:

\[ H_R : p < 0.1667 \]

To test this, I roll the die 100 times and get a 6 a total of 9 times.

Example #4: McDowell’s

McDowell’s claims that the weight of a pounder hamburger patty (before cooking) is at least 16 ounces, with a standard deviation of \(\sigma = 1\) ounce.

In symbols, this is

\[ H_R : \mu \ge 16 \]

To test this, we weigh a stack of \(n=100\) patties and find that the total weight was 1575 ounces. Does this offer sufficient evidence that McDowell’s is incorrect in their claim?

Example #4: McDowell’s

Since the research hypothesis is \(\mu = 16\), we know that the hypotheses are:

\[ \begin{aligned} H_0: \mu = 16 \\ H_A: \mu \le 16 \end{aligned} \]

Because the sample size \(n\) is greater than 30, which is our rule of thumb for the Central Limit Theorem, we know that:

\[ \begin{aligned} T \sim \text{Normal}(\mu_T = n \mu = 1600, \sigma^2_T = n \sigma^2 = 100). \end{aligned} \]

So if the total weight is too low, we know that the alternative hypothesis is more likely to be correct.

Example #4: McDowell’s

The next step is to calculate the \(p\)-value:

The \(p\)-value is the probability of observing sample data this extreme — or more so — given the null hypothesis is true.

Thus, the \(p\)-value is

\[ \begin{aligned} \mathbb{P}(T \le 1490 \,\vert\, T \sim \text{Normal}(\mu = 1600, \sigma = 10)) \end{aligned} \]

Example #4: McDowell’s

In R:

pnorm(1575, mean=1600, sd = 10)
[1] 0.006209665

The probability of observing data this extreme (or more so), given the null hypothesis is true, is \(0.006209665\).

Example #4: McDowell’s

The probability of observing data this extreme (or more so), given the null hypothesis is true, is \(0.006209665\).

As our \(p\)-value is less than \(\alpha = 0.05\), we reject the null hypothesis, and conclude that this sample does provide sufficient evidence that McDowell’s is incorrect in their claim that their burgers weigh one pound.

Example #5: McDowell’s

McDowell’s claims that the weight of a pounder hamburger patty (before cooking) is at least 16 ounces, with a standard deviation of \(\sigma = 1\) ounce.

In symbols, this is

\[ H_R : \mu \ge 16 \]

To test this, we weigh a stack of \(n=100\) patties and find that the average weight was 15.75 ounces. Does this offer sufficient evidence that McDowell’s is incorrect in their claim?

Example #6: McDowell’s

McDowell’s claims that the weight of a quarter-pounder hamburger patty (before cooking) is 4 ounces, with a standard deviation of \(\sigma = 1\) ounce.

In symbols, this is

\[ H_R : \mu = 4 \]

To test this, we weigh a stack of \(n=25\) patties and find that the average weight is only 3.76 ounces. Does this offer sufficient evidence that McDowell’s is incorrect in their claim?

Example #7: McDowell’s

McDowell’s claims that the weight of a pounder hamburger patty (before cooking) is at least 16 ounces, with a standard deviation of \(\sigma = 1\) ounce.

In symbols, this is

\[ H_R : \mu \ge 16 \]

To test this, we weigh a stack of \(n=100\) patties and find that the average weight is only 15.9 ounces. Does this offer sufficient evidence that McDowell’s is incorrect in their claim?

Example #8: McDowell’s

McDowell’s claims that the number of calories in a Big Mc is at most 350, with a standard deviation of \(\sigma = 50\) calories. \[ H_R : \mu \le 350 \]

To test this, we perform a calorimetry test on a stack of \(n = 100\) Big Mcs and find that the average number of calories is 343. Is there sufficient evident that McDowell’s is incorrect in their claim?