Lecture #19: The Practice of Hypothesis Testing & \(p\)-values

The Theory Behind Hypothesis Testing

The theory behind hypothesis testing is:

1. State the research hypothesis and the null hypothesis.
2. Determine how much the data support the null hypothesis:
   • Determine the parameter tested.
   • Determine appropriate statistic.
   • Determine distribution of that statistic assuming the null hypothesis is true.
   • Determine how likely it is to observe the statistic (data) if the null hypothesis is true.
3. Interpret that level of support.

Example #1: The Big Mick

McDowell’s claims that the weight of a Big Mick (before cooking) is 4 ounces.

\[ \begin{aligned} H_R : \mu = 4 \end{aligned} \]

To test this, a dietary scientist collects data by weighing a random sample of \(n = 10\) patties. Here are the weights she collected:

Does this provide sufficient evidence that McDowell’s is incorrect in their claim?

Bootstrapping

Again, here are the data she collected:

Note that the hypothesis concerns the population mean and the likelihood of a claimed population mean. If the data are representative of the population, we could generate the distribution of sample means and see how likely the hypothesized mean is.

This is called “bootstrapping the data”. It only requires the data are representative of the population. It makes no distributional assumptions about the process that generated the data.

Bootstrapping

Here is the R-code to analyze the population mean using bootstrapping:

# The Data 
weight = c(3, 2, 1, 2, 2, 4, 2, 3, 5, 3) 

# Initialize variables 
B = 10000                               ## Number of iterations 
ts = numeric()                          ## Tell R to set aside memory 

# Simple bootstrapping of the mean 
for (i in 1:B) { 
    x = sample(weight, replace = TRUE)  ## Sample from data 
    ts[i] = mean(x)                     ## Calculate mean 
}

# Analysis Calculations 
2*mean(ts > 4)                          ## p-value 

[1] 2e-04

quantile(ts, c(0.025, 0.975))           ## Confidence Interval 

 2.5% 97.5% 
  2.1   3.4 

Bootstrapping

Here is the distribution for sample means:

Bootstrapping

So, with this distribution of sample means, what can we conclude about the null hypothesis?

• First, we can directly test the null hypothesis. Since the \(p\)-value is approximately 0.0002, we can reject the null hypothesis and conclude that the average patty weight is not \(4\) ounces.

• Second, we can get a \(95\%\) confidence interval from 2.1 to 3.4 ounces.

  • Since the hypothesized value of \(\mu = 4\) is not in the interval (between 2.1 and 3.4), we can conclude that the data do not support the null hypothesis at the \(95\%\) confidence level.

• We conclude that the alternative hypothesis is correct, that \(\mu \ne 4\). In fact, based on the confidence interval, we can also conclude \(\mu < 4\) ounces, that the average weight of a Big Mick at McDowell’s is less than the advertised 4 ounces.

Bootstrapping

Here is the distribution for sample means with the confidence interval highlighted:

Non-Parametric

Bootstrapping only requires that the data are representative of the population. If we are willing to also assume that the population is symmetric, then we can obtain a test that is more-powerful — the Wilcoxon test (1945).

How can we tell if it is reasonable to state that the population is symmetric?

The Hildebrand Rule will accomplish that for us:

hildebrand.rule(weight) 

  Group     Ratio    Skew
1   xx  0.1724879 No Skew

Thus, it is reasonable to treat the data as if they came from a symmetric process.

Non-Parametric

So, this data allows us to use the Wilcoxon test. Here is the code to perform it in R for our original alternative hypothesis of \(H_A : \mu \ne 4\).

wilcox.test(weight, mu=4, conf.int=TRUE)

    Wilcoxon signed rank test with continuity correction

data:  weight
V = 2.5, p-value = 0.01868
alternative hypothesis: true location is not equal to 4
95 percent confidence interval:
 1.999960 3.500001
sample estimates:
(pseudo)median 
      2.499996 

Be able to interpret every part of this output.

Non-Parametric

Conclusion:

We are asked to determine if the average weight of hamburger patties is 4 ounces, before cooking. Because the Hildebrand ratio is \(H = 0.1725\), we are able to use the Wilcoxon procedure to test our hypothesis.

The p-value of 0.0187 is less than our usual alpha-value of \(\alpha = 0.05\). Thus, we reject the null hypothesis. There is significant evidence that the average weight of Big Mick at McDowell’s is less than a quarter of a pound. In fact, a \(95\%\) confidence interval for the average weight of a Big Mick is from 2.0 to 3.5 ounces.

Non-Parametric

Note: Again, the purpose of the conclusion is more than just conveying your numerical results. A good conclusion provides the procedures used, a brief defense of that procedure, and an interpretation of the results. Use numbers in the conclusion.

Parametric

Recall that bootstrapping only requires that the data are representative of the population (which is what we always assume about our sample). The Wilcoxon test also requires that the data are from a symmetric population.

If we are also willing to assume that the data come from a Normal distribution, then we can obtain a test that is even more powerful than the Wilcoxon test — the Student’s \(t\)-test.

How can we tell if it is reasonable to state that the population is Normal? There are a large number of “normality tests” available. One of the best is the Shapiro-Wilk test (1965).

Shapiro-Wilk Normality Test

Here is how to perform the Shapiro-Wilk test in R:

weight = c(3, 2, 1, 2, 2, 4, 2, 3, 5, 3) 
shapiroTest(weight) 

    Shapiro-Wilk normality test

data:  y
W = 0.91645, p-value = 0.3283

The null hypothesis for this test is that the data are from a Normal distribution. Thus, since the \(p\)-value is greater than 0.005, we conclude that it is reasonable to conclude the data came from a Normal process.

Student’s \(t\)-Test

Since it is reasonable to conclude the data are from a Normal process, we can perform the \(t\)-test.

The \(t\)-test assumes that, under the null hypothesis \(H_0\), the test statistic follows a \(t\) distribution with \(df = n - 1\) degrees of freedom.

\[ \begin{aligned} T = \frac{\overline{x} - \mu_0}{\frac{s}{\sqrt{n}}} \sim t_{n-1} \end{aligned} \]

Student’s \(t\)-Test

Then, we reject the null hypothesis if:

• \(T \le t_{0.05, n-1}\) if \(H_A: \mu < \mu_0\)
• \(T \ge t_{0.05, n-1}\) if \(H_A: \mu > \mu_0\)
• \(\lvert T \rvert \ge t_{0.05, n-1}\) if \(H_A: \mu \ne \mu_0\)

We can get the value for \(t_{0.05, n-1}\) or the \(p\)-value from a \(t\)-table.

Student’s \(t\)-Test

Here is the code to perform it in R for our hypothesis:

    t.test(weight, mu=4)

    One Sample t-test

data:  weight
t = -3.5455, df = 9, p-value = 0.00626
alternative hypothesis: true mean is not equal to 4
95 percent confidence interval:
 1.870542 3.529458
sample estimates:
mean of x 
      2.7 

Parametric

Conclusion:

We are asked to determine if the average weight of hamburger patties is 4 ounces, before cooking. Because the Shapiro-Wilk test suggests that the data are from a Normal distribution (\(p\)-value \(= 0.3283\)), we are able to use the one-sample \(t\)-test on our hypothesis.

The p-value of 0.0063 is less than our usual alpha-value of \(\alpha = 0.05\). Thus, we reject the null hypothesis. There is significant evidence that the average weight of the Big Mick at McDowell’s is less than a quarter of a pound. In fact, a \(95\%\) confidence interval for the average weight of a Big Mick burger is from 1.87 to 3.53 ounces.

Analysis Procedure Summary

Notice the procedure: We use the test that has the most requirements met by the data. The more assumptions met, the more powerful the test.

A powerful test is better able to reject a false null hypothesis. All things being equal, it has a lower Type II error rate. Because of this, it is a better test. One should use it if allowed.

Example #2: Drunk Driving

It is well known that drinking alcohol increases reaction time. When driving a car, this effect can be fatal.

To determine if this “common wisdom” is correct, I gathered a sample of STAT 200 students. I first measured the reaction time of each person sober. I then measured their reaction after drinking sufficient quantities of Hawkeye vodka to have a BAC of 0.05%.

Here are the change in reaction times (in milliseconds):

Example #2: Drunk Driving

The first step is to determine the appropriate test. I would like to use the most-powerful test, but those tests have a lot of assumptions.

The following load the data and test Normality:

rxJ = c(143, 182, 192, 205, 115, 362, 172, 192, 203, 61) 
shapiroTest(rxJ) 

    Shapiro-Wilk normality test

data:  y
W = 0.87506, p-value = 0.1144

According to the Shapiro-Wilk test (\(p\)-value = 0.1144), there is no reason to believe that the data are not of a Normal distribution. As such, we can use the \(t\)-test.

Example #2: Drunk Driving

The code to perform the appropriate \(t\)-test is:

t.test(rxJ, mu=0, alternative="greater") 

    One Sample t-test

data:  rxJ
t = 7.437, df = 9, p-value = 1.973e-05
alternative hypothesis: true mean is greater than 0
95 percent confidence interval:
 137.6669      Inf
sample estimates:
mean of x 
    182.7 

Example #2: Drunk Driving

Conclusion:

We are asked to determine if the average difference in reaction times between sober and drunk (BAC = 0.05%) is positive. Because the Shapiro-Wilk test suggests that the data are from a Normal distribution (\(p\)-value = 0.1144), we are able to use the one-sample \(t\)-test on our hypothesis.

The \(p\)-value of less than 0.0001 is less than our usual alpha-value of \(\alpha = 0.05\). Thus, we reject the null hypothesis. There is significant evidence that the average difference in reaction times is positive. In fact, we are \(95\%\) confident that the average increase in reaction time is at least 137.67 milliseconds.

Example #3: Distracted Driving

While it is well known that drinking alcohol increases reaction time, it is also well known that talking on a telephone does the same.

To determine which affects reaction time most, I use the same people in the first example. After they get sober, I measure their reaction time while talking on their cell phones.

The change in reaction times between no-phone and yes-phone (in milliseconds) is provided below.

Example #3: Distracted Driving

The first step is to determine the appropriate test. I would like to use the most-powerful test, but those tests have a lot of assumptions.

The following load the data and test Normality:

rxP = c (88, 233, 240, 207, 188, 194, 184, 198, 193, 208) 
shapiroTest(rxP) 

    Shapiro-Wilk normality test

data:  y
W = 0.77916, p-value = 0.008085

The Shapiro-Wilk test (\(p\)-value = 0.0081) indicates that the data are not from a Normal distribution. However, the Hildebrand Rule suggests that the data are generated from a symmetric distribution (\(H = 20.0653\)). As such, we should use the Wilcoxon test to estimate the population mean (and test that it differs from zero).

Example #3: Distracted Driving

The code to perform the appropriate Wilcoxon test is:

wilcox.test(rxP, mu=0, alternative="greater", conf.int=TRUE) 

    Wilcoxon signed rank exact test

data:  rxP
V = 55, p-value = 0.0009766
alternative hypothesis: true location is greater than 0
95 percent confidence interval:
 184 Inf
sample estimates:
(pseudo)median 
           198 

Example #3: Distracted Driving

Conclusion:

We are asked to determine if the average difference in reaction times between using a phone and not is positive. The Shapiro-Wilk test suggests that the data are not from a Normal distribution (\(p\)-value = 0.0081), we are unable to use the one-sample \(t\)-test on our hypothesis. The Hildebrand ratio (\(H = -0.0653\)) suggests the data are from a symmetric distribution. As such, we should use the Wilcoxon procedure.

The \(p\)-value (0.0010) is less than our usual alpha-value of \(\alpha = 0.05\). Thus, we reject the null hypothesis. There is significant evidence that the average difference in reaction times is positive. In fact, we are \(95\%\) confident that the average increase in reaction time of a phone-user is at least 184 milliseconds.

Example #4: Distracted Driving vs. Drunk Driving

It is known that alcohol and cell phone use increase reaction time. However, this raises the question of which is worse. To determine this, I use the previously-collected data and appropriately compared their means.

The change in reaction times between sober and drunk, and between no-phone and yes-phone (in milliseconds) is provided below.

Example #4: Distracted Driving vs. Drunk Driving

We are to compare the means of the two populations. To achieve the best power, we should test symmetry and Normality of both samples.

The phone data suggests non-Normality. It is the most (limiting) of the two subsets. Because of this, we should perform the two-sample Mann-Whitney test:

Example #4: Distracted Driving vs. Drunk Driving

The following load the data and test Normality:

rxJ = c(143, 182, 192, 205, 115, 362, 172, 192, 203, 61) 
rxP = c (88, 233, 240, 207, 188, 194, 184, 198, 193, 208) 
wilcox.test(rxJ, rxP, conf.int=TRUE) 

    Wilcoxon rank sum test with continuity correction

data:  rxJ and rxP
W = 32, p-value = 0.1857
alternative hypothesis: true location shift is not equal to 0
95 percent confidence interval:
 -60.99997  10.99994
sample estimates:
difference in location 
             -15.99997 

Example #4: Distracted Driving vs. Drunk Driving

Conclusion:

We are asked to determine which suppresses reaction times more: being legally drunk or using a cell phone. The Shapiro-Wilk test suggests that the phone data are not from a Normal distribution (\(p\)-value = 0.0081). Thus, we should not use the two-sample \(t\)-test. As such, we should use the Mann-Whitney procedure.

The \(p\)-value (0.1857) is greater than our usual alpha-value of \(\alpha = 0.05\). Thus, we do not know if being drunk or talking on the telephone slows reaction time more. Note that this conclusion may change with additional data, but this is the conclusion based on what we currently know.

Example #5: Big 12 Conference

There is always a lot of discussion about the “home field advantage” in sports. If such an advantage existed, then we would see home teams tend to score more points than the visiting teams.

To check this, I will use the 2015 Big 12 Conference (which does not have twelve teams).

dt = read.csv("http://rfs.kvasaheim.com/data/big12football2015.csv") 
attach(dt) 
names(dt)

 [1] "Team"         "Game"         "opp"          "ptsFor"       "ptsAgainst"  
 [6] "locale"       "conference"   "yardsFor"     "yardsAgainst" "turnBy"      
[11] "turnGot"      "D1for"        "D1against"    "passComp"     "passAtt"     

Example #5: Big 12 Conference

The two variables we will compare are ptsFor and ptsAgainst.

We would like to use the most powerful test. However, those tests have requirements that must be met. Let us check the requirements for this data:

shapiroTest(ptsFor) 

    Shapiro-Wilk normality test

data:  y
W = 0.97232, p-value = 0.0563

shapiroTest(ptsAgainst) 

    Shapiro-Wilk normality test

data:  y
W = 0.96983, p-value = 0.03784

Example #5: Big 12 Conference

The test:

wilcox.test(ptsFor, ptsAgainst, alternative="greater", conf.int=TRUE)

    Wilcoxon rank sum test with continuity correction

data:  ptsFor and ptsAgainst
W = 4573, p-value = 0.01905
alternative hypothesis: true location shift is greater than 0
95 percent confidence interval:
 0.99998     Inf
sample estimates:
difference in location 
              6.000033 

Example #5: Big 12 Conference

Conclusion:

We are asked to determine whether there is evidence of a “home field advantage” using the 2015 Big 12 football season. We will compare the average points scored by the home team to that scored by the away team. The Shapiro-Wilk test suggests that the point scored by the visiting team are not from a Normal distribution (\(p\)-value = 0.0378). Thus, we should not use the two-sample \(t\)-test. We should use the Mann-Whitney procedure.

The \(p\)-value (0.0191) is less than our usual alpha-value of \(\alpha = 0.05\). Thus, we reject the null hypothesis. There is significant evidence that the average points scored by the home team is greater than those scored by the visiting team (a \(95\%\) confidence interval of at least 1 point). Thus, these data provide evidence of a small home-field advantage.

Example #6: Risky Business

One of the more important parameters to estimate in financial mathematics is risk. Typically, “risk” is defined as the variance of the stock prices. Higher variance in prices is evidence of higher risk.

I would like to determine whether International Business Machines (IBM) or Apple (AAPL) has a higher risk.

To test this, we will need to import the prices for the two stocks, adjust their prices to ensure they are on a common scale, and finally compare their variances over time.

The first step is aided by the package that can be added to R. The third step just requires us to divide each by their averages. The last step is just a two-sample variance procedure.

Example #6: Risky Business

This is how to install and load the package:

### Preamble 
install.packages("quantmod")    ## Only once per computer 
library(quantmod)               ## Each time you use it 

Example #6: Risky Business

It is a new package, so working your way around it is slow at first. However, the following lines will download the IBM and Apple data from Yahoo Finance from Jan 3, 2007, until Aug 12, 2022, and store them in two variables.

library(quantmod)

getSymbols("IBM", src="yahoo") 

[1] "IBM"

stock1 = as.numeric(IBM$IBM.Close) 
getSymbols("AAPL", src="yahoo") 

[1] "AAPL"

stock2 = as.numeric(AAPL$AAPL.Close)

Example #6: Risky Business

Next, we have to adjust the prices so that they have the same average. This is done because risk is defined in terms of a set portfolio value of $1. The adjustment also allows us to interpret the changes in values as percent changes.

stock1 = stock1 / mean(stock1) 
stock2 = stock2 / mean(stock2) 

Example #6: Risky Business

Now, we just compare the two variances.

var.test(stock1, stock2)

    F test to compare two variances

data:  stock1 and stock2
F = 0.066105, num df = 4806, denom df = 4806, p-value < 2.2e-16
alternative hypothesis: true ratio of variances is not equal to 1
95 percent confidence interval:
 0.06247002 0.06995051
sample estimates:
ratio of variances 
        0.06610454 

Example #6: Risky Business

Conclusion:

We are asked to determine which of two stocks, IBM and Apple (AAPL), is more risky. To do this, we will use the F test.

According to this test, there is a significant difference in variances between the two stocks (p-value \(\ll\) 0.0001), with Apple being the riskier stock. A \(95\%\) confidence interval for the ratio of Apple to IBM variance is from 26.87 to 30.46.

Example #7: Knox Students

According to SAMHSA’s 2004–2005 National Surveys on Drug Use and Health, males had higher rates than females for all measures of alcohol drinking in the past month. In fact, they found males were twice as likely as females to have met the criteria for alcohol dependence or abuse in the past year (10.5% vs. 5.1%).

I would like to determine if students at Knox College are typical in both proportions by gender and differences by gender.

To do this, I survey 50 students who identify as male and 25 who identify as female. For each student surveyed, I performed a typical “Alcohol Abuse Survey” (AAS). In my sample, 24 males and six females tested positive on the AAS.

Example #7: Knox Students

This problem compares two population proportions. Thus, we have two options: bootstrapping and the proportions test.

Note: While the sample size is “sufficient” to use the proportions test, let us use both to get a better feel for the data and what it is telling us.

Example #7: Knox Students

There are several hypotheses being asked. The first is that the male proportion testing positive on the AAS is 10.5%.

genM = rbinom(1e6, size=50, prob=0.105) ## Based on H0 
mean(genM >= 24) 

[1] 0

genM = rbinom(1e6, size=50, prob=24/50) ## Based on data 
quantile(genM, c(0.025, 0.975))/50 

 2.5% 97.5% 
 0.34  0.62 

Example #7: Knox Students

The \(p\)-value from the simulation test is much, much less than our usual alpha-value of \(\alpha = 0.05\). As such, we conclude that the rate of a positive AAS assessment for males at Knox College is larger than the national average.

We are \(95\%\) confident that the proportion of males at Knox who test positive on AAS are from 34% to 62%.

Example #7: Knox Students

The second hypothesis is that the female proportion testing positive on the AAS is 5.1%.

genF = rbinom(1e6, size=25, prob=0.051) ## Based on H0 
mean(genF >= 24) 

[1] 0

genF = rbinom(1e6, size=50, prob=6/25) ## Based on data 
quantile(genF, c(0.025, 0.975))/25

 2.5% 97.5% 
 0.24  0.72 

Example #7: Knox Students

The \(p\)-value from the simulation test (0.001325) is less than our usual alpha-value of \(\alpha = 0.05\). As such, we conclude that the rate of a positive AAS assessment for females at Knox College is larger than the national average.

We are \(95\%\) confident that the proportion of females at Knox who test positive on AAS are from 8% to 40%. The accepted proportion of 5.1% is not in this interval.

Example #7: Knox Students

Alternatively, you can use the built-in Binomial test:

binom.test(24, 50, p=0.105) 

    Exact binomial test

data:  24 and 50
number of successes = 24, number of trials = 50, p-value = 2.492e-11
alternative hypothesis: true probability of success is not equal to 0.105
95 percent confidence interval:
 0.3366051 0.6258481
sample estimates:
probability of success 
                  0.48 

Example #7: Knox Students

Alternatively, you can use the built-in Binomial test:

binom.test(6, 25, p=0.051) 

    Exact binomial test

data:  6 and 25
number of successes = 6, number of trials = 25, p-value = 0.001343
alternative hypothesis: true probability of success is not equal to 0.051
95 percent confidence interval:
 0.09356444 0.45128802
sample estimates:
probability of success 
                  0.24 

Example #7: Knox Students

These lines provide the same substantive results, although the confidence intervals will be slightly different:

We are \(95\%\) confident that the proportion of males at Knox College who are AAS-positive is between 33.7% and 62.6%; of females, between 9.4% and 45.1%.

Example #7: Knox Students

The third embedded hypothesis is that males are twice as likely as females to be AAS-positive.

One way of testing this is simply to double the female successes in the sample and/or doubling the female success rate in the population. This is not the “optimal” method, but it usually does a sufficient job. A second method is to halve the number of “successes” in the male-identifying population.

What I tend to do is both of these. If the two methods tell the same story, then I am confident in my conclusion. If, on the other hand, they give contradicting results, then I report both test results and conclude that, if there is a detected effect, then it is quite minor.

Example #7: Knox Students

Here is the code and the output from the first method.

prop.test(c(24, 6*2), c(50, 25*2))

    2-sample test for equality of proportions with continuity correction

data:  c(24, 6 * 2) out of c(50, 25 * 2)
X-squared = 5.2517, df = 1, p-value = 0.02192
alternative hypothesis: two.sided
95 percent confidence interval:
 0.03781821 0.44218179
sample estimates:
prop 1 prop 2 
  0.48   0.24 

I leave it as an exercise to show that the second method gives similar results.

Example #7: Knox Students

Conclusion:

According to SAMHSA, the proportion of males who test positive on the AAS assessment is about double that of females. The male proportion is 10.5%; the female, 5.1%.

For Knox College, the proportion of males testing positive on the AAS Assessment is significantly higher, from 34% to 62%. Similarly, the proportion of females testing positive is also significantly higher, from 8% to 40%.

Example #7: Knox Students

The SAMHSA noted that the proportion of males testing positive is about twice the proportion that females tested positive. In this, Knox College students seem to be typical. There is no evidence that the ratio is anything other than what SAMHSA found in the population at large.

All of this suggests that the relationship between male and female problematic drinking holds at Knox College, even if the actual rates of AAS-positive findings are higher.