Lecture #21: The Analysis of Variance (ANOVA)

Example: GPA and Majors

I would like to determine if the average GPA is the same for the four types of majors: MNS, HSS, HUM, and ART. To test this, I asked 200 full-time students at Knox College, 50 of each major type, and asked two questions:

• What is your major type?
• What is your GPA?

Example: GPA and Majors

The Theory of ANOVA

There are a few equivalent ways of looking at this question:

• Do the means in each group significantly differ?
• Are the group and the GPA independent?
• Does including the group identifier improve our ability to estimate?

The Theory of ANOVA

Think of this as the idea behind the Analysis of Variance procedure.

• Measure the variance of the original data.
• Measure the variance unexplained in the model.
• Calculate the ratio of the explained variance to the unexplained.
• This last ratio is the test statistic.

The ANOVA Table

With that background, let us calculate the test statistic (and \(p\)-value) using the following ANOVA Table:

The \(SS\) Column

The column marked “\(SS\)” contains the “sum of squares” for the three sources. The sum of squared is just the sum of the deviation between the observation and the mean. As such:

\[ \begin{aligned} SS_{Model} &= \sum_i \sum_j \left( \overline{y}_j - \overline{\overline{y}} \right)^2 \\ SS_{Error} &= \sum_i \sum_j \left( y_{ij} - \overline{y}_j \right)^2 \\ SS_{Total} &= \sum_i \sum_j \left( y_{ij} - \overline{\overline{y}} \right)^2 \end{aligned} \]

\(i\) represents a value within a group, and \(j\) represents a group. \(\overline{y}_j\) denotes the average of the measurements in group \(j\), and \(\overline{\overline{y}}\) is the average of all measurements (the grand mean).

The \(SS\) Column

Note: \(SS_M + SS_E = SS_T\). This is an interesting result. Check the formulas for \(SS_M\) and \(SS_E\) and marvel at this fact. For statisticians, this means that the Model and what remains (the Error) are independent.

The \(df\) Column

The column marked “\(df\)” contains the “degrees of freedom” for the three sources. What are degrees of freedom? They are parameters that reflect the amount of information contributed by each source.

\[ \begin{aligned} df_{Model} &= k - 1 \\ df_{Error} &= k(n-1) = N - k \\ df_{Total} &= N - 1 \end{aligned} \]

\(k\) represents the number of groups, \(n\) represents the sample size within each group and \(N\) represents the total sample size.

The \(df\) Column

Calculating the number of degrees of freedom is rather easy. Here are the results:

Note: \(df_M + df_E = df_T\).

The \(MS\) Column

The column marked “\(MS\)” contains the “mean squares” for the three sources. These are the estimates of the individual variances. They are the \(SS\) divided by the \(df\) for each source. Recall our Chapter 3 definition of sample variance. It is just the sum of squares divided by the degrees of freedom, \(n - 1\).

\[ \begin{aligned} MS_{Model} &= \frac{SS_{Model}}{df_{Model}} \\ MS_{Error} &= \frac{SS_{Error}}{df_{Error}} \end{aligned} \]

The \(MS\) Column

Note: We could also calculate \(MS_{Total}\). It is not used in ANOVA, so we do not. The formula is:

\[ \begin{aligned} MS_{Total} = \frac{1}{N - 1} \sum_i \sum_j (y_{ij} - \overline{\overline{y}})^2 \end{aligned} \]

This is just the sample variance of the measurements.

The \(MS\) Column

Calculating the mean squares values is rather easy. Here are the results:

Note: Note: \(MS_M + MS_E \ne MS_T\). That is, the total variance is not partitioned between the two sources.

The \(F\) Column

The column marked “\(F\)” contains the value of the “\(F\)-statistic” for the model.

\[ \begin{aligned} F = \frac{MS_{Model}}{MS_{Error}} \end{aligned} \]

The \(F\) Column

As you should guess, the distribution of the \(F\) statistic is the \(F\) distribution — officially, it is “Snedecor’s \(F\)” distribution. This distribution is named after George W. Snedecor, who used Fisher’s statistical definition to calculate the probability density function. By the way, Snedecor also founded the first Statistics Department in the United States at Iowa State University in 1933.

The \(F\) Column

Calculating the \(F\) statistic is rather easy. Here are the results:

The \(p\) Column

The column marked “\(p\)” contains the “\(p\)-value” for the model. It is interpreted as usual.

\[ \begin{aligned} p = \mathbb{P}(F \le f) = 1 - \mathbb{P}(F > f) \end{aligned} \]

Here, \(f\) is the value of the test statistic you calculated above. As with all \(p\)-values, it is a measure of how far the data are from the null hypothesis. Compare it to your selected value of \(\alpha\). If the \(p\)-value is less than \(\alpha\), then you reject the null hypothesis.

The \(p\) Column

So, what is the null hypothesis? These three are equivalent:

• \(H_0 : \mu_1 = \mu_2 = \dots = \mu_n\). (All population means are the same.)
• \(H_0 :\) The numeric variable is independent of the categorical variable.
• \(H_0 :\) The model does not significantly improve our predictive ability.

The \(p\) Column

Here is the distribution of the test statistic and what we observed. Note that the \(p\)-value – the area to the right of the observed test statistic value – is extremely small. Thus, we would expect to reject the null hypothesis.

The \(p\) Column

The Brief Conclusion

Brief Conclusion: Because the \(p\)-value of 0.00115 is less than out usual \(\alpha = 0.05\), we reject the null hypothesis. At least one of the population means differ. The two variables are not independent. The model helps in our prediction accuracy.

Example #1: Rice Yields

Does rice variety influence average yield amongst these four varieties?

One of the typical examples for introducing ANOVA concerns comparing rice yields across four different varieties. Here are the raw data:

Example #1: Rice Yields

Example #1: Rice Yields

Here is the blank ANOVA table. Let’s perform the calculations by hand.

Example #1: Rice Yields

The column marked “\(SS\)” contains the “sum of squares” for the three sources. The sum of squared is just the sum of the deviation between the observation and the mean. As such:

\[ \begin{aligned} SS_{Model} &= \sum_i \sum_j \left( \overline{y}_j - \overline{\overline{y}} \right)^2 \\ SS_{Error} &= \sum_i \sum_j \left( y_{ij} - \overline{y}_j \right)^2 \\ SS_{Total} &= \sum_i \sum_j \left( y_{ij} - \overline{\overline{y}} \right)^2 \end{aligned} \]

\(i\) represents a value within a group, and \(j\) represents a group. \(\overline{y}_j\) denotes the average of the measurements in group \(j\), and \(\overline{\overline{y}}\) is the average of all measurements (the grand mean).

Example #1: Rice Yields

From the data:

\[ \begin{gathered} \overline{y}_1 = 984.50 \\ \overline{y}_2 = 928.25 \\ \overline{y}_3 = 938.50 \\ \overline{y}_4 = 1156.50 \\ \\ \overline{\overline{y}} = 991.9375 \end{gathered} \]

Example #1: Rice Yields

\[ \begin{aligned} SS_{Model} &= \sum_i \sum_j (\overline{y}_j - \overline{\overline{y}})^2 \\ &= (\overline{y}_1 - \overline{\overline{y}})^2 + (\overline{y}_2 - \overline{\overline{y}})^2 + (\overline{y}_3 - \overline{\overline{y}})^2 + (\overline{y}_4 - \overline{\overline{y}})^2 \\ &+ (\overline{y}_1 - \overline{\overline{y}})^2 + (\overline{y}_2 - \overline{\overline{y}})^2 + (\overline{y}_3 - \overline{\overline{y}})^2 + (\overline{y}_4 - \overline{\overline{y}})^2 \\ &+ (\overline{y}_1 - \overline{\overline{y}})^2 + (\overline{y}_2 - \overline{\overline{y}})^2 + (\overline{y}_3 - \overline{\overline{y}})^2 + (\overline{y}_4 - \overline{\overline{y}})^2 \\ &+ (\overline{y}_1 - \overline{\overline{y}})^2 + (\overline{y}_2 - \overline{\overline{y}})^2 + (\overline{y}_3 - \overline{\overline{y}})^2 + (\overline{y}_4 - \overline{\overline{y}})^2 \end{aligned} \]

Example #1: Rice Yields

\[ \begin{aligned} SS_{Model} &= (984.50 - 991.9375)^2 + (928.25 - 991.9375)^2 \\ &+ (938.50 - 991.9375)^2 + (1156.50 - 991.9375)^2 \\ &+ (984.50 - 991.9375)^2 + (928.25 - 991.9375)^2 \\ &+ (938.50 - 991.9375)^2 + (1156.50 - 991.9375)^2 \\ &+ (984.50 - 991.9375)^2 + (928.25 - 991.9375)^2 \\ &+ (938.50 - 991.9375)^2 + (1156.50 - 991.9375)^2 \\ &+ (984.50 - 991.9375)^2 + (928.25 - 991.9375)^2 \\ &+ (938.50 - 991.9375)^2 + (1156.50 - 991.9375)^2 \end{aligned} \]

\[ \begin{aligned} SS_{Model} &= 89931 \end{aligned} \]

Example #1: Rice Yields

This should illustrate why calculation by hand is no longer realistic. Here are the results:

Example #1: Rice Yields

The column marked “\(df\)” contains the “degrees of freedom” for the three sources. What are degrees of freedom? They are parameters that reflect the amount of information contributed by each source.

\[ \begin{aligned} df_{Model} &= k - 1 = 4 - 1 = 3 \\ df_{Error} &= k(n-1) = 4(4-1) = 12\\ df_{Total} &= N - 1 = 16 - 1 = 15 \end{aligned} \]

\(k\) represents the number of groups, \(n\) represents the sample size within each group and \(N\) represents the total sample size.

Example #1: Rice Yields

Here are the results:

The \(MS\) Column

The column marked “\(MS\)” contains the “mean squares” for the three sources. These are the estimates of the individual variances. They are the \(SS\) divided by the \(df\) for each source. \[ \begin{aligned} MS_{Model} &= \frac{SS_{Model}}{df_{Model}} = \frac{89931}{3} = 29977 \end{aligned} \]

\[ \begin{aligned} MS_{Error} &= \frac{SS_{Error}}{df_{Error}} = \frac{49876}{12} = 24156 \end{aligned} \]

Example #1: Rice Yields

Here are the results:

Example #1: Rice Yields

The column marked “\(F\)” contains the value of the “\(F\)-statistic” for the model.

\[ \begin{aligned} F = \frac{MS_{Model}}{MS_{Error}} = \frac{29977}{4156} = 7.212 \end{aligned} \]

Example #1: Rice Yields

Here are the results:

Example #1: Rice Yields

The column marked “\(p\)” contains the “\(p\)-value” for the model. It is interpreted as usual.

\[ \begin{aligned} p = \mathbb{P}(F \le f) = 1 - \mathbb{P}(F \le f) \end{aligned} \]

Here, \(f\) is the value of the test statistic you calculated above. As with all \(p\)-values, it is a measure of how far the data are from the null hypothesis. Compare it to your selected value of \(\alpha\). If the \(p\)-value is less than \(\alpha\), then you reject the null hypothesis.

Example #1: Rice Yields

Here is the distribution of the test statistic and what we observed. Note that the \(p\)-value – the area to the right of the observed test statistic value – is extremely small. Thus, we would expect to reject the null hypothesis.

Example #1: Rice Yields

Here are the final results:

Brief Conclusion: Because the \(p\)-value of 0.00503 is less than our usual \(\alpha = 0.05\), we reject the null hypothesis. At least one of the population means differ. The two variables are not independent. The model helps in our prediction accuracy.

Example #1: Rice Yields (with R)

Does rice variety influence average yield amongst these four varieties?

The data can be downloaded from the expected place in the usual manner:

dt = read.csv("http://rfs.kvasaheim.com/data/rice.csv")
summary(dt)

   variety              yield       
 Length:16          Min.   : 840.0  
 Class :character   1st Qu.: 934.8  
 Mode  :character   Median : 969.5  
                    Mean   : 991.9  
                    3rd Qu.:1031.2  
                    Max.   :1191.0  

attach(dt)

Example #1: Rice Yields (with R)

Example #1: Rice Yields (with R)

Here is the code to run the ANOVA in R:

ricemod = aov(yield ~ variety) 
summary(ricemod)

            Df Sum Sq Mean Sq F value  Pr(>F)   
variety      3  89931   29977   7.212 0.00503 **
Residuals   12  49876    4156                   
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Example #1: Rice Yields (with R)

Brief Conclusion:

Because the \(p\)-value of 0.00503 is less than our usual \(\alpha = 0.05\), we reject the null hypothesis. The average yield per acre for the four varieties is not the same. The yield and variety variables are dependent. Our ability to predict the yield for a plot depends on knowing the rice yield planted.

Example #2: Fisher 38

Ronald Fisher introduced the ANOVA procedure to the world in 1925 in his book Statistical Methods for Research Workers. In that book, he described the following experiment:

Collect a sample of pond water. Divide that water amongst four different beakers. Separate the beakers to ensure that there is no cross-contamination. For each beaker, take four samples and count and record the number of amoeba present.

The results of this experiment, he provided in Table 3.8 in his book.

They are also available as the fisher38 datafile.

Example #2: Fisher 38

Example #2: Fisher 38

Here is the entire script I used to determine if the average number of amoeba differed among the four beakers.

dt = read.csv("http://rfs.kvasaheim.com/data/fisher38.csv")
attach(dt)

ickymod = aov(colonies ~ sample) 
summary(ickymod)

            Df Sum Sq Mean Sq F value Pr(>F)
sample       3     95   31.65   0.525  0.669
Residuals   24   1446   60.25               

Example #2: Fisher 38

Here is the distribution of the test statistic and what we observed. Note that the \(p\)-value – the area to the right of the observed test statistic value – is not that small. Thus, we would not expect to reject the null hypothesis.

Example #2: Fisher 38

Brief Conclusion:

Because the \(p\)-value of \(0.6690\) is greater than our usual \(\alpha = 0.05\), we cannot reject the null hypothesis. We did not detect a difference in the average number of amoeba in the samples across the four beakers.