Analysis of Variance (ANOVA)

“Analysis of variance (ANOVA) is a collection of statistical models and their associated estimation procedures (such as the “variation” among and between groups) used to analyze the differences among means.”

Analysis of variance (ANOVA) is the name given to a statistical procedure used to test whether two population means are different from one another. If you have two groups that may be different in some way, you can first calculate the F statistic, and then use ANOVA to see if the means are truly different from one another. ANOVA is typically used as one-way or two-way.

ANOVA is an extension of the concepts of a Z-test or t-test, with a key difference. Unlike with a t-test, which is used to test the means of two groups, ANOVA will test the means of three or more groups.

Cheat sheet

• Description: Tests if the variance of three normally-distributed samples, $\vec{x}_1$, $\vec{x}_2$, $\vec{x}_3$, are equal.

• Sum of Squares (SS) or SS Total (SST): Measures the total variation in the data.

  $$\text{SST} = \sum (X_i - \bar{X})^2$$

  where:

  • $X_i$ is each individual data point.

  • $\bar{X}$ is the overall mean.

  • SS Between Groups (SSB): Measures the variation between group means.

    $$\text{SSB} = \sum n_j (\bar{X}_j - \bar{X})^2$$

    where:

    • $n_j$ is the number of observations in group $j$.
    • $\bar{X}_j$ is the mean of group $j$.

  • SS Within Groups (SSW): Measures the variation within each group.

    $$\text{SSW} = \sum \sum (X_{ij} - \bar{X}_j)^2$$

    where:

    • $X_{ij}$ is the $i$th observation in group $j$.

• Degrees of Freedom (df):

  • df Total (dfT): Total number of observations minus 1.

    $$\text{dfT} = N - 1$$

  • df Between Groups (dfB): Number of groups minus 1.

    $$\text{dfB} = k - 1$$

  • df Within Groups (dfW): Total number of observations minus the number of groups.

    $$\text{dfW} = N - k$$

• Mean Squares (MS):

  • MS Between Groups (MSB): SSB divided by dfB.

    $$\text{MSB} = \frac{\text{SSB}}{\text{dfB}}$$

  • MS Within Groups (MSW): SSW divided by dfW.

    $$\text{MSW} = \frac{\text{SSW}}{\text{dfW}}$$

• F-Statistic:

  • The test statistic for ANOVA, calculated as the ratio of MSB to MSW.

    $$F = \frac{\text{MSB}}{\text{MSW}}$$

Description

ANOVA is a statistical test that is used to evaluate the variation between three or more groups.

There are several types of ANOVA that can be used based on what you want to test:

• One-way ANOVA: Used when there’s only one independent variable (factor) with multiple levels or groups. For example, comparing test scores of students from different schools.
• Two-way ANOVA: Used when there are two independent variables. This allows you to examine the individual effects of each variable, as well as their interaction effect. For instance, analyzing the effect of both gender and age on income.
• Repeated Measures ANOVA: Used when the same individuals are measured multiple times under different conditions. This could involve testing the same group of participants before and after a treatment.

Assumptions: ANOVA makes certain assumptions about the data, such as normality of the data within each group and the homogeneity of variances across groups. It’s important to check these assumptions before using ANOVA.

Example:

Suppose we have three groups of basketball teams (Group A, Group B, and Group C) who scored the following amount of points. We want to see if there is a significant difference in their average scores.

Steps:

1. Calculate Group Means and Overall Mean:

   • Mean of Group A ($\bar{X}_A$) = $\frac{85 + 79 + 90 + 82}{4} = 84$
   • Mean of Group B ($\bar{X}_B$) = $\frac{92 + 88 + 95 + 90}{4} = 91.25$
   • Mean of Group C ($\bar{X}_C$) = $\frac{78 + 72 + 80 + 75}{4} = 76.25$
   • Overall Mean ($\bar{X}$) = $\frac{84 + 91.25 + 76.25}{3} = 83.83$

2. Calculate Sum of Squares (SS):

   • SS Total (SST): The sum of squared deviations of each score from the overall mean.

     $$\text{SST} = (85 - 83.83)^2 + (79 - 83.83)^2 + \dots + (75 - 83.83)^2 = 169.67$$

   • SS Between Groups (SSB): The sum of squared deviations of each group mean from the overall mean, weighted by the number of scores in each group.

     $$\text{SSB} = 4 \times (84 - 83.83)^2 + 4 \times (91.25 - 83.83)^2 + 4 \times (76.25 - 83.83)^2 = 152.67$$

   • SS Within Groups (SSW): The sum of squared deviations of each score from its group mean.

     $$\text{SSW} = (85 - 84)^2 + (79 - 84)^2 + \dots + (75 - 76.25)^2 = 17$$

3. Calculate Degrees of Freedom (df):

   • df Total (dfT) = $N - 1 = 12 - 1 = 11$

   • df Between Groups (dfB) = $k - 1 = 3 - 1 = 2$

   • df Within Groups (dfW) = $N - k = 12 - 3 = 9$

     where $N$ is the total number of observations and $k$ is the number of groups.

4. Calculate Mean Squares (MS):

   • MS Between Groups (MSB) = $\frac{\text{SSB}}{\text{dfB}} = \frac{152.67}{2} = 76.33$
   • MS Within Groups (MSW) = $\frac{\text{SSW}}{\text{dfW}} = \frac{17}{9} = 1.89$

5. Calculate F-Statistic:

   • $F = \frac{\text{MSB}}{\text{MSW}} = \frac{76.33}{1.89} = 40.39$

6. Compare F to Critical Value:

   • Look up the critical F-value from an F-distribution table using dfB and dfW at a chosen significance level (e.g., $\alpha = 0.05$). If the calculated F-statistic is greater than the critical value, you reject the null hypothesis and conclude that there are significant differences among the group means.

Important Note:

This is a simplified example with a small dataset. In practice, you would typically use statistical software for ANOVA calculations, especially with larger datasets. However, understanding the manual calculation process helps you grasp the underlying concepts of ANOVA.

This calculation can also be easily implemented in Python.