In this section, you’ll learn what a hypothesis test is, when to use it, and how to calculate it.

A hypothesis is a formalized guess about the value of a random variable. A hypothesis test is a statistical tool that allows you to make an inference about a population from a sample drawn from that population.

Hypothesis tests are organized into a null hypothesis ($H_0$) and an alternative hypothesis ($H_1$).

The goal of hypothesis testing is to determine whether there is enough evidence in the sample data to reject the null hypothesis in favor of the alternative hypothesis.

Use Case

For example, you might hypothesize that a new drug will reduce blood pressure in a population of patients with high blood pressure. You would then split your patients into two groups, giving one of them the drug and one of them a placebo and measuring their blood pressure.

The hypotheses might look like this:

$H_0$: The drug had no impact on patients’ high blood pressure.

$H_A$: The drug did have an impact on patients’ high blood pressure.

You would then calculate the test statistic to determine if the data obtained from the sample is enough to reject the null in favor of the alternative. Different hypothesis tests can have different test statistics, it really depends on what you’re comparing.

The Null Hypothesis is No Effect

Note that the null and alternative hypotheses are set up in this way, with the hypothesis you’re interested in testing set as the alternative hypothesis.

That is, your guess of the value of the variable is the alternative hypothesis. The null hypothesis is often that there was no effect, no difference, or no change in the population.

The alternative hypothesis is the effect, difference, or change you’re trying to support, and you collect data to see if you can reject the null in favor of the alternative.

The Null and Alternative Hypothesis

Together, the null hypothesis $H_0$ and the alternative hypothesis $H_A$ represent all possible outcomes being tested.

A common hypothesis test is to check if some statistic $\theta$ is different from a given value, denoted $\theta_0$.

For example, you might want to check if the mean of your sample ($\mu$) is different from a set number, such as 5. Here $\mu$ is called the test statistic. The hypotheses look like this:

$H_0: \mu = 5$

$H_A: \mu \neq 5$

Or you may want to see if the mean $\mu$ is greater than 5.

$H_0: \mu \leq 5$

$H_A: \mu > 5$

Notice that your guess is in the alternative hypothesis, not the null! Let’s look at why.

Why $H_0$ Is No Effect

In statistics, you can never be absolutely sure of an outcome. So you would never say that you accept a hypothesis. Instead, you say that there’s enough evidence to reject a hypothesis.

Your data can’t prove a hypothesis. However, you can show how unlikely it would be to get the data you collected if your hypothesis were false.

For example, if out of 20 people you surveyed, 13 said that they like coffee more than tea, you can’t say this definitively proves that more people like coffee than tea. But, you can calculate how likely it would be to get this result if, in reality, only 50% of people prefer coffee to tea.

Statistical Significance of a Hypothesis Test

A hypothesis test has a power or significance level of the test, denoted by $\alpha$, and takes values in (0,1). The significance level is also sometimes called a p-value.

The significance level is the probability of rejecting $H_0$ when $H_0$ is actually true.

The significance level is chosen by you the researcher before carrying out the hypothesis test. Commonly used values are 0.05, 0.01, or 0.10. The lower the significance level, the harder it is to reject the null hypothesis. But the lower the significance level is, the more that a result that rejects the null is statistically significant.

Using a Significance Level

After a significance level is chosen, the next step is to calculate the test statistic. The test statistic depends on the kind of test being done: for example, a t-test, a Chi-square test, or an F-test.

The test statistic computed from the sample data is then compared to a threshold of that same statistic that corresponds to the previously chosen p-value or significance level $\alpha$. If the test statistic is below the threshold, then you can reject the null hypothesis at the chosen significance level, for example, 0.05 or 0.01.

Alternatively, you can compute a p-value directly. If the p-value computed from the sample data is below your previously chosen significance level, you reject the null hypothesis in favor of the alternative.

When you reject the null in favor of the alternative, it means that the data you collected is unlikely to have looked that way if the null hypothesis were true. This is what it means to have a statistically significant result.

Type I and Type II Error

The type of error where you reject $H_0$ when it is true is called a type I error (read: “type one error”) or a false positive.

Similarly, the type of error where you fail to reject $H_0$ when it is false is called a type II error (read: “type two error”) or a false negative.

One-Sided and Two-Sided Tests

The sidedness of a test is the number of scenarios where $H_0$ is false. A test can be one-sided or two-sided.

A two-sided test:

$H_0: \theta = 0$

$H_1: \theta ≠ 0$

$H_0$ can be rejected by either concluding that $\theta > 0$ or $\theta < 0$, making it a two-sided test.

A one-sided test:

$H_0: \theta \leq 0$

$H_1: \theta > 0$

The only way to reject $H_0$ is to conclude $\theta > 0$, so this is a one-sided test.

Practically, the only difference between one-sided and two-sided tests is that two-sided tests need their value of $\alpha$ adjusted to $\alpha/2$ to take into account the multiple ways to reject $H_0$.

How to Carry Out a Hypothesis Test

The following reviews the steps you’d carry out when using a hypothesis test.

1. Formulate your hypotheses
   • Null Hypothesis $H_0$: The default hypothesis, which often states that there is no effect, no difference, or no change in the population.
   • Alternative Hypothesis $H_A$: The opposite of the null hypothesis, suggesting an effect, difference, or change. It is what you are trying to support with the data collected.
2. Choose a Significance Level α
   • The significance level, denoted by α, is the probability of rejecting the null hypothesis when it is actually true. Commonly used values are 0.05, 0.01, or 0.10.
   • The smaller the significance level, the harder it will be to reject the null hypothesis.
   • Conversely, the smaller the significance level is, the more confidence you have in the alternative hypothesis if you end up rejecting the null. This is statistical significance.
3. Collect and Analyze Data
   • Collect a sample of data and calculate the test statistic. This could involve calculating means, standard deviations, or conducting tests like t-tests or chi-square tests.
4. Make a Decision
   • Compare the test statistic with the significance level. If the p-value is less than or equal to α, you reject the null hypothesis. If the p-value is greater than α, you fail to reject the null hypothesis.
5. Draw Conclusions
   • If you reject the null hypothesis, you conclude that there is evidence to support the claim of the alternative hypothesis.
   • If you fail to reject the null hypothesis, you do not have enough evidence to support the alternative hypothesis.
   • You never say that you accept the alternative hypothesis, only that you reject the null hypothesis or fail to reject the null.