Back to Statistics & AB Testing Interview
Statistics & AB Testing Interview

Statistics & AB Testing Interview

38 of 77 Completed

Two-sample Tests

The tests we examined in the previous section were one-sample tests, meaning they tested a statistic of a sample against a single value. While such tests do have broad applications, they are limited because we might not know what value we want to set for μ0\mu_0 at the offset. For example, in A/B Tests, we want to compare the means two samples against each other.

You might say that we could run a one-sample test and set μ0\mu_0 equal to the mean of one of the samples, but this doesn’t work because adding more samples increases the variance of the sample. You can see this if you model the process abstractly using random variables. If X1N(μ1,σ12)X_1\sim\mathcal{N}(\mu_1,\sigma_1^2) and X2N(μ2,σ22)X_2\sim\mathcal{N}(\mu_2,\sigma_2^2) then Var[X1+X2]=σ12+σ22\mathsf{Var}[X_1+X_2]=\sigma_1^2+\sigma_2^2.

Thus, we need to take into account the increased variance home how. The way in which we do that depends on 1. If the sample sizes are equal 2. If the samples have equal (population) variances 3. If the samples are independent or not

This gives us four different versions of the two-sample tt-test: 1. Independent samples with equal variance and sample size 2. Independent samples with equal variance, but different sample size 3. Independent samples with difference variance 4. Dependent samples

Independent Two-sample tt-test With Equal Variances and Sample Sizes

Cheat sheet

  • Description: Tests if the mean of one sample (μ1\mu_1) is different/less than/more than the mean of another sample (μ2\mu_2) when the samples have equal sample size, variance, and are independent

  • Statistic: μ1μ2\mu_1-\mu_2 (mean)

  • Distribution: T(2n2)\mathcal{T}(2n-2) (tt)

  • Sidedness: Either

  • Null Hypothesis: H0:μ1=μ2H_0: \mu_1 = \mu_2 (two-sided), μ1μ2,μ1μ2\mu_1 \geq\mu_2,\mu_1\leq\mu_2 (one-sided)

  • Alternative Hypothesis: Ha:μ1μ2H_a: \mu_1 \neq \mu_2 (two-sided), μ1<μ2,μ1>μ2\mu_1 \lt\mu_2,\mu_1\gt\mu_2 (one-sided)

  • Test Statistic: τ=μ^1μ^2s12+s22n \tau=\frac{\hat{\mu}_1-\hat{\mu}_2}{\sqrt{\frac{s_1^2+s_2^2}{n}}}

    Description

    Let’s go over what the title of this test really means

Independent: This means that both of the samples have no relation to each other. In medical terms, there is no “treatment” between the samples, they just represent different populations.

Same Sample Size: Both samples have the same number of data points, duh.

Same Variance: This condition has a misleading name. It’s generally impossible to determine if two populations actually have the exact same variance. Because of this, “same variance” is defined as such: both samples have variances that are at least half and at most half of the other. Mathematically, if you use the sample variances to estimate the population variances, this can be expressed as:

12s12s222 \frac{1}{2}\leq\frac{s_1^2}{s_2^2}\leq2 This test is useful if you have two different classes or categories of the object you’re studying. This could be something like the total time spent watching content of viewers in two different geographical regions.

Independent Two-sample tt-test With Equal Variances and Unequal Sample Sizes

Cheat sheet

  • Description: Tests if the mean of one sample (μ1\mu_1) with n1n_1 data points is different/less than/more than the mean of another sample (μ2\mu_2) with n2n_2 data points when the samples have equal variances and are independent

  • Statistic: μ1μ2\mu_1-\mu_2 (mean)

  • Distribution: T(n1+n22)\mathcal{T}(n_1+n_2-2) (tt)

  • Sidedness: Either

  • Null Hypothesis: H0:μ1=μ2H_0: \mu_1 = \mu_2 (two-sided), μ1μ2,μ1μ2\mu_1 \geq\mu_2,\mu_1\leq\mu_2 (one-sided)

  • Alternative Hypothesis: Ha:μ1μ2H_a: \mu_1 \neq \mu_2 (two-sided), μ1<μ2,μ1>μ2\mu_1 \lt\mu_2,\mu_1\gt\mu_2 (one-sided)

  • Test Statistic: τ=μ^1μ^2(n11)s12+(n21)s22n1+n22n1+n2n1n2 \tau=\frac{\hat{\mu}_1-\hat{\mu}_2}{\sqrt{\frac{\left(n_1-1\right)s_{1}^2+\left(n_2-1\right)s_{2}^2}{n_1+n_2-2}}\cdot\sqrt{\frac{n_1+n_2}{n_1n_2}}}

    Description

    While this test is more or less the same from a mathematical perspective from the previous test, the complexity of the denominator might cause some confusion.

The denominator is defined in such a way so that it remains a unbias estimator for σ\sigma, the population variance common to both samples, no matter the actual sample sizes n1n_1 and n2n_2. In fact, if you set n1=n2n_1=n_2 and a lot of algebra, you’ll see that τ\tau is the same as the previous case.

Independent Two-sample tt-test

  • Description: Tests if the mean of one sample (μ1\mu_1) with n1n_1 data points is different/less than/more than the mean of another sample (μ2\mu_2) with n2n_2 data points when the samples have unequal variances and are independent
  • Statistic: μ1μ2\mu_1-\mu_2 (mean)
  • Distribution: T(df)\mathcal{T}(df) (tt) where df=(s12n1+s22n2)2(s12/n1)2n11+(s22/n2)2n21 df = \frac{\left(\frac{s_1^2}{n_1} + \frac{s_2^2}{n_2}\right)^2}{\frac{\left(s_1^2/n_1\right)^2}{n_1-1} + \frac{\left(s_2^2/n_2\right)^2}{n_2-1}}
  • Sidedness: Either
  • Null Hypothesis: H0:μ1=μ2H_0: \mu_1 = \mu_2 (two-sided), μ1μ2,μ1μ2\mu_1 \geq\mu_2,\mu_1\leq\mu_2 (one-sided)
  • Alternative Hypothesis: Ha:μ1μ2H_a: \mu_1 \neq \mu_2 (two-sided), μ1<μ2,μ1>μ2\mu_1 \lt\mu_2,\mu_1\gt\mu_2 (one-sided)
  • Test Statistic: τ=μ^1μ^2s12n1+s22n2 \tau=\frac{\hat{\mu}_1-\hat{\mu}_2}{\sqrt{\frac{s_1^2}{n_1}+\frac{s_2^2}{n_2}}}

Description

The test is used in the case where the variance of one sample is greater more than double of the variance of the other sample (s12>2s22s_1^2\gt2s_2^2 or s22>2s12s_2^2\gt2s_1^2).

The complicated form of the degrees of freedom is because it is actually an approximation of the true degrees of freedom of this test. The reasoning behind this is well beyond the scope of this course, and data science in general, but those who are curious can look up the “Behrens–Fisher problem” for details.

Paired (Dependent) tt-test

Cheat Sheet

  • Description: Tests if the difference of the same sample before and after some treatment is equal/less than/more than some number μ0\mu_0

  • Statistic: μΔ\mu_\Delta (mean of differences between pairs)

  • Distribution: T(n1)\mathcal{T}(n-1) (tt)

  • Sidedness: Either

  • Null Hypothesis: H0:μΔ=μ0H_0: \mu_\Delta = \mu_0 (two-sided), μΔμ0,μΔμ0\mu_\Delta \geq\mu_0,\mu_\Delta\leq\mu_0 (one-sided)

  • Alternative Hypothesis: Ha:μΔμ0H_a: \mu_\Delta \neq \mu_0 (two-sided), μΔ<μ0,μΔ>μ0\mu_\Delta \lt\mu_0,\mu_\Delta\gt\mu_0 (one-sided)

  • Test Statistic: τ=μΔμ0sΔ/n \tau=\frac{\mu_\Delta-\mu_0}{s_\Delta/\sqrt{n}}

    Description

    In this case, there aren’t really two samples. Rather, there is one sample that is taken in two different points in time. This could be the heights of a certain animal, a productivity metric after implementing some new software at a workplace, or blood sugar levels after taking a dose of medicine. We’ll call the pre-treatment data set x\vec{x} and the post-treatment data set x\vec{x}'

Because of this, we are not interested in the difference between the means of the two “samples” (μ1μ2)(\mu_1-\mu_2), but rather the mean difference between the data points at the two different points in time.

μ^Δ=1ni=1nΔxi=1ni=1n(xixi) \hat{\mu}_\Delta=\frac{1}{n}\sum_{i=1}^n\Delta x_i=\frac{1}{n}\sum_{i=1}^n(x_i-x_i') Likewise, the test statistic uses sΔs_\Delta, the standard deviation of the differences of the pairs

sΔ=1n1i=1n(Δxiμ^Δ) s_\Delta=\sqrt{\frac{1}{n-1}\sum_{i=1}^n(\Delta x_i-\hat{\mu}_\Delta)}

Good job, keep it up!

49%

Completed

You have 39 sections remaining on this learning path.