Now that you know a little bit more about A/B testing, let’s get into some real-world scenarios.

In interviews, you will most likely be asked about less-than straightforward situations. Some could involve testing multiple elements on a page or understanding the effect of a feature change on different customer segments. Similarly, there can be situations where a test can be constrained by practical or technical concerns, or where there are tradeoffs to split testing. Since you, the data scientist, will ultimately have to make choices and argue your case to stakeholders, it is important to know how to navigate these kinds of situations.

The next couple of chapters will cover common scenarios and concepts involved in A/B testing. As A/B testing involves statistical concepts, there may be terms that you need refreshing on. If you get stuck, bolded terms will be covered in more detail under the Statistics chapter.

Case Study:

For most of these scenarios, we will work from an example case study:

You are a data scientist at an e-commerce company selling clothing called ‘Sweatshirts4U.ai’.

You and the product team notice that not many customers who check out are signing up for an account, instead opting for ‘guest checkout’.