Let’s start with a general framework for A/B testing. In practice, an A/B test can be summarized into four steps.

1. Choose and characterize metrics to evaluate your experiments. What do you care about? How do you want to measure the effect?

2. Choose the significance level, power, the length of the test, and calculate the required sample size.

3. Implement the A/B test with control/treatment groups and run the test.

4. Analyze the results and draw valid conclusions

In the next few sections, we’ll dive into each step in full detail.

To frame our understanding, let’s say that we’re looking to design an experiment around different features of Interview Query.