There are scenarios where A/B testing is not necessarily the best course of action. Often, there are technical, infrastructure, or practical concerns that come up while planning an A/B test.

Some of these issues include:

1. You don’t have enough traffic

If you don’t have significant traffic on a page, it is probably best to assess a different page or wait until there is sufficient traffic to run an A/B test.

2. You don’t have the time to conduct a test.

Related to the traffic concern, if there is not enough time to create a large enough sample of customers, the A/B test will not be valid.

3. There are tradeoffs between large and small changes.

When the testing website changes, tension can emerge in the issue of the change magnitude. While small, incremental changes can be more interpretable, they are also more time and resource-consuming, especially on low-traffic websites. Conversely, large changes can result in a ‘winner’ more quickly but may be less interpretable.

In addition to these technical issues, A/B testing as a strategy is not necessarily suitable for all kinds of investigations.

1. There can be ethical issues surrounding A/B testing.

For instance, in a charity that deals with high-risk clients, experimenting with the efficacy of reminder letters could have negative impacts on some clients. Depending on the nature of the item being tested, there needs to be a consideration of the ethical implications of giving clients different (possibly ‘worse’ versions) of a product. For those curious, this is related to the principle of equipoise, which suggests that it is not ethical to randomize people into groups where you have strong reason to believe that one of the experiment arms would be harmful to the participants. This principle comes up in many RCTs, like clinical trials (Randomly Controlled Trials).

2. A/B testing isn’t suitable for some kinds of scenarios.

As an analyst investigating the sales of certain products at a brick-and-mortar store, you won’t be able to easily randomize streams of people. Other methods may be better for analysis, like counterfactuals or ‘natural experiments’.