Move fast #1: Eliminate unnecessary A/B testing
A/B testing is wildly popular in Silicon Valley. A/B tests allow you to experiment your way into product decisions rather than relying solely on intuition or guesswork: Can’t decide if you should do option A or B? Do a bake-off in an A/B test and let the users show you which option they prefer: very little thinking required – yay!
However, A/B testing is not free, and while often necessary, it is important to be aware of the costs of conducting an A/B test, including:
Developing & testing several variants: The direct cost of developing multiple variants. Efforts can be minimized with the right infrastructure and tools, but testing will still drive a delay in the full ramp.
Managing the ramp: The cycles spent ramping up and down the variants and managing potential user confusion should they notice.
Analyzing the results: The cycles spent gathering and analyzing the results. While many tools automate A/B analysis, ad-hoc analysis is often required to understand impacted cohorts' full site impact or downstream engagement/monetization.
Opportunity costs: The cost of losing focus. A/B testing lends itself to thinking incrementally, so you may find yourself making micro-tweaks and losing sight of the big leaps that will deliver orders of magnitude more valuable for your users. The clear cause-effect output metrics from A/B testing can be comforting in the mysterious world of consumer behaviors.
With these costs in mind, it is helpful to ask oneself these four questions before launching an A/B test:
What decision will this test drive? You should be able to articulate the decision ahead of the test. Being able to split out the attribution of several product changes is not a decision. Quantifying the impact of a ‘must-do’ feature is not a decision.
Is this the most important decision you need to make right now? Some see a day without an A/B test as a lost learning opportunity. Given the abovementioned costs, I don’t believe in testing for testing’s sake, even if you have capacity to ramp. Testing the full rainbow of colors for your CTA button will not materially improve the value for your customer.
Which metric do I care about for this test? You need to measure the impact of the test on ‘the metric that matters’ (topic for another post). This may not be the most direct metric impacted by your test. For example, suppose you are testing a copy in an outbound marketing email. In that case, you need to test the impact on a customer's down-funnel purchase or lifetime value, not the initial email open or click.
Which variant do you believe will win? A/B testing is not an excuse for not having a point of view on your users. Make sure that the variant you have the highest confidence in is ramped the first and the most aggressively – nowhere does it say that the most aggressive feature should be the least or last ramped. Testing your ‘killer’ feature at 5% ramp for two months will delay time to market 2 months for 95% of your users.
Instituting a hurdle based on these four questions will curb your a/b testing, retain velocity, and maintain your product development focus on what matters instead of what is easy to test and measure.