Beyond the Basics of A/B Tests: Highly Innovative Experimentation Tactics You Need to Know

Editor's Notes

1. Statsig was founded 3 years ago… We’re a scrappy but growing team based out of the beautiful Pacific Northwest. We are famous for our 100% in-person office culture. We’re also a dog-friendly office. I’m sitting in the front row with my dog Parker who lucked out and came on a day where we took our company photo.
2. Statsig is an experimentation and… feature flagging platform that powers companies like Notion, Figma, and OpenAI. We help teams manage their feature rollouts, setup experiments and provide results. In general, we make it easy to be data-driven. We have 2 products Statsig Cloud Process>200B events a day 20k experiments across >1B unique user identifiers. Statsig WHN You get the full power of Statsig Cloud, but the compute happens within our customers’ data warehouses as the raw data never leaves.
3. In this talk I’ll review what is A/B Testing, why it’s becoming best practices in product development, and how experimentation is the foundation of a data-driven culture. Experimentation 201 - covers the more popular advanced techniques.
4. Why A/B Test? There are many tools in the Product analytics toolbox, but most of them are correlation-based analyses. Experimentation however measures causation. Experimentation is the basis of the scientific method, and is the gold standard for measuring causality. This is where data-driven decision making starts. If we do A, B will happen and by about X%. Data can trump opinions, but you have to collect it first. Skeptics will criticize this, and say they don’t need AB testing. They hire smart people who have good product sense and intuition. But the truth is, among top tech companies, only 1/3rd of all ideas work (published data). Turns out intuition is often wrong. This is because building products is fundamentally hard. But even if your intuition is good, experimentation still lets you quantify the impact while producing richer insights. If you’re still not convinced, then talk to Sean Taylor who’s here at Data Council…
5. How it Works You have a heterogeneous population of users, and the first thing you do is is randomly assign the users into two groups Randomization is the secret sauce With enough users, it produces two equal and comparable groups. If your user base contains 10% power users, it will make sure that about half of those are in test and the other half are in control. Same thing all other user traits: Android/iOS splits, new users, and gender. But the really cool thing is that it not only controls for all known confounding variables, it also controls all unknown factors. For example, what if your competitors are experimenting on your users and offering a competing promo? With these groups, we then subject them to two different experiences, called the Test and Control. This is done over the same time period so that seasonal effects affect both equally. Any difference we observe in behavior between the two groups can be attributed to the difference in their experience This is causality, we gave a coupon to one group and not only were first-time purchases up 5%, total monthly revenues were up 2%.
6. There are a lot of ways to do experimentation wrong Here’s a list of things to watch out for First, you should always start with a hypothesis. You should know what you’re changing and what you expect to happen. Focus on the primary effect, the first observable metric you expect to change. If you shorten your signup flow, maybe you expect more signups to happen. But also ask yourself, what else can happen? More timespent, more invites? What can go wrong? What critical business metrics might change. All of this should be included in your scorecard. Next, you’ll also want to run a power analysis. This let’s you estimate how many users you need to detect the results you expect. This ensures your experiments have a reasonable chance of succeeding. It’ll also guide how many users and for how long you need to run the experiment for. I also recommend standardizing the methodology across your experimentation program. You want to use the same statistics, the same metrics, and the same decision-making framework. This ensures results are comparable between experiments, across teams and people are speaking the same language. I’m a big fan of using 95% confidence intervals by default. While there are certainly reasons to increase or decrease it, unless you’re able to clearly articulate these reasons BEFORE an experiment, stick with 95% please. It is a practical threshold that makes running successful experiments achievable while maintaining a reasonably low false positive rate. Lastly, don’t fret about interaction effects. This is when experiments collide and interfere with each other. There’s research that says interaction effects are fairly rare. And even when they do occur, they often won’t result in different decisions. Unfortunately to eliminate interaction effects, people will often dividing up their user base, or run experiments sequentially. This is poison. This reduces their pace of experimentation, and slows down their rate of innovation.
7. I want to introduce a character the Hippo. It’s short for Highest paid person’s opinion and it’s how many businesses make decisions. It’s what you do when you aren’t data-driven. I’ve learned that experimentation is great for producing concrete and simple facts like This feature increased retention by 2%. That recommendation model reduced revenue by 3%. It’s hard to ignore data like this because it’s a causal statement. This helps companies become grounded in data.
8. I’ll talk a lot in the later section about the importance of advanced statistics. But don’t get distracted. It’s far more important to focus on culture rather than fancy stats. Focus should be on the people and processes. It should be trivial for an engineer to set up, execute and analyze a simple AB test. Focus on democratizing data. Everyone on the product team… PMs, Engineers, and Data scientists should be involved. Experiment review is a critical part of a company’s data culture. The scientific method was designed to invite questions. This is where discussions can take place, where assumptions are challenged, and where knowledge and best practices are shared. Lastly, your company should be optimizing for velocity. Find ways to remove friction so that more ideas are tested.
9. Welcome to experimentation 201 - We’ll cover some of the more popular ways to address the limitations of standard AB testing.
10. CUPED Popularized in online experimentation by Microsoft (Kohavi paper published in 2013) variance reduction technique Not all variance is purely random. User-level variance comes from pre-existing factors! Example of high variance situations car purchases based on prior purchase Benefits 30-60% reduction in variance across real Statsig experiments Faster experiments, lower sample sizes, more precise decision making Considerations Less effective for new users. Must use user attributes
11. The next challenge is sometimes referred to as the Winner’s Curse Winner’s in A/B testing don’t live up to the hype. We’ve had a customer who’s old experimentation platform told them they were up 40% on revenue. Great right? But the problem is overall revenue was only up 10%. This is a great way to lose trust in your experimentation tooling.
12. There are several reasons this may happen: First is long-term sustainability. Are the metric lifts you observe going to hold up over time? Or are these just novelty effects? Run longer or wait for metrics to stabilize Next is underpowered experiments. If you’re short on users and time, you may be tempted to underpower your experiments. This can lead to a large amount of statistical error in your lift estimates. Null hypothesis is in black: this is if there were no effect. We set the threshold for rejecting the null hypothesis at a p-value of 0.05. If you have a good experiment, that’s confidently above this threshold, you’ll get a probabilistic value from this distribution, that's in green. For example…
13. Another reason is false positives: You don’t actually have an experimental lift and the test group is the same as control. In this case, you have a 5% chance of finding a statistically significant result when there isn’t actually any lift.
14. Next is over-estimations. If your experimental effect is not quite above the threshold, you can still have a chance of declaring this a winner. This isn’t that bad, it’s still a lift. But you’ll overestimate this for sure.
15. And lastly, biased decision making. This is human error. We tend to look at results with rose-colored glasses and can sometimes cherry pick results. This means sometimes an experiment is just bad… the results are bad. But we find ways to ignore these, and ship anyways. All of this can lose trust in experimentation. And people can start to game the system.
16. Solution: Holdouts definition: small subset of users intentionally withheld from a treatment after a full rollout. Typically long-term (>3 months). Usages: Accuracy in long-term measurement Meta notifications holdout. [MORE RESEARCH] Cumulative estimates measurement of a team or an experimentation program. How good are the wins really? Are we making proper decisions? Reroll of randomization. A “second” opinion Can be used in performance reviews, for resourcing, and for keeping teams honest. Also powerful as a debugging tool. I won’t get into this, but sudden outages and metrics movements can be quickly isolated to a set of features using a network of holdouts. Downside: Holdouts are deceptively expensive. Engineering teams have to maintain two branches. And if they don’t do it right, you’ll contaminate the holdout. My advice: Make sure you have top-level buy-in. Make sure you readout holdouts at a fixed cadence (visibility/utility)
17. Now, let’s talk about the peeking problem The standard hypothesis test is based on statistics that generate a 5% false positive rate. This is based on a single observation, at the full duration of the experiment. Problem is that we’ve been telling PMs and engineers to be monitoring their product dashboards. The data-driven ones want to see experimental results as soon as they launch. Watching how your experiment is going is simply human nature. There are other reasons one may want to peek: Finding wins and locking them in early. Detecting regressions and aborting experiments Finding issues and fixing them. There are practical considerations when trying to solve the peeking problem and all of this makes the stats hard. How often will you peek? What’s the schedule? Are you going to make a decision? How will you ensure you’re optimizing for long-term effects and not just overreacting to novelty effects? How do you adjust for multiple metrics and their tradeoffs?
18. The solution is called Sequential Testing It’s a generic term for statistical methods that account for continuous or periodic monitoring. There are many methods All of them pose tradeoffs between factors like: Statistical power Sensitivity (can they detect effects early?) Speed We ended up selecting mSPRT after careful evaluation using real data across hundreds of experiments and thousands metrics. We found that 60% of experimental effect were detected at the halfway mark while guaranteeing the false positive rate of 5%. mSPRT has the added advantage of not requiring a tuning parameter. There’s another problem: What if your new ranking model is generating a ton of clicks early and sequential testing says this is statistically significant… should you ship early? Well what about guardrail metrics? What if revenue is between -5% and +5%… is that okay? Our recommendation is to use sequential testing to identify regressions that are worthy of aborting. And wait for the full duration of the experiment to fully evaluate all metrics. This gives you full statistical power across your scorecard. I personally really like model. It’s human nature to be rooting for an experiment, and we are giving you permission to abort early, but you must be patient for the win.
19. Now let’s talk about randomization Everyone familiar with Canadian coins? Well it’s just like US money, but there’s a $1 coin called a loonie, and a BIG bimetallic coin called the toonie because it’s worth $2. And technically the penny is no longer, but I’ve kept it here for this example. If I take a pile of Canadian coins and randomly split it into two groups, it’s very likely i’ll end up with an imbalance like this. Why? Well some coins are worth orders of magnitude more than other coins, and randomization doesn’t work so well here.
20. Instead what if we carefully balanced the two groups? We can generate two equivalent groups. This is comparable. This is a real problem in B2B experiments.
21. B2B typically suffers from skew and small sample sizes. For example, Statsig has skew. If you’re us, what if Atlassian and OpenAI are in the same group? To balance an experiment, we apply a technique called Stratified Sampling. We can further analyze the experiment by subgroups to understand impact across the entire user base.
22. Another limitation of AB testing is fixed allocation. This is where we run a 50/50 test, and hold that constant. But what if one of your groups is doing better than the other? Don’t you want to shift traffic? And what if time is in really short supply and there’s an urgency to strike while the iron’s hot? This sort of situation happens for things like Black Friday sales where you cannot afford to wait a week to measure the impact… you want to get to the winning variant within hours if not minutes.
23. One solution is multiarmed bandits The example I’ll use is sporting websites that may want to test 8 different video thumbnails or headlines for last night’s games. They want to converge on the best variant while the game is still recent and people are still interested, rather than a week from now. This example is great, because multiarmed bandits automatically allocate traffic without manual decision making. It’s also great for situations with lots of variants as it will eliminate the poor performers fairly quickly. One major downside though, is that this isn’t great for creating generalized learnings, nor situations where decision-making is complicated and tradeoffs between metrics are unknown.
24. Network Effects can be a big problem. This is where the test and control groups can affect each other. It violates our assumption of independence. This is most commonly found in two-sided marketplaces like social networks, online marketplaces, and communication platforms. Imagine you’re a ride sharing company and want to test a different rider matching algorithm. This can causes riders in one group to book more rides, depleting the supply of drivers, which negatively impacts the control group. The results you’ll get from such a test, difference between the test and control groups, will not be indicative of what happens when you fully roll out this feature.
25. Companies like Lyft and Uber pioneered switchback testing as a way to solve this. Instead of splitting your user base 50/50, it switches time blocks randomly. Now the entire ecosystem can have a rider algorithm applied. This sort of testing is also ideal for infra and backend experiments. There are some important considerations here: Picking your time interval is critical. You want frequent switches so you have a lot of observations, but you want a time interval that allows the ecosystem to fully stabilize and be measured.
26. Finally, let’s talk about heterogeneous effects In AB testing, we measure the average treatment effect. But we all know there’s no such thing as the average user. A major software company shared a story with me of how they ran a signup flow test that generated a small up-lift, say 0.4%. But when they split by gender, they found it was divergent. Interesting… this is still positive though so they still made the right decision. But they then looked prior 10 experiments that were shipped and they were all biased towards men. This was pretty bad because cumulatively, this means their product is now less successful for half the population. The area that this is more common though is looking for technical bugs. What if your new feature doesn’t work on specific browser types? Or Android devices with small screens? To do this right, you’ll want to run automated detection across a set of attributes, correcting for Type I errors due to the multiple comparison problem.
27. Lastly let’s talk about Experimental Meta Analysis After you’ve ran dozens or hundreds of experiments, you now have a small dataset containing causal observations. There’s a LOT you can do with this and this is an area we’re exploring and don’t have all the answers. But we want to surface data like: relationship between metrics identifying proxy metrics Understanding what movements are possible
28. Conclusion I did a whirlwind tour through a bunch of solutions to specific problems in AB testing. The key takeaway here is there’s a lot you can do. But I do want to leave you with a lasting thought: While these are really cool, stats engines don’t build culture. I don’t think folks should overthink experimentation, it’s better to run an experiment than to talk yourself out of it.
29. This concludes my talk. I want to thank you for watching. If you want to hear more, please follow me on LinkedIn. If you want to get in touch, my email and twitter accounts are here as well if you prefer. And if you don’t care about experimentation but like dogs, visit statsig.com/pets Thank you very much.