07-11, 14:45–15:15 (Europe/Amsterdam), Else (1.3)
Conventional A/B testing often falls short in industries such as airlines, ride-sharing, and delivery services, where challenges like small samples and subtle effects complicate testing new features. Inspired by its significant impact in leading companies like Uber, Lyft, and Doordash, we introduce the switchback design as a practical alternative to conventional A/B testing. By addressing small sample size limitations and the need to detect subtle effects quickly, this approach boosts statistical power while reducing variability and interference. We guide the audience through the challenges of marketplace experimentation and implementing this approach, period length optimization and switch frequency using a case study from the airline industry.
In this talk, we introduce switchback design, a method that addresses key challenges in marketplace experimentation faced by sectors like airlines and ride-sharing. It addresses traditional A/B testing's limitations on small samples and subtle effects, highlighted by successes in companies like Uber, Lyft, and Doordash. This approach aims to increase the precision of experiment results and, thus decrease the required experiment runtime.
Talk outline:
- Why do we use experimentation
- Challenges in marketplace experimentation
- What is switchback design
- Considerations when implementing switchback design
- Q&A
The intended audience: Targeted at data scientists, data analysts, product managers, and anyone interested in data-driven decision-making. Ideal for those curious about experimentation and causal inference in sectors like tech and e-commerce.
No background needed: Open to all levels, no prior knowledge needed. Concepts will be explained simply, focusing on practical insights without complex math.
The takeaway for the audience: The audience will understand the effectiveness of switchback design in overcoming marketplace experimentation hurdles, such as dealing with small samples and subtle effects.
No prior knowledge expected.
Naz is a data scientist at ACMetric, a Data Science & Artificial Intelligence Consulting firm. She specializes in leveraging causal inference and machine learning to improve experimentation and analysis. Alongside her work, she is in her final stages of Ph.D. in Quantitative Marketing. She is skilled in Python and R, turning complex data into clear insights and recommendations for stakeholders. Passionate about reproducible science, she is a data science blogger and speaker.
Working on the edge of Business Strategy and Data Science. For me, it is all about converting business challenges into scalable data (science) solutions to create tangible value for the client. Currently leading the implementation of a company-wide experimentation & measurement platform in the airline industry and consulting on organizational change.
Besides being a "techy", I like to make sure everyone understands what we do, why we do it and how it adds value for their business.
