Maximizing marketplace experimentation: switchback design for small samples and subtle effects
Nazli M. Alagoz, Joël Gastelaars, amira mahfoudhi
Conventional A/B testing often falls short in industries such as airlines, ride-sharing, and delivery services, where challenges like small samples and subtle effects complicate testing new features. Inspired by its significant impact in leading companies like Uber, Lyft, and Doordash, we introduce the switchback design as a practical alternative to conventional A/B testing. By addressing small sample size limitations and the need to detect subtle effects quickly, this approach boosts statistical power while reducing variability and interference. We guide the audience through the challenges of marketplace experimentation and implementing this approach, period length optimization and switch frequency using a case study from the airline industry.