Back to Data Science
About this interview
A technical interview on Experimentation and A/B Testing, pitched at the medium level. A voice AI interviewer leads the conversation, adapts its questions to your answers, keeps you on topic, and afterward gives you honest, specific feedback on where you were strong and where to improve. Expect roughly 30 minutes.
What you'll be assessed on
Design a sound A/B test including randomization checks, sample size calculation, and stopping rules
Identify common pitfalls such as peeking, novelty effects, network interference, and multiple comparisons
Apply Bonferroni correction and explain when family-wise error rate corrections are necessary
Describe how to handle skewed metrics, opt-in features, and multi-variant experiments
Topics covered
FoundationsHypothesis TestingMetric SelectionSample Size and PowerRandomization ChecksStopping RulesExperiment PitfallsSkewed MetricsMultiple ComparisonsExperiment DesignNetwork Effects and InterferenceRandomization DesignVariance ReductionDecision Making from Results
A few sample questions
Just examples to set expectations - the real interview has many more and adapts to your responses.
“In plain terms, what is an A/B test and why do companies run them instead of just shipping whatever change they think is better?
“What is the novelty effect in experimentation, how do you detect it, and what would you do if you suspect your positive result is driven by it?
“What is the difference between the family-wise error rate and the false discovery rate? When would you prefer to control one versus the other in a large-scale experimentation platform?