Product Management for AI II: Execution, Experiments and Metrics

<p>Follow Renata Cho, a product manager at the fictional secondhand marketplace Thistle, as she takes an already-built AI feature called Snap List from a good-looking offline number to a real, defensible product decision. Across five lessons you will learn how to design an online A/B test for an AI feature (choosing a randomization unit, pre-registering a primary metric and guardrails, and guarding against the novelty effect), replace a single accuracy score with precision/recall tradeoffs and business-aligned metrics, read a mixed experiment result honestly instead of rounding it up or down, close the loop between offline evaluation and real online impact, and catch and respond to a live model regression caused by drift and a silent upstream model update &mdash; all taught through Renata's actual shipping and iteration process rather than lecture-then-practice.</p>

Tools: A/B Testing for AI Features, Eval Metrics Design, Offline vs Online Evaluation, Model Drift Monitoring, Experimentation