Email A/B Testing in Python: Is the Uplift Real?

<p>Ananya Bose runs email at Monsoon Leaf Tea Co., a Kolkata tea company that sells online across India. Her Diwali gift-box email tested two subject lines, and the agency says B won: opens up by a third. The founder wants to make urgency subject lines the default for the festive season. Ananya has to answer the real question: did B sell more tea, or did it just get opened more?</p><p>Across five lessons you analyse the test in Python on your own copy of the raw exports: cleaning a send log of retries, internal seed inboxes and bounces; turning event logs into people; matching paid orders inside a 7-day attribution window; the two-proportion z-test, p-values and confidence intervals; the sample ratio check, Welch's t-test on revenue per recipient, and a decision rule set before the test. In every lesson you build the step yourself first, then hand the same job to your AI (ChatGPT, Claude, Copilot or Gemini) and learn exactly where it goes wrong: the wrong denominator, a one-sided p-value, a case-sensitive filter, a test on opens instead of orders. In the capstone your AI runs the whole analysis from one brief, and you prove every number before it reaches the founder.</p><p>Bartman runs your script on its own servers against your data and checks the files it writes, so every check is about your real output, not a screenshot.</p>

Tools: Python, pandas, SciPy, A/B Testing, Hypothesis Testing, Email Marketing Analytics, matplotlib, AI Assistants for Python