Data Observability: Catching Silent Pipeline Failures Before Your CEO Does

<p>A pipeline can run perfectly and still hand your CEO a $0 revenue dashboard. This project follows a data engineer at a fast-growing D2C company through a string of near-miss incidents -- a silently expired API token, a pagination bug that crushed row counts, a renamed field that broke a payments model, and a promo-code bug that quietly drifted average order value -- and builds the monitoring that would have caught each one. You'll write real freshness and volume SQL checks, a schema-diffing system, a Python-based statistical anomaly detector, and a data incident response runbook with a severity matrix and stakeholder communication templates, so the next silent failure gets caught by a monitor instead of by a board deck.</p>

Tools: SQL, Snowflake, Python, Airflow, dbt, PagerDuty, Slack