Data Engineering Projects (2026)

Data engineering projects that build pipelines end to end: Airflow, Spark and PySpark, dbt models, warehouses on Snowflake and BigQuery, and pipeline testing.

21 projects: 1 beginner, 15 intermediate, 5 advanced. Every step is checked as you go, an AI mentor helps when you are stuck, and you finish with a verifiable certificate.

Data engineering projects

  1. Apache Kafka for Event Streaming — Learn Kafka through a logistics company's story: core concepts, building a producer/consumer with correct ordering and delivery guarantees, and running it reliably in production under real load. (intermediate · $2)
  2. Data Engineering Capstone: Ingestion to Dashboard End to End — Follow a data engineer building a full pipeline for a growing retailer: extract sources, load a warehouse, transform clean models, orchestrate with alerts, and ship a BI dashboard. (advanced · $2.50)
  3. GCP Serverless and Data: Cloud Functions, Cloud Run, and BigQuery Basics — Learn GCP serverless and data basics by helping an online retailer automate image processing, containerize its checkout service, and stream order events into BigQuery for analysis. (intermediate · $2)
  4. Testing and CI/CD for Data Pipelines — Test dbt models and Python transforms, wire CI to run on every PR, and roll out pipeline schema changes safely with a real rollback plan. (intermediate · $1.50)
  5. Amazon Redshift Fundamentals: Distribution Styles and Performance Tuning — Learn Redshift's architecture, distribution styles, sort keys, workload management, and vacuuming through a streaming company's fight against a slow, skewed cluster. (intermediate · $1.50)
  6. Analytics Engineering Fundamentals: Modeling Data for BI Consumption — Learn to transform raw data into clean, trusted, BI-ready tables through staging, intermediate, and mart layers, with real SQL, semantic consistency, and dbt-style testing. (intermediate · $1.50)
  7. BigQuery for Data Engineers: Partitioning, Clustering, and Query Optimization — Learn BigQuery's serverless architecture, partitioning, clustering, and query cost optimization through a logistics startup's real data-scaling story. (intermediate · $1.50)
  8. Data Modeling Fundamentals: Star Schema, Snowflake Schema, and Normalization — Learn star schema, snowflake schema, and normalization through a furniture company's real data modeling journey, from 3NF to SCD Type 2. (intermediate · $1)
  9. dbt Fundamentals: Data Transformations as Code — Learn dbt through a real analytics engineering story: build models with ref()/source(), test data quality, generate docs and lineage, and write reusable macros in Jinja. (intermediate · $1.50)
  10. ETL vs ELT: Building Your First Data Pipeline — Follow a first-time data engineer at a coffee company as she replaces manual CSV exports with an automated ELT pipeline: extraction, transformation, warehouse loading, and orchestration. (beginner · $1.50)
  11. Real-Time Analytics: Streaming Data From Kafka Into a Warehouse Dashboard — Learn when real-time analytics beats batch, then stream Kafka events into a warehouse table and build a live, deduplicated dashboard with lag monitoring and outage recovery. (advanced · $2)
  12. Snowflake Fundamentals: Warehousing, Scaling, and Cost Management — Learn Snowflake's architecture, warehouse scaling, efficient data loading, query performance, and cost governance through a real retailer's migration story. (intermediate · $1.50)
  13. Apache Airflow Fundamentals: DAGs, Scheduling, and Orchestration — Follow a data engineer at a growing freight company as he replaces fragile cron jobs with Apache Airflow DAGs, retries, alerting, and dynamic task generation. (intermediate · $1.50)
  14. Apache Spark Fundamentals: Distributed Data Processing — Follow a growing logistics company's data analyst as she rebuilds a crashing pandas job in Apache Spark -- architecture, DataFrames, partitioning, shuffles, and performance tuning. (intermediate · $1.50)
  15. Building a Metrics Layer for Self-Service Analytics — Learn to build a metrics layer so every team gets the same revenue and user numbers, no matter which BI tool they use. (advanced · $2)
  16. Data Governance Fundamentals: Cataloging, Lineage, and Access Control — A data engineer builds a lightweight catalog, lineage impact checks, and access control for a growing e-commerce company -- real data governance, not compliance theater. (intermediate · $1.50)
  17. Data Lake Architecture: Parquet, Delta Lake, and the Lakehouse Pattern — Turn a messy S3 data swamp into a governed lakehouse with Parquet, Delta Lake, medallion layering, and compaction techniques, using real PySpark code. (intermediate · $1.50)
  18. Data Observability: Catching Silent Pipeline Failures Before Your CEO Does — Learn the five pillars of data observability, build real freshness, volume, schema, and anomaly monitoring, and design an incident response process for data pipeline failures. (intermediate · $1.50)
  19. Data Quality Engineering: Validation and Testing with Great Expectations — Stop shipping silent data bugs. Learn Great Expectations to write validation suites, catch bad data before it spreads, and build quality reports stakeholders trust. (intermediate · $1.50)
  20. Production-Grade ELT: Airflow + dbt + a Cloud Warehouse End to End — A capstone project: orchestrate dbt transformations from Airflow, enforce data quality gates, handle safe incremental backfills, and monitor a production ELT pipeline end to end. (advanced · $2)
  21. Spark Structured Streaming for Real-Time Pipelines — Build real-time PySpark pipelines with Structured Streaming: consume Kafka events, apply windowed aggregations with watermarking, and write reliable, exactly-once streaming output. (advanced · $2)

Practise data engineering more

Frequently asked questions

Do I need to install anything for these data engineering projects?

Most projects run in labs inside your browser (SQL, Excel, Python, Power BI, Git and more). Cloud projects say up front when they use your own AWS account or a simulator instead.

How is a Bartman project different from a tutorial?

You do the work yourself, step by step. Each step is checked automatically, and an AI mentor helps when you are stuck, so you finish with work you can show rather than a video you watched.

Do I get a certificate?

Yes. Finishing a project earns a certificate with its own ID that anyone can verify on bartman.ai.

What is Bartman?

Bartman (bartman.ai) is a learning platform where you build real projects with an AI mentor, practise in free browser labs, take mock tests and practice exams, and run job simulations. It was founded in 2025 and is based in Hyderabad, India.

More project topics