Data Pipelines with Apache Airflow

Course Overview

This course teaches you to design, build, and operate production-grade data pipelines with Apache Airflow. The challenge in modern data engineering usually isn’t processing data — it’s coordinating hundreds of interdependent tasks, handling failures gracefully, and keeping visibility into what actually ran. You’ll start with Airflow’s core architecture (scheduler, executor, workers, metadata database) and DAG fundamentals, then build up through operators, sensors, and common ETL/ELT patterns, including data quality gates and incremental loading.

The second half is where the course earns its “production-grade” framing: dynamic DAG generation, cross-DAG dependencies with external task sensors, retry and backfill strategies, and — critically for STACKIT — running Airflow on STACKIT Kubernetes Engine with the KubernetesPodOperator for task isolation, plus the CI/CD practices needed to ship DAGs safely. Every module pairs the concept with a practical exercise deployed against real STACKIT infrastructure, so the skills transfer directly to pipelines you’d actually operate.

What You’ll Learn

• Design DAGs with the right operators, sensors, and task dependencies for a given workflow
• Build dynamic, configuration-driven pipelines and manage cross-DAG dependencies
• Implement retry strategies, alerting, and backfilling for resilient pipeline operations
• Deploy and scale Airflow on STACKIT Kubernetes Engine with CI/CD for production DAGs

Modules

• Airflow Architecture and Concepts
• DAGs, Operators, and Sensors
• Advanced Patterns and Monitoring
• Kubernetes Integration and CI/CD

STACKIT documentation university.stackit.cloud View Course on STACKIT University Open the documentation