STACKIT Data Engineer

Data and AI stackit university learning cloud basics iaas paas security

Cloud Computing Fundamentals

STACKIT

Build a solid foundation in cloud computing: service and deployment models, core architecture patterns, and security principles for sovereign environments.

STACKIT documentation university.stackit.cloud View Course on STACKIT University Open the documentation

STACKIT

The sovereign European cloud provider behind the framework, delivering IaaS and PaaS from German and Austrian data centers with full digital independence.

Course Overview

A comprehensive introduction to cloud computing for anyone new to the field, structured around three building blocks: what “the cloud” actually is and how it differs from traditional IT, the service and deployment models providers offer (IaaS, PaaS, SaaS; public, private, hybrid, and multi-cloud), and the architectural and security principles that hold a cloud environment together. Each module is a self-contained, beginner-friendly introduction — no prior cloud experience assumed.

This is the entry point for the Data Engineer path: before working with STACKIT’s data and lakehouse services, it’s worth having a precise, shared vocabulary and mental model for what cloud computing is and isn’t.

What You’ll Learn

Distinguish IaaS, PaaS, and SaaS, and reason about which model fits a given workload
Compare public, private, hybrid, and multi-cloud deployment strategies and their trade-offs
Describe core cloud architecture patterns: elasticity, resilience, and the shared responsibility model
Explain the baseline security principles behind sovereign, compliant cloud environments

Modules

Introduction to Cloud Computing
Cloud Service and Deployment Models
Cloud Architecture and Security

BASE

STACKIT Platform Fundamentals

Architecture stackit university learning stackit portal iaas paas organization

STACKIT Platform Fundamentals

STACKIT

Get hands-on with the STACKIT Portal, core IaaS and PaaS services, and the project and resource model behind every STACKIT deployment.

STACKIT documentation university.stackit.cloud View Course on STACKIT University Open the documentation

STACKIT

The sovereign European cloud provider behind the framework, delivering IaaS and PaaS from German and Austrian data centers with full digital independence.

Course Overview

The foundational entry point into the STACKIT sovereign cloud, structured as five progressively hands-on modules. You start with an orientation to STACKIT itself, then move into the STACKIT Portal and management interfaces where day-to-day work actually happens, before working through the core infrastructure (IaaS) services and the platform and application (PaaS) services that sit on top of them. The course closes by covering how projects, resources, and organizations are structured — the account model every subsequent action on the platform depends on.

This is the first STACKIT-specific course in every learning path, and deliberately so: before you provision a VM, deploy an app, or configure networking, you need to know where things live in the platform’s project/organization hierarchy and which interface — Portal, CLI, API, or Terraform — fits the task at hand. Getting this structure right early avoids the misconfigured projects and permission confusion that otherwise surface later, once real workloads depend on them.

What You’ll Learn

Navigate the STACKIT Portal and management interfaces with confidence
Provision and configure core IaaS resources: compute, storage, and networking
Work with PaaS services for running applications without managing infrastructure
Structure and organize STACKIT projects, resources, and organizations correctly
Build the practical fluency needed before hands-on platform, security, or engineering work

Modules

Introduction to STACKIT
Portal & Management Interfaces
Core Infrastructure Services (IaaS)
Platform & Application Services (PaaS)
Projects, Resources, and Organization

STEP

Data Engineering Fundamentals

Data and AI stackit university learning data engineering architecture governance

Data Engineering Fundamentals

STACKIT

Learn the fundamentals of data engineering: the data engineer's role, the data lifecycle, modern data architectures, and governance and security.

STACKIT documentation university.stackit.cloud View Course on STACKIT University Open the documentation

STACKIT

The sovereign European cloud provider behind the framework, delivering IaaS and PaaS from German and Austrian data centers with full digital independence.

Course Overview

This foundational course introduces the role and responsibilities of a data engineer and maps out the complete data lifecycle, from generation to analysis, before tracing how data architectures have evolved — from traditional data warehouses, through data lakes, to the lakehouse and data mesh patterns used in modern platforms. It’s the bedrock course of the Data Engineer learning path: everything that follows, from Airflow orchestration to Dremio lakehouses, assumes the shared vocabulary and mental model built here.

The course closes on governance and security, treating them not as an afterthought but as core non-functional requirements of any data platform — particularly relevant in a sovereign cloud context where data residency and access control carry real regulatory weight. If you’re new to data engineering or coming from an adjacent role like software development or analytics, this is the course that establishes why the discipline exists and what a data engineer is actually responsible for day to day.

What You’ll Learn

Describe the data engineer’s role and how it differs from adjacent roles like data science and analytics
Map the end-to-end data lifecycle, from generation and ingestion to analysis
Differentiate data warehouses, data lakes, data lakehouses, and the data mesh
Apply core data governance, quality, and security principles in a sovereign cloud environment

Modules

The Data Engineer Role and Responsibilities
Data Lifecycle and Architectures
Data Governance and Security

LIFT

SQL and Relational Database Fundamentals

Data and AI stackit university learning sql databases

SQL and Relational Database Fundamentals

STACKIT

Relational database concepts, SQL querying, query optimization, and basic database administration as a foundation for data engineering.

STACKIT documentation university.stackit.cloud View Course on STACKIT University Open the documentation

STACKIT

The sovereign European cloud provider behind the framework, delivering IaaS and PaaS from German and Austrian data centers with full digital independence.

Course Overview

A foundational course in relational databases and SQL for anyone about to work with data warehouses and pipelines. It starts with relational database concepts — tables, keys, and normalization — before building up SQL querying skills from basic selects and joins to more advanced querying and optimization, and closes with basic database administration.

It’s a prerequisite for the Data Modeling and Warehousing course: comfort with joins, normalization, and basic database design is assumed from here on.

What You’ll Learn

Explain core relational database concepts and normalization
Write SQL queries covering selects, joins, and aggregations
Apply advanced SQL techniques and basic query optimization
Perform basic database administration tasks

Modules

Relational Database Concepts
SQL Querying Fundamentals
Advanced SQL and Optimization
Database Administration Basics

OPS

Data Modeling and Warehousing

Data and AI stackit university learning data warehousing dimensional modeling

Data Modeling and Warehousing

STACKIT

Design dimensional models for analytics: star and snowflake schemas, fact and dimension tables, slowly changing dimensions, and warehouse architecture.

STACKIT documentation university.stackit.cloud View Course on STACKIT University Open the documentation

STACKIT

The sovereign European cloud provider behind the framework, delivering IaaS and PaaS from German and Austrian data centers with full digital independence.

Course Overview

Data modeling is what separates good data engineers from great ones: this course teaches the art and science of designing data structures optimized for analytics and reporting, transforming normalized transactional data into dimensional models that enable fast business intelligence. It covers both the “how” and the “why” behind dimensional modeling — when to denormalize for performance and how to handle scenarios like slowly changing dimensions.

Across four modules it moves from dimensional modeling foundations (facts, dimensions, star schemas) to building fact and dimension tables, handling slowly changing dimensions with the standard SCD strategies, and warehouse architecture patterns (Kimball vs. Inmon) with performance optimization.

What You’ll Learn

Design and implement star and snowflake schemas for data warehouses
Create effective fact and dimension tables that support business analytics
Handle slowly changing dimensions (SCDs) using appropriate strategies
Understand the transition from normalized to denormalized models
Build data warehouse architectures that scale
Optimize query performance in analytical environments

Modules

Foundations of Dimensional Modeling
Building Fact and Dimension Tables
Slowly Changing Dimensions
Data Warehouse Architecture and Optimization

AUTO

Data Lakehouse Fundamentals

Data and AI stackit university learning lakehouse open table formats architecture

Data Lakehouse Fundamentals

STACKIT

Learn the core concepts of the data lakehouse architecture: open table formats, how it compares to data warehouses and data lakes, and lakehouse strategy.

STACKIT documentation university.stackit.cloud View Course on STACKIT University Open the documentation

STACKIT

The sovereign European cloud provider behind the framework, delivering IaaS and PaaS from German and Austrian data centers with full digital independence.

Course Overview

This course introduces the data lakehouse architecture and the open table formats that underpin it, working through what a lakehouse actually is, how it differs architecturally from a traditional data warehouse or a plain data lake, and where each approach still makes sense on its own. It’s a foundational, vendor-agnostic course: the goal is to leave you able to reason about the trade-offs between these architectures before you commit to a specific implementation.

That grounding matters because it’s a prerequisite for the more specialized data engineering courses that follow, including the Apache Iceberg and Dremio deep dives — this is where you build the conceptual map before working with a specific table format or query engine. The closing module turns theory into decision-making, walking through how to evaluate and build a lakehouse strategy for your own organization rather than adopting one by default.

What You’ll Learn

Explain the data lakehouse architecture and how open table formats make it possible
Compare lakehouse, data warehouse, and data lake approaches on their actual trade-offs
Recognize when each architecture is the better fit for a given workload
Evaluate the building blocks needed to define a lakehouse strategy for an organization

Modules

Data Lakehouse Architecture
Open Table Formats
Lakehouse vs Warehouse vs Lake
Building a Lakehouse Strategy

SAFE

Apache Iceberg Fundamentals

Data and AI stackit university learning iceberg lakehouse open table formats

Apache Iceberg Fundamentals

STACKIT

Apache Iceberg for the data lakehouse: table architecture, table management, schema evolution and partitioning, and time travel and branching.

STACKIT documentation university.stackit.cloud View Course on STACKIT University Open the documentation

STACKIT

The sovereign European cloud provider behind the framework, delivering IaaS and PaaS from German and Austrian data centers with full digital independence.

Course Overview

An open table format course covering Apache Iceberg, the technology behind modern data lakehouse architectures. It builds on Data Lakehouse Fundamentals with a deep dive into how Iceberg tables are structured and managed: table operations, schema evolution, partitioning strategies, and the time-travel and branching features that make lakehouse data behave a lot like a version-controlled dataset.

What You’ll Learn

Explain Iceberg’s table architecture and core concepts
Perform table management operations
Apply schema evolution and partitioning strategies
Use time travel and branching for reproducible, auditable datasets

Modules

Iceberg Architecture and Concepts
Table Management and Operations
Schema Evolution and Partitioning
Time Travel and Branching

LIFT

Data Pipelines with Apache Airflow

Data and AI stackit university learning airflow data pipelines dags orchestration

Data Pipelines with Apache Airflow

STACKIT

Learn to orchestrate data pipelines with Apache Airflow: DAGs, operators and sensors, Kubernetes integration, and CI/CD for production workflows on STACKIT.

STACKIT documentation university.stackit.cloud View Course on STACKIT University Open the documentation

STACKIT

The sovereign European cloud provider behind the framework, delivering IaaS and PaaS from German and Austrian data centers with full digital independence.

Course Overview

This course teaches you to design, build, and operate production-grade data pipelines with Apache Airflow. The challenge in modern data engineering usually isn’t processing data — it’s coordinating hundreds of interdependent tasks, handling failures gracefully, and keeping visibility into what actually ran. You’ll start with Airflow’s core architecture (scheduler, executor, workers, metadata database) and DAG fundamentals, then build up through operators, sensors, and common ETL/ELT patterns, including data quality gates and incremental loading.

The second half is where the course earns its “production-grade” framing: dynamic DAG generation, cross-DAG dependencies with external task sensors, retry and backfill strategies, and — critically for STACKIT — running Airflow on STACKIT Kubernetes Engine with the KubernetesPodOperator for task isolation, plus the CI/CD practices needed to ship DAGs safely. Every module pairs the concept with a practical exercise deployed against real STACKIT infrastructure, so the skills transfer directly to pipelines you’d actually operate.

What You’ll Learn

Design DAGs with the right operators, sensors, and task dependencies for a given workflow
Build dynamic, configuration-driven pipelines and manage cross-DAG dependencies
Implement retry strategies, alerting, and backfilling for resilient pipeline operations
Deploy and scale Airflow on STACKIT Kubernetes Engine with CI/CD for production DAGs

Modules

Airflow Architecture and Concepts
DAGs, Operators, and Sensors
Advanced Patterns and Monitoring
Kubernetes Integration and CI/CD

WARN

Big Data Processing with Apache Spark

Data and AI stackit university learning spark big data streaming

Big Data Processing with Apache Spark

STACKIT

Distributed data processing with Apache Spark: PySpark DataFrames, structured streaming, performance tuning, and production deployment on STACKIT.

STACKIT documentation university.stackit.cloud View Course on STACKIT University Open the documentation

STACKIT

The sovereign European cloud provider behind the framework, delivering IaaS and PaaS from German and Austrian data centers with full digital independence.

Course Overview

A course about harnessing distributed computing for data at scale — and, just as important, about knowing when you actually need it: not every dataset requires Spark, sometimes PostgreSQL is all you need. It covers Spark’s architecture and execution model, then moves into writing efficient PySpark code, implementing stream processing pipelines, and optimizing jobs for performance and resource use.

Across four modules it goes from “what is Big Data and when do you need distributed processing” through PySpark DataFrame operations, structured streaming with Kafka sources, and performance tuning and production operations — deploying and monitoring Spark on STACKIT infrastructure, including Kubernetes.

What You’ll Learn

Define “Big Data” and identify when distributed processing is necessary
Understand Spark’s architecture and execution model
Write efficient PySpark code for data transformation and analysis
Implement stream processing pipelines for continuous data flows
Optimize Spark jobs for performance and resource utilization
Choose between Spark and traditional databases based on data characteristics
Deploy and monitor Spark applications on STACKIT infrastructure

Modules

Understanding Big Data and Apache Spark
Data Processing with PySpark
Stream Processing with Structured Streaming
Performance Tuning and Production Operations

OPS

Data Lakehouse with Dremio (DeepDive)

Data and AI stackit university learning dremio apache iceberg apache arrow nessie

Data Lakehouse with Dremio (DeepDive)

STACKIT

Take a technical deep dive into Dremio on STACKIT: Apache Arrow, query federation, the semantic layer, Data Reflections, Apache Iceberg, and Project Nessie.

STACKIT documentation university.stackit.cloud View Course on STACKIT University Open the documentation

STACKIT

The sovereign European cloud provider behind the framework, delivering IaaS and PaaS from German and Austrian data centers with full digital independence.

Course Overview

The scenario this course is built around: your organization needs a unified analytics platform where business analysts get fast, self-service access to curated data products while data scientists query raw data in place — without moving or copying it unnecessarily, and without breaking GDPR compliance along the way. You’ll start with the strategic case for the STACKIT and Dremio partnership, then go deep on the technical internals: Dremio’s distributed query engine, how Apache Arrow enables zero-copy, vectorized execution, and how query federation lets Dremio analyze data across multiple sources without ever moving it.

From there the course builds the layers that make a lakehouse usable at organizational scale — a Universal Semantic Layer for consistent, business-friendly data products, Data Reflections to accelerate query performance dramatically without duplicating pipelines, and Apache Iceberg with Project Nessie for ACID guarantees, schema evolution, time travel, and Git-like branching over your data. Each of the three technical modules pairs the concept with a hands-on lab, so you leave able to both explain the architecture and actually operate it.

What You’ll Learn

Explain how Dremio’s distributed engine and Apache Arrow deliver zero-copy, vectorized query performance
Run federated queries across multiple data sources without moving or duplicating data
Build a semantic layer and use Data Reflections to accelerate query performance
Apply Apache Iceberg and Project Nessie for ACID tables, schema evolution, and Git-style data branching

Modules

Dremio Architecture and Apache Arrow
Query Federation and Semantic Layer
Reflections and Performance
Apache Iceberg and Project Nessie

AUTO

Docker Fundamentals

Architecture stackit university learning docker containers

Docker Fundamentals

STACKIT

Open Source In 2 trails

A comprehensive introduction to Docker and containerization: building images, running containers, and orchestrating them with Docker Compose.

STACKIT documentation university.stackit.cloud View Course on STACKIT University Open the documentation

STACKIT

The sovereign European cloud provider behind the framework, delivering IaaS and PaaS from German and Austrian data centers with full digital independence.

Course Overview

A hands-on introduction to containerization with Docker: how it packages applications and their dependencies into isolated, portable containers so they run the same on a laptop, in a development environment, or in the cloud. The course walks through building and managing containers efficiently, then orchestrating multi-container applications with Docker Compose, using a “Hello STACKIT” website as a running practical exercise.

It’s built as the foundation for the follow-on Kubernetes course, both on STACKIT VMs and locally with Docker Desktop.

What You’ll Learn

Understand the basic concepts of containerization and how Docker works
Create Docker images and efficiently start, manage, and optimize containers
Gain practical experience with Docker on STACKIT VMs and locally with Docker Desktop
Work with Docker Compose to orchestrate multi-container applications
Implement practical use cases through hands-on exercises in each module
Build a solid foundation for the advanced Kubernetes course

Modules

Containerization Concepts
Building and Running Containers
Docker Compose
Hands-On: Hello STACKIT

STEP

Kubernetes Fundamentals

Architecture stackit university learning kubernetes containers ske cloud native

Kubernetes Fundamentals

STACKIT

Open Source In 2 trails

Master Kubernetes architecture, core objects, networking, and storage, then apply it hands-on with the STACKIT Kubernetes Engine (SKE).

STACKIT documentation university.stackit.cloud View Course on STACKIT University Open the documentation

STACKIT

The sovereign European cloud provider behind the framework, delivering IaaS and PaaS from German and Austrian data centers with full digital independence.

Course Overview

An eight-module, hands-on introduction to Kubernetes built for cloud engineers approaching the topic for the first time — no prior Kubernetes knowledge required, just basic Docker and Linux familiarity. You start with cluster architecture and a local environment (Minikube or Kind), then work through core objects and YAML manifests, Services and Ingress, persistent storage and StatefulSets, RBAC and secrets management, and scheduling and scaling, building fluency with kubectl and multi-container patterns like sidecars and init containers along the way.

The course can be completed entirely in a local cluster or on a live STACKIT Kubernetes Engine (SKE) cluster, and a dedicated module walks through deploying your own application to SKE so the concepts land as production practice, not just theory. A closing module leaves you with links, tips, and exam guidance for the CNCF CKAD certification, making this course a solid foundation whether your goal is day-to-day cluster operations or a formal certification path.

What You’ll Learn

Work confidently with Pods, Deployments, Services, ConfigMaps, Secrets, Volumes, and Namespaces
Write and customize YAML manifests to define Kubernetes resources
Apply multi-container patterns: sidecars and init containers
Configure networking, persistent storage, and RBAC for real workloads
Deploy and operate applications on the STACKIT Kubernetes Engine (SKE)
Prepare for the CNCF CKAD certification exam

Modules

Kubernetes Architecture and Local Environments
Core Objects: Pods, Deployments, and YAML Manifests
Services, Ingress, and Network Policies
Persistent Storage and StatefulSets
RBAC, Configuration, and Secrets Management
Scheduling, Affinity, and Workload Scaling
STACKIT Kubernetes Engine (SKE) Integration
CKAD Exam Preparation

LIFT

Kubernetes for Data Engineering

Data and AI stackit university learning kubernetes data engineering

Kubernetes for Data Engineering

STACKIT

Run data workloads on Kubernetes: stateful services and persistent storage, Airflow and Spark on Kubernetes, Kafka, and data platform operations.

STACKIT documentation university.stackit.cloud View Course on STACKIT University Open the documentation

STACKIT

The sovereign European cloud provider behind the framework, delivering IaaS and PaaS from German and Austrian data centers with full digital independence.

Course Overview

Kubernetes excels at stateless applications, but data engineering brings different challenges: stateful workloads, persistent storage, ordered operations, and resource-intensive jobs. This course, building on basic Kubernetes knowledge, addresses those challenges head-on and teaches production-ready data infrastructure on Kubernetes.

Across four modules it covers StatefulSets and persistent storage for databases, running Apache Airflow on Kubernetes with the KubernetesExecutor, deploying distributed processing frameworks (Spark, Flink) and Kafka, and data platform operations — resource management, monitoring, security, and disaster recovery.

What You’ll Learn

Deploy and manage stateful data services on Kubernetes
Implement persistent storage strategies for databases and data lakes
Run Apache Airflow on Kubernetes with the KubernetesExecutor
Deploy distributed data processing frameworks (Spark, Flink)
Build event streaming platforms with Kafka on Kubernetes
Optimize resource allocation for data-intensive workloads
Handle backup, disaster recovery, and data migration

Modules

Stateful Workloads and Storage
Airflow on Kubernetes
Distributed Processing Frameworks
Data Platform Operations

LIFT

STACKIT Dremio: Getting Started

Data and AI stackit university learning dremio lakehouse

STACKIT Dremio: Getting Started

STACKIT

Get started with STACKIT Dremio, the managed sovereign data lakehouse: instance setup, SSO/OIDC authentication, and connecting data sources.

STACKIT documentation university.stackit.cloud View Course on STACKIT University Open the documentation

STACKIT

The sovereign European cloud provider behind the framework, delivering IaaS and PaaS from German and Austrian data centers with full digital independence.

Course Overview

An introduction to STACKIT Dremio, a fully managed data lakehouse service running entirely on STACKIT’s sovereign infrastructure — Kubernetes orchestration, elastic engine nodes, and storage integration are all handled for you. The course covers what sets STACKIT Dremio apart from Dremio Cloud and self-hosted Dremio, then walks through creating and configuring your own instance, setting up SSO/OIDC authentication and access policies, and connecting data sources through the Open Catalog.

It’s the practical bridge between the conceptual Data Lakehouse and Dremio (DeepDive) courses and the guided exercises in STACKIT Dremio: Hands-On Lab.

What You’ll Learn

Explain what STACKIT Dremio is and how it differs from Dremio Cloud and self-hosted Dremio
Create and configure a STACKIT Dremio instance through the STACKIT Portal
Navigate the Dremio web interface
Configure SSO with OpenID Connect and Microsoft Entra ID
Set up service users and row/column-level access control
Connect STACKIT Object Storage and configure the Open Catalog (Nessie)

Modules

Introduction to STACKIT Dremio
Creating and Configuring Your Instance
Authentication and Security
Connecting Data Sources and Open Catalog

GOAL

STACKIT Dremio: Hands-On Lab

Workshop

STACKIT Dremio: Hands-On Lab

STACKIT

Data and AI stackit university learning dremio hands-on iceberg lab

Build a complete data lakehouse on STACKIT Dremio in this hands-on lab: instance setup, Iceberg tables, the semantic layer, Data Reflections, and Nessie branching.