Big Data Processing with Apache Spark

Course Overview

A course about harnessing distributed computing for data at scale — and, just as important, about knowing when you actually need it: not every dataset requires Spark, sometimes PostgreSQL is all you need. It covers Spark’s architecture and execution model, then moves into writing efficient PySpark code, implementing stream processing pipelines, and optimizing jobs for performance and resource use.

Across four modules it goes from “what is Big Data and when do you need distributed processing” through PySpark DataFrame operations, structured streaming with Kafka sources, and performance tuning and production operations — deploying and monitoring Spark on STACKIT infrastructure, including Kubernetes.

What You’ll Learn

• Define “Big Data” and identify when distributed processing is necessary
• Understand Spark’s architecture and execution model
• Write efficient PySpark code for data transformation and analysis
• Implement stream processing pipelines for continuous data flows
• Optimize Spark jobs for performance and resource utilization
• Choose between Spark and traditional databases based on data characteristics
• Deploy and monitor Spark applications on STACKIT infrastructure

Modules

• Understanding Big Data and Apache Spark
• Data Processing with PySpark
• Stream Processing with Structured Streaming
• Performance Tuning and Production Operations

STACKIT documentation university.stackit.cloud View Course on STACKIT University Open the documentation