Caching and Memory Management — Big Data Processing & Lakehouse Engin…
Using memory deliberately, not by accident
Steps in Caching and Memory Management
- Caching and Persistence in Spark — beginner · When and how to keep a DataFrame in memory across multiple uses
- Storage Levels Explained — beginner · Memory-only, disk-only and mixed persistence options
- When Caching Hurts Performance — beginner · Recognizing when caching adds overhead instead of saving time
- Broadcast Variables and Accumulators — beginner · Sharing read-only data and aggregating values efficiently across a cluster
- Memory Management Internals — beginner · How Spark divides memory between execution and storage
Part of
- Big Data Processing & Lakehouse Engineering roadmap — the full learning path