Big Data & Data Engineering Basics — Data Science Roadmap
Working with data too large or fast-moving for a single machine or spreadsheet
Steps in Big Data & Data Engineering Basics
- Big Data Concepts & the Three Vs — advanced · Volume, velocity and variety — what actually makes data 'big'
- Introduction to Apache Spark — advanced · Processing large datasets across a cluster of machines instead of one
- Distributed Data Processing Basics — advanced · Partitioning, parallelism and why distributed jobs behave differently than local ones
- ETL vs ELT Pipeline Concepts — advanced · Extract-transform-load vs extract-load-transform, and why the order changed with cloud warehouses
- Data Warehouses (Snowflake/BigQuery) Overview — advanced · Where analytical data typically lives and how it differs from a transactional database
- Workflow Orchestration Overview (Airflow) — advanced · Scheduling and monitoring pipelines made of many dependent steps
Part of
- Data Science roadmap — the full learning path