ETL/ELT Pipeline Engineering Roadmap — Step-by-step Learning Path
Building reliable data ingestion and transformation pipelines: extraction, loading, transformation patterns, orchestration and pipeline reliability
The ETL/ELT Pipeline Engineering roadmap
- ETL/ELT Fundamentals — beginner
- Data Extraction Techniques — beginner
- Extraction Patterns Overview — beginner
- Connecting to Databases for Extraction — beginner
- Extracting from APIs — beginner
- Extracting from Files and Object Storage — beginner
- Handling Extraction Failures Gracefully — beginner
- File Formats and Serialization — beginner
- Common Data Serialization Formats — beginner
- Columnar vs Row-Based Storage — beginner
- Schema Evolution in File Formats — beginner
- Compression Trade-offs — beginner
- Choosing a Format for a Given Pipeline — beginner
- API and Database Ingestion — beginner
- Designing Reliable API Ingestion Jobs — beginner
- Authentication Patterns for Data Sources — beginner
- Database Change Streams and Log-Based Ingestion — beginner
- Ingesting from SaaS Tools — beginner
- Managed Ingestion Tools (Fivetran, Airbyte) — beginner
- Data Loading Strategies — beginner
- Loading Strategies Overview — beginner
- Loading into Cloud Data Warehouses — beginner
- Staging Tables and Load Patterns — beginner
- Upserts and Merge Strategies — beginner
- Loading Performance Optimization — beginner
- Handling Schema Drift During Load — beginner
- Batch Processing Patterns — beginner
- Batch Processing Fundamentals — beginner
- Batch Window Design — beginner
- Partitioning Strategies for Batch Jobs — beginner
- Handling Late-Arriving Data in Batch Pipelines — beginner
- Batch Pipeline Performance Tuning — beginner
- Incremental Loading and CDC — beginner
- Incremental Loading Fundamentals — beginner
- Change Data Capture (CDC) Patterns — beginner
- Watermarking and High-Water Marks — beginner
- Handling Out-of-Order and Late Changes — beginner
- CDC Tooling (Debezium and Alternatives) — beginner
- Transformation Fundamentals — beginner
- Transformation Fundamentals — beginner
- SQL-Based vs Code-Based Transformations — beginner
- Data Cleaning and Standardization — beginner
- Joins and Enrichment in Pipelines — beginner
- Transformation Testing Basics — beginner
- Pipeline Design Patterns — beginner
- Pipeline Architecture Patterns — beginner
- Designing for Reprocessing and Backfills — beginner
- Modularizing Pipeline Logic — beginner
- Data Contracts Between Pipelines — beginner
- Medallion Architecture (Bronze/Silver/Gold) — beginner
- Case Study: Designing a Multi-Source Ingestion Pipeline — beginner
- Orchestrating ETL/ELT Pipelines — beginner
- Orchestration Fundamentals for Data Pipelines — beginner
- DAGs and Dependency Management — beginner
- Airflow Fundamentals — beginner
- Dagster and Modern Orchestration Alternatives — beginner
- Choosing an Orchestration Tool — beginner
- Error Handling and Retries — beginner
- Designing for Pipeline Failure — beginner
- Retry Logic and Backoff Strategies — beginner
- Dead Letter Queues for Bad Records — beginner
- Alerting on Pipeline Failures — beginner
- Root Cause Analysis for Pipeline Failures — beginner
- Testing Data Pipelines — beginner
- Why Testing Data Pipelines Is Different — beginner
- Unit Testing Transformation Logic — beginner
- Data Quality Tests in Pipelines — beginner
- Integration Testing for Pipelines — beginner
- Testing in CI for Data Pipelines — beginner
- Idempotency and Backfills — beginner
- Idempotency Fundamentals — beginner
- Designing Idempotent Loads — beginner
- Backfilling Historical Data Safely — beginner
- Handling Partial Failures Mid-Pipeline — beginner
- Exactly-Once vs At-Least-Once Processing — beginner
- Auditing Pipeline Runs for Idempotency Violations — beginner
- Scheduling and Dependency Management — beginner
- Scheduling Fundamentals for Pipelines — beginner
- Dependency-Aware Scheduling — beginner
- SLAs and Freshness Requirements — beginner
- Handling Pipeline Backpressure — beginner
- Monitoring Schedule Adherence — beginner
- ETL/ELT Tools and Platforms — beginner
- Comparing Managed ELT Platforms — beginner
- Open-Source vs Managed Pipeline Tools — beginner
- Building Custom Connectors — beginner
- Evaluating New Data Tools — beginner
- Migrating Between Pipeline Tools — beginner
- Scaling Pipelines — beginner
- Scaling Pipelines for Data Volume Growth — beginner
- Parallelizing Pipeline Execution — beginner
- Resource Management for Pipeline Workloads — beginner
- Cost Optimization for Data Pipelines — beginner
- Multi-Team Pipeline Ownership at Scale — beginner
- Career and Capstone — advanced