API and Database Ingestion — ETL/ELT Pipeline Engineering Roadmap
Pulling data from live systems and services
Steps in API and Database Ingestion
- Designing Reliable API Ingestion Jobs — beginner · Structuring extraction jobs that handle a flaky external API
- Authentication Patterns for Data Sources — beginner · OAuth, API keys and token refresh in ingestion pipelines
- Database Change Streams and Log-Based Ingestion — beginner · Reading directly from database transaction logs
- Ingesting from SaaS Tools — beginner · Common patterns for pulling data from CRMs, ad platforms and similar tools
- Managed Ingestion Tools (Fivetran, Airbyte) — beginner · When a managed connector beats a custom-built one
Part of
- ETL/ELT Pipeline Engineering roadmap — the full learning path