Data Engineer Roadmap
Comprehensive 4-phase roadmap for Data Engineer covering 12 key topics with practical deliverables.
Curriculum Syllabus & Milestones
Phase 1: Foundations
Focus area covering 3 core topics in Foundations.
-
Python & SQL
Sharpen Python scripting and advanced SQL (window functions, CTEs).
- Study primary concepts for: Python & SQL
- Actionable guide: Sharpen Python scripting and advanced SQL (window functions, CTEs).
- Build or test a practical mini-project applying Python & SQL.
-
Data Sources & Formats
Work with CSV, JSON, Parquet and understand structured vs unstructured data.
- Study primary concepts for: Data Sources & Formats
- Actionable guide: Work with CSV, JSON, Parquet and understand structured vs unstructured data.
- Build or test a practical mini-project applying Data Sources & Formats.
-
Linux & Git
Practice shell scripting and Git workflows used in data pipelines.
- Study primary concepts for: Linux & Git
- Actionable guide: Practice shell scripting and Git workflows used in data pipelines.
- Build or test a practical mini-project applying Linux & Git.
Phase 2: Core Skills
Focus area covering 3 core topics in Core Skills.
-
Build a Data Pipeline
Write an ETL script that extracts, transforms and loads data into a database.
- Study primary concepts for: Build a Data Pipeline
- Actionable guide: Write an ETL script that extracts, transforms and loads data into a database.
- Build or test a practical mini-project applying Build a Data Pipeline.
-
Orchestration with Airflow
Convert your script into a scheduled Airflow DAG.
- Study primary concepts for: Orchestration with Airflow
- Actionable guide: Convert your script into a scheduled Airflow DAG.
- Build or test a practical mini-project applying Orchestration with Airflow.
-
Data Warehousing
Load and model data in a warehouse (BigQuery/Snowflake) using star schema.
- Study primary concepts for: Data Warehousing
- Actionable guide: Load and model data in a warehouse (BigQuery/Snowflake) using star schema.
- Build or test a practical mini-project applying Data Warehousing.
Phase 3: Applied Practice
Focus area covering 3 core topics in Applied Practice.
-
Streaming Basics
Explore Kafka fundamentals with a simple producer/consumer example.
- Study primary concepts for: Streaming Basics
- Actionable guide: Explore Kafka fundamentals with a simple producer/consumer example.
- Build or test a practical mini-project applying Streaming Basics.
-
dbt for Transformations
Model raw data into clean tables using dbt with tests.
- Study primary concepts for: dbt for Transformations
- Actionable guide: Model raw data into clean tables using dbt with tests.
- Build or test a practical mini-project applying dbt for Transformations.
-
Data Quality & Testing
Add data validation checks (Great Expectations or dbt tests) to your pipeline.
- Study primary concepts for: Data Quality & Testing
- Actionable guide: Add data validation checks (Great Expectations or dbt tests) to your pipeline.
- Build or test a practical mini-project applying Data Quality & Testing.
Phase 4: Career
Focus area covering 3 core topics in Career.
-
Cloud Data Services
Use a managed cloud data pipeline service (Glue/Dataflow) for one small job.
- Study primary concepts for: Cloud Data Services
- Actionable guide: Use a managed cloud data pipeline service (Glue/Dataflow) for one small job.
- Build or test a practical mini-project applying Cloud Data Services.
-
Infrastructure as Code
Provision your pipeline infra (buckets, warehouse) with Terraform.
- Study primary concepts for: Infrastructure as Code
- Actionable guide: Provision your pipeline infra (buckets, warehouse) with Terraform.
- Build or test a practical mini-project applying Infrastructure as Code.
-
Portfolio Pipeline
Publish an end-to-end pipeline project with documentation and architecture diagram.
- Study primary concepts for: Portfolio Pipeline
- Actionable guide: Publish an end-to-end pipeline project with documentation and architecture diagram.
- Build or test a practical mini-project applying Portfolio Pipeline.
Related Career Paths & Cross-Skill Roadmaps
Expand your engineering versatility by exploring interconnected role roadmaps:
Track Your Data Engineer Roadmap Progress
Sign in with Google to check off completed topics, save code notes, and monitor your placement readiness score.
Start Tracking Roadmap