Design, build and maintain robust, scalable data pipelines. Master Airflow, Spark, modern architectures and the cloud to become an operational Data Engineer.
Data Lake, Data Warehouse, Lakehouse, ETL vs ELT and dimensional modelling.
Overview of the tools and the Data Engineer's role in the company.
Advanced Pandas, batch processing, connecting to sources (APIs, databases, files).
Automation, logging, error handling and ETL best practices.
Core concepts, building DAGs, scheduling and dependencies.
Retries, monitoring, alerts and complex DAGs.
Spark Core, DataFrames, Spark SQL and distributed processing.
Performance tuning and running on a cluster.
SQL/NoSQL databases, cloud warehouses and object storage.
Complete end-to-end pipeline, testing, industrialisation and presentation.
Ready to build robust data pipelines?