Introduction to Data Pipelines
Configuration & Secrets (.env)
Separation of Concerns (I/O vs Logic)
Dataclasses for Data Objects
OOP vs Functional Programming
Functional Composition
Testing with Pytest
Practice
Assignment: A Clean Pipeline
Gotchas & Pitfalls
Back to Track
1. Introduction to Data Pipelines
Concepts to Cover (suggestions)
- What is a data pipeline and why it matters in practice
- ETL vs ELT patterns: Extract-Transform-Load vs Extract-Load-Transform
- Real-world pipeline examples: batch processing, streaming, scheduled jobs
- Pipeline architecture: source → transformation → storage → consumption
- Common pipeline challenges: data quality, schema evolution, failure recovery
- Tools in the ecosystem: Apache Airflow, dbt, custom Python solutions
*https://hackyourfuture.net/*
Found a mistake or have a suggestion? Let us know in the feedback form.