Introduction to Data Pipelines
Configuration & Secrets (.env)
Separation of Concerns (I/O vs Logic)
Dataclasses for Data Objects
OOP vs Functional Programming
Functional Composition
Testing with Pytest
Practice
Assignment: A Clean Pipeline
Gotchas & Pitfalls
Back to Track
9. Assignment: Refactoring to a Clean Pipeline
Content coming soon...
Concepts to Cover
- Refactor a "God Script" into a modular pipeline: config, data types, cleaning, orchestration
- Separation of concerns: parsing, validation, and loading as distinct steps
- Dataclasses for structured records
- Pure functions for cleaning logic
- Comprehensive test coverage for transformation functions
- No hardcoded paths, no global variables, full type hint coverage
*https://hackyourfuture.net/*
Found a mistake or have a suggestion? Let us know in the feedback form.