Week 2 - Structuring Pipelines

The HackYourFuture curriculum is licensed under CC BY-NC-SA 4.0 *https://hackyourfuture.net/*

Built with ❤️ by the HackYourFuture community · Thank you, contributors

Found a mistake or have a suggestion? Let us know in the feedback form.

Introduction to Data Pipelines

Configuration & Secrets (.env)

Separation of Concerns

OOP vs Functional Programming

Dataclasses for Data Objects

Functional Composition

Testing with Pytest

Linting and Formatting with Ruff

Practice

Gotchas & Pitfalls

Week 2 Assignment: Clean Pipeline

Week 2 Glossary

Going Further

Career relevance: Week 2

Week 2 Kickoff Slides

Week 2 - Structuring Pipelines

Welcome to Week 2! Now that you know the Python basics, it's time to move from "writing scripts" to "building engineering systems." This week is all about architecture. You will learn how to structure your code so that it is readable, testable, and robust against the messy reality of production data.

By the end of this week, you will have refactored a messy "god script" into a professional, modular pipeline that survives missing files, malformed rows, and missing config: configuration, data modeling, and business logic each live in their own module.

Learning goals

• Understand the core architecture of a production data pipeline (ETL vs ELT patterns)
• Manage configuration and sensitive secrets using .env files and a centralized config.py module
• Use Python Dataclasses to define structured data models and move away from generic dictionaries
• Distinguish between Object-Oriented and Functional programming paradigms and know when to apply each in a data context
• Apply Functional Composition to build readable and reusable transformation logic
• Apply Separation of Concerns to decouple I/O operations (files/APIs) from transformation logic
• Write and execute automated unit tests using the Pytest framework
• Lint and format Python code using ruff, and configure format-on-save in VS Code
• Refactor messy, stateful code into a clean, modular, and well-tested engineering pipeline

Chapters

1. Introduction to Data Pipelines
2. Configuration & Secrets
3. Separation of Concerns
4. OOP vs Functional Programming
5. Dataclasses for Data Objects
6. Functional Composition
7. Testing with Pytest
8. Linting and Formatting with Ruff
9. Practice
10. Gotchas & Pitfalls
11. Assignment

Supplementary

• Glossary: every term introduced this week, in chapter order, with stable anchors.
• Going Further: optional deeper resources: books, full courses, advanced videos.
• Career relevance: Week 2: how this week's skills show up in NL data postings.