Week 2 - Structuring Data Pipelines

Introduction to Data Pipelines

Configuration & Secrets (.env)

Separation of Concerns (I/O vs Logic)

Dataclasses for Data Objects

OOP vs Functional Programming

Functional Composition

Testing with Pytest

Practice

Assignment: A Clean Pipeline

Gotchas & Pitfalls

• Teachers

Back to Track

3. Separation of Concerns (I/O vs Logic)

Goal: Architecture that makes testing easy.

Concepts to Cover

The Pattern*: Input -> Business Logic -> Output.

*The "Humble Object"**: Keep the I/O layer (reading files, API calls) thin and devoid of logic.

Business Logic*: Pure functions that transform data (easy to test).

Dependency Injection (Light): Passing data into functions rather than fetching it inside*.

Repository Pattern*: Abstracting "Fetching Data" from "Using Data".

Suggested Exercises

• Refactor a "God Function" that reads, cleans, and writes, into 3 separate functions.

*https://hackyourfuture.net/*

Found a mistake or have a suggestion? Let us know in the feedback form.