Week 2 - Structuring Data Pipelines

Introduction to Data Pipelines

Configuration & Secrets (.env)

Separation of Concerns (I/O vs Logic)

Dataclasses for Data Objects

OOP vs Functional Programming

Functional Composition

Testing with Pytest

Practice

Assignment: A Clean Pipeline

Gotchas & Pitfalls

• Teachers

Back to Track

6. Functional Composition

Goal: Moving from class-heavy OOP to clean, testable functional pipelines.

Concepts to Cover

Pure Functions*: Same input = same output, no side effects.

Composition*: Chaining functions (Result of A -> Input of B).

Immutability*: Why modifying data in-place (e.g., list.append) causes bugs in pipelines.

Type Hints for Functions*: Using Callable, Iterable in signatures.

When to use Classes*: Only for holding state (Dataclasses), not for logic/actions.

Suggested Exercises

• Refactor a stateful class Processor into a series of standalone functions.

• Build a small transformation pipeline: read -> clean -> standardize -> write.

*https://hackyourfuture.net/*

Found a mistake or have a suggestion? Let us know in the feedback form.