Week 2 - Structuring Data Pipelines

Introduction to Data Pipelines

Configuration & Secrets (.env)

Separation of Concerns (I/O vs Logic)

Dataclasses for Data Objects

OOP vs Functional Programming

Functional Composition

Testing with Pytest

Practice

Assignment: A Clean Pipeline

Gotchas & Pitfalls

• Teachers

Back to Track

2. Configuration & Secrets (.env)

Goal: Stop hardcoding sensitive data and learn professional configuration management.

Concepts to Cover

The Problem*: Why API_KEY="123" in git is a security disaster.

Environment Variables*: OS-level variables accessed via os.environ.

The .env Pattern*: Using python-dotenv to load local secrets.

Config Module*: Creating a config.py to centralize settings (e.g., from config import DB_URL).

Gitignore*: Ensuring .env never gets committed.

Suggested Exercises

• Create a script that fails if an API key is missing.

• Refactor a script with hardcoded paths to use a .env file.

*https://hackyourfuture.net/*

Found a mistake or have a suggestion? Let us know in the feedback form.