Week 2 - Structuring Pipelines
Introduction to Data Pipelines
Configuration & Secrets (.env)
Linting and Formatting with Ruff
Week 2 Assignment: Clean Pipeline
This page is optional. Nothing here is required for Week 2's learning goals or the assignment. Use it after you finish the week if you want to keep learning, or come back later when a specific topic from Week 2 starts mattering more in your day-to-day work.
Sections are grouped by topic: architecture and design (SOLID and clean-code longer reads), functional Python (composition, toolz, and friends), testing in depth (the longer route into pytest and TDD), what comes next (forward-references to weeks that build on this one), setup and tooling beyond the basics (the next layer of dev tools), books, videos, and community.
<aside> 💡 Links that already appear inside a Week 2 chapter's Extra reading section are not duplicated here. This page is the home for resources that are too broad to fit any single chapter, or that go meaningfully deeper than the chapter's tightly-scoped reading.
</aside>
pipe(), curry(), and groupby() helpers are the production version of the pipe function you wrote in Functional Composition.Maybe, Result, and IO types from Haskell-flavoured FP into Python. Worth a look once the composition pattern feels natural and you want to handle errors functionally instead of with exceptions.functools: the standard-library home for reduce, partial, lru_cache, cache, and wraps. The chapter showed reduce; partial and lru_cache are the two next ones worth knowing well.The patterns from this week show up almost immediately in later weeks. These are the named landings, with one-line previews so you can recognise them when you arrive.
@dataclass. Same field-declaration shape; replaces hand-written __post_init__ checks with declarative validators that emit field-path-aware error messages. The mCoding video Why Pydantic Will Replace Dataclasses (15 min) is the right teaser..env. Same os.environ.get(...) interface from your code's point of view, but the secret lives in a managed vault with access policies, automatic rotation, and an audit log of who read what when. Microsoft Learn: Key Vault overview is the canonical entry point.pip install-able with a CI pipeline that runs ruff check + pytest on every push. The natural follow-up once one project's worth of pure-functions-and-tests starts getting reused across multiple pipelines.Once you are comfortable with the Week 2 stack (python-dotenv, @dataclass, pytest, ruff), these are the next-tier tools that show up in NL postings and production codebases.
ruff format + ruff check (and any other linters you configure) automatically before every commit. Catches style and lint issues at the moment you'd be most annoyed to find them in CI.pytest tests actually exercise. Run pytest --cov=src once and you'll learn which branches you've never tested.pip + venv replacement from the same team that built ruff. Installs dependencies in seconds where pip takes minutes; drop-in compatible with requirements.txt and pyproject.toml.__init__ and one other method, write a function instead" argument. Reinforces the OOP vs Functional Programming decision rule from the chapter.#help channels get fast, high-quality answers.The HackYourFuture curriculum is licensed under CC BY-NC-SA 4.0 *https://hackyourfuture.net/*
Built with ❤️ by the HackYourFuture community · Thank you, contributors
Found a mistake or have a suggestion? Let us know in the feedback form.