Week 3 - Ingesting and Validating Data

Introduction to Data Ingestion

Ingesting from APIs

Production Error Handling

Reading Multiple File Formats

Data Validation with Pydantic

Writing to Databases

Gotchas & Pitfalls

Practice

Assignment: Build a Validated Ingestion Pipeline

Career relevance: Week 3

Week 3 Glossary

Going Further: Optional Deep Dives

Week 3 Kickoff Slides

History: APIs and Data Transfer

📚 Going Further: Optional Deep Dives

This page is optional. Nothing here is required for Week 3's learning goals or the assignment. Use it after you finish the week if you want to keep learning, or come back later when a specific topic from Week 3 starts mattering more in your day-to-day work.

Sections are grouped by topic: full courses and tutorials for the long-form route, deep dives on specific topics for going one layer below this week's chapters, declarative ingestion for the production frameworks (dlt, Airbyte, Meltano) that wrap the patterns you built by hand, videos for tangential talks, and community and books for the bigger picture.

<aside> 💡 Links that already appear inside a Week 3 chapter's Extra reading section are not duplicated here. This page is the home for resources that are too broad to fit any single chapter, or that go meaningfully deeper than the chapter's tightly-scoped reading.

</aside>

Full courses and tutorials

The chapters' Extra reading sections deliberately stay short. The big ones live here.

• DataTalks.Club: Data Engineering Zoomcamp: free multi-week course. Module 1 (Docker and Postgres) and Module 2 (Workflow Orchestration with Kestra) are the natural extensions of Week 3's ingestion-to-database flow.
• Real Python: API Integration in Python: long-form tutorial that takes the requests patterns of Ch2 further into auth, pagination, and rate-limit handling.
• Pydantic V2 Migration Guide: the official walk-through of the V1 → V2 changes. Useful if you join a team with code on both sides of the jump.
• Talking to APIs (FastAPI tutorial, first 6 chapters): not because you will build APIs as a junior DE, but because reading the FastAPI tutorial cements the Pydantic patterns you learned in Ch5.

Deep dives on specific topics

One layer below what each chapter taught.

• The Apache Parquet specification: how columnar storage actually works on disk (row groups, column chunks, dictionary and run-length encoding). Read after Ch4 if you ever need to debug a corrupt Parquet file.
• SQLite: Write-Ahead Logging: how SQLite handles concurrent reads during writes. The page that explains why SQLite is more production-grade than its reputation suggests.
• httpx: async HTTP client: the modern async alternative to requests. Same API shape, but lets you fetch from 50 APIs in parallel. Worth meeting once you have multiple ingestion sources.
• Real Python: Python's requests library deep dive: sessions, adapters, and urllib3.Retry. The chapter touched on requests; this is the rest.

Declarative ingestion: dlt, Airbyte, Meltano

Week 3 builds an ingestion pipeline by hand on purpose: you need to feel the pain before you reach for the tools. In production, teams often hand the boilerplate to a declarative framework.

• dlt (data load tool): Python library where you declare sources and destinations and dlt handles schema inference, normalization, and incremental loads. NL data community is very active on this one.
• Airbyte: connector-based ingestion platform with 300+ pre-built sources. Common in Dutch data-platform teams that want to avoid maintaining bespoke connector code.
• Singer / Meltano: the open standard for tap (extract) and target (load) plugins. Older than dlt but still widely used.

Videos

Tangential talks that do not belong inside a specific chapter's Extra reading.

• DataTalks.Club: Data Engineering Zoomcamp 2024: hour-long overview of the data-engineering landscape, covering what the Zoomcamp builds end-to-end. Pairs with Ch1's framing of the ingestion pipeline.
• Corey Schafer: Python Context Managers: the deep version of the with statement and context-manager pattern you used in Ch6 for database connections.

Community

• dlt Slack community: the dlt project's community channel. Useful if you adopt declarative ingestion at work.
• Pydantic Help and Support: official help channels (GitHub Discussions, Discord).
• r/dataengineering: active subreddit. NL data-job market threads show up regularly.

Books

• Fundamentals of Data Engineering by Joe Reis and Matt Housley: the canonical introduction to data-engineering work. The chapter on "ingestion patterns" is exactly the Week 3 territory at career-spanning depth.
• Designing Data-Intensive Applications by Martin Kleppmann: the long-form analogue of Week 3's storage and validation chapters. Chapters 4 (Encoding) and 5 (Replication) are the deep version. Read after Week 6.

What comes next

Week 4 is Pandas for in-memory data processing; Week 5 hardens the pipeline shape; Week 6 is the first cloud-deployed pipeline on Azure. The ingestion habits you built this week (timeouts, retries, validation, idempotent writes) are the foundation every later week assumes you can write without thinking.

The HackYourFuture curriculum is licensed under CC BY-NC-SA 4.0 *https://hackyourfuture.net/*

Built with ❤️ by the HackYourFuture community · Thank you, contributors

Found a mistake or have a suggestion? Let us know in the feedback form.