Week 7 - Mid-Track Project

Project Brief

Project Requirements

Week 7 Gotchas & Pitfalls

Project Brief

This is your mid-track project. You have one week to design, build, and deploy a data pipeline that runs as a Container App Job on Azure and stores its results in Azure storage. You pick the data source and use case. The technical requirements and deadline are in the chapters below.

The week ends with a technical interview where you present your project and explain your decisions.

What you are building

A complete data pipeline that:

1. Ingests data from a live external API
2. Validates the data before storing it (Pydantic)
3. Transforms the validated data using pandas before storage
4. Stores results in Azure Postgres (rows) and Azure Blob Storage (raw JSON)
5. Runs as a Container App Job on Azure on a daily schedule

This is the same architecture you built in Week 6, extended with a pandas transform step, a data source you choose yourself, and a daily CRON schedule so the pipeline runs automatically without you triggering it.

┌─── Azure Container App Job ──────────────────────────────────────────────────┐
│  External API ──► pipeline.py ──► Pydantic ──► pandas ──► Azure Storage      │
└──────────────────────────────────────────────────────────────────────────────┘

<aside> 💡 In the wild: Open-source tools like dlt (data load tool) follow the same fetch-validate-store pattern you are building this week. Your project is a simplified version of what production data teams run every day.

</aside>

Choosing your data source

Pick something you find interesting. The best projects come from genuine curiosity. Your data source must be:

• Free to access (no paid API keys)
• Reliable (available when you need it, not flaky or rate-limited to 10 requests/day)
• Returns structured data (JSON or CSV you can parse)

<aside> ⚠️ Verify your data source works on Day 1. Call the API, inspect the response, and confirm you can parse it. Do not discover on Day 4 that the API requires OAuth or returns HTML instead of JSON.

</aside>

Example project ideas

These are starting points, not requirements. You can combine ideas or invent your own.

Live APIs

These return fresh data on every run: good for pipelines that run on a schedule.

Static datasets (bonus)

These are snapshot CSV files you can add on top of your API pipeline as a bonus. They don't change between runs, so they work best as a second data source alongside your live API: load the CSV once to seed a reference table, then join or enrich your API data against it. Good for practising data cleaning with pandas when the point is the transformation, not the freshness.

<aside> 💡 If you are stuck choosing, start with Open-Meteo: no API key, clean JSON, and the Week 6 examples already use it. A note on the others: Football-Data.org requires a free key (register at football-data.org, takes 2 minutes, store the key as an env var); the GitHub API allows 60 unauthenticated requests per hour, which is tight during development: create a free Personal Access Token to raise it to 5,000.

</aside>

You are not limited to this list. If you have a different API in mind, check with your teacher before you start building.

<aside> ⚠️ Using your own API? Your teacher will verify it has enough structure to warrant real Pydantic validation and pandas work. If the response is too flat or simple, your teacher may ask you to also add one of the bonus CSV datasets above so the transform step has enough to work with.

</aside>

Scope guidance

Keep it focused. A working pipeline with one data source and one storage target is a better project than an ambitious plan that is half-finished.

Good scope:

• One live API, one pipeline script, one table in Postgres, raw results in Blob Storage
• Fetches data, validates with Pydantic, transforms with pandas, inserts rows, logs the count
• Runs as a Container App Job on Azure

Too ambitious for one week:

• Three APIs merged into a data warehouse with transformations and a dashboard
• Real-time streaming with event triggers
• A web frontend that displays the data

You can always add stretch goals after the core pipeline works end to end.

Timeline

<aside> ⌨️ Hands on: Deploy the hello-world container on Day 1, before your pipeline code is written. This proves your Azure setup works early. If you hit firewall issues or image pull errors, you have four days to fix them instead of four hours.

</aside>

When you are ready to start coding, see Chapter 2: Project Requirements for the starter template and the requirements checklist.

<aside> 💡 Using AI to help: Use an LLM to help you explore API documentation, generate Pydantic models from example JSON responses, or draft your README. Document what you used in AI_ASSIST.md. (⚠️ Ensure no PII or sensitive company data is included!)

</aside>

Extra reading

• Public APIs list: curated directory of free APIs for projects

Next up: Project Requirements, where you will find the minimum requirements checklist and what a complete project run looks like.

The HackYourFuture curriculum is licensed under CC BY-NC-SA 4.0 *https://hackyourfuture.net/*

Built with ❤️ by the HackYourFuture community · Thank you, contributors

Found a mistake or have a suggestion? Let us know in the feedback form.