Projects, Python and Datasets

Technical interviews for data roles are usually a take-home assignment or a conversation about a project you built. Your own projects on GitHub prepare you for both, so they are the preferred way to keep practicing.

Projects

Build your own project

Pick a dataset or API from the Datasets list below and build a small pipeline around it: ingest, validate, transform, store, and show the result. Use the Python and SQL links on this page when you get stuck, and think about the system design choices as you go: where the data lives, what happens when a load fails, and how the pipeline runs every day.

Start small and finish a first version, then add one feature at a time.
Write a README that explains what the project does, how to run it, and why you made your choices.
Push to a public repository on your own GitHub account, so employers can see it.

Keep your final project running

Your HYF student account and the shared class resources (PostgreSQL, Metabase, Airflow, Databricks) are not available forever. It might be a good idea to make sure your project is running somewhere outside the class infrastructure and you can do this quite easily.

Push all code to a repository on your own GitHub account, and export work that only lives in the cloud, such as Databricks notebooks. Keep secrets out of public repositories, as in Week 2.
Run the stack on your own laptop with Docker Compose: the official PostgreSQL image, Airflow in Docker, and Metabase on Docker. dbt Core runs against the same database, or against a local file with dbt-duckdb.

Free places to host parts of the project:

GitHub Actions: Free for public repositories. A scheduled workflow can run your ingestion and dbt once a day.
Databricks Free Edition: Free, for learning and personal projects.
Azure free account: 12 months of free amounts for popular services, plus services that stay free. Sign-up asks for a payment card, so set a budget alert as in Week 6.

Datasets

Use these to find data for your own practice project.

Awesome Public Datasets: Free. A large list of open datasets on GitHub, grouped by topic.
Public APIs: Free. A list of free APIs on GitHub, useful to practice ingestion as in Week 3.

Databricks

Orchestrate and Run Your dbt Jobs: Tutorial by Databricks on running dbt workloads on their platform, useful to learn the Databricks-native way.

Python

Exercism Python track: Free. Small exercises with optional feedback from volunteer mentors.
Real Python learning paths: Many articles are free, some content needs a subscription.

dbt

[DataGym.io](https://transformation-lab.datagym.io/?lang=en): hands-on dbt exercises in the browser. Very recent site that uses a real dbt project within a vscode-like environment to teach fundamentals of dbt.
## SQL

- [PostgreSQL Tutorial](https://neon.com/postgresql/tutorial): Free. PostgreSQL-focused reference with exercises. Good match for your own Docker Postgres or dbt projects on Postgres.
- [DataLemur SQL questions](https://datalemur.com/questions): Free account. Short, interview-style problems (joins, aggregates, windows, CTEs) aimed at data analyst and data engineer roles.

The HackYourFuture curriculum is licensed under CC BY-NC-SA 4.0 https://hackyourfuture.net/

CC BY-NC-SA 4.0 Icons

Built with ❤️ by the HackYourFuture community · Thank you, contributors

Found a mistake or have a suggestion? Let us know in the feedback form.