Azure Setup and Account Access
Week 1 Assignment: The Data Cleaning Pipeline
Career relevance: Week 1 in the NL data job market
Going Further: Optional Deep Dives
In this section, you'll set up a professional Python development environment. Having a consistent, reproducible setup is crucial for data engineering work - it ensures your code runs the same way everywhere.
Python has become the lingua franca of the data world. Whether you are building complex data pipelines, training machine learning models, or automating cloud infrastructure, Python is likely the tool you'll use.
Python dominates the data landscape for three main reasons:
As a Data Engineer, you will use Python for:
By learning Python, you're not just learning a programming language; you're gaining the ability to orchestrate the entire lifecycle of data.
Where Python sits in the language hierarchy: Python (interpreted, high-level) sits at the top above C/C++/Rust (compiled), Assembly Language, Machine Code (0s and 1s), and Hardware (CPU, RAM) at the bottom.
Think of programming languages as layers in a pyramid. At the very bottom is hardware, the physical CPU and RAM. Just above that is machine code, the raw 0s and 1s that your processor understands. Then comes assembly language, a slightly more human-readable form of machine code. Climbing higher, we find compiled languages like C, C++, and Rust, which translate your code into machine instructions before running.
At the top of this pyramid sits Python. It's an interpreted, high-level language. "High-level" means Python handles a lot of the low-level details (like memory management) for you, so you can focus on the problem you're solving, not the machine you're running on. "Interpreted" means your Python code is translated and executed line-by-line at runtime, rather than being compiled into machine code beforehand. This makes it incredibly flexible and fast to iterate with, which is exactly what you want when building data pipelines.
<aside> 🤓 The Curious Geek: Want to know about the difference between compiled and interpreted languages? Check out Compiled vs Interpreted Languages on freeCodeCamp.
</aside>
Python was created by Dutch Programmer Guido van Rossum and first released in 1991. Named after Monty Python (not the snake!), it was designed to be readable and beginner-friendly while remaining powerful enough for complex applications.
Key milestones:
| Year | Event |
|---|---|
| 1991 | Python 0.9.0 released |
| 2000 | Python 2.0 - list comprehensions, garbage collection |
| 2008 | Python 3.0 - major backwards-incompatible release |
| 2020 | Python 2 officially sunset (end of life) |
| 2022 | Python 3.11 - significant performance improvements (minor bump) |
<aside> 🤓 Curious Geek: Reading version numbers
Most modern software follows semantic versioning: MAJOR.MINOR.PATCH. Bump MAJOR for breaking changes, MINOR for new features that keep old code working, PATCH for bug fixes. So 3.11.12 → 3.11.13 is a safe bug-fix; 3.11 → 3.12 adds features without breaking your scripts; 2 → 3 is the rare breaking jump that took the Python world about a decade to migrate. Full spec at semver.org.
</aside>
The Python 2 → 3 jump is exactly the kind of major-version migration semver warns about. It is also why the next callout matters in 2026:
<aside> ⚠️ Never use Python 2. It's been dead since January 2020. If you see python vs python3 on your system, always use python3. Some old tutorials still reference Python 2 - ignore them.
</aside>
In data engineering, Python versions are critical because:
This is why we use virtual environments (covered below) and always specify Python versions.
<aside> 🎬 Want more history? Watch Python: The Documentary (1h 30m) - interviews with Guido van Rossum and how Python became one of the world's most popular languages.
</aside>
https://www.youtube.com/watch?v=GfH4QL4VqJ0
<aside> ⚠️ Even if you have Python installed from the Core program, make sure you have Python 3.11 specifically. Data engineering projects often require specific Python versions for compatibility.
</aside>
The recommended way to install Python on macOS is using Homebrew:
# Install Homebrew if you don't have it
/bin/bash -c "$(curl -fsSL <https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh>)"
# Install Python 3.11
brew install [email protected]
# Verify installation
python3.11 --version
# Python 3.11.12
python --version
# Python 3.11.12
sudo apt update
sudo apt install python3.11 python3.11-venv python3-pip
# Verify installation
python3.11 --version
# Python 3.11.12
<aside> 💡 Virtual environments isolate your project's dependencies from other projects and your system Python. This is essential for reproducible data pipelines.
</aside>
A virtual environment is like a clean room for your Python project. Each project gets its own isolated set of packages.
How a virtual environment isolates a project: each project gets its own pinned package versions
# Navigate to your project folder
cd my-data-project
# Create a virtual environment named 'venv'
python3.11 -m venv venv
# Activate it (macOS/Linux)
source venv/bin/activate
# Activate it (Windows PowerShell)
.\venv\Scripts\Activate.ps1
# Your prompt should now show (venv)
<aside>
💡 Pro Tip: The modern way with uv
While venv and pip are standard, many professional Data Engineers now use uv. It's an incredibly fast Python package manager written in Rust.
If you have uv installed, you can replace the steps above with:
uv venv # Create venv
uv pip install X # Install X
Or even better, use uv run script.py to run a script with its dependencies automatically! We use uv to manage this repository.
</aside>
This pattern of managing dependencies mirrors the workflow you used in the Core program with npm and package.json.
<aside>
📘 Core program connection: In the Core program with JavaScript you used npm, package.json, and package-lock.json to install packages and keep versions reproducible. In Python, uv solves a very similar problem with pyproject.toml and uv.lock. The exact tools are different, but the goal is the same: every machine and CI runner should install the same dependency set. Refresh the Core program chapter here: https://www.notion.so/hackyourfuture/Package-managers-2b250f64ffc9800d8c76e5fec3aa8095
</aside>
With your virtual environment activated:
# Install a package
pip install pandas
# Install multiple packages
pip install pandas numpy
# Save your dependencies to a file
pip freeze > requirements.txt
# Install from requirements file (on another machine)
pip install -r requirements.txt
The venv folder is a per-machine artifact: thousands of files, hundreds of megabytes, and not portable across operating systems. Committing it bloats the repo and stops being correct the moment a teammate on a different OS clones the project.
Commit the recipe, not the cooked meal:
requirements.txt (or uv.lock). A small text file that lists exactly which packages and versions a fresh venv should install.venv/ (or .venv/). Add it to .gitignore instead.Create a .gitignore file at the root of your project:
# Python virtual environment
.venv/
venv/
# Python bytecode
__pycache__/
*.pyc
A teammate who clones your repo then reproduces your environment in two commands:
python3.11 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt