Week 6 - Cloud and Azure Essentials
Introduction to Cloud and Azure
Week 6 Assignment: Deploy Your Pipeline to Azure
The Azure CLI (command-line interface) is your primary tool for managing cloud resources. As a Data Engineer, you will often be working in the terminal with CLI tooling. They are often more powerful than the Portal (website interface) that cloud providers offer. This week you will use both, so this chapter builds the fluency you need before working with storage accounts, databases, and container jobs.
By the end of this chapter, you should be able to log in, navigate resources, and run basic commands with confidence.
Cloud platforms like Azure, AWS, and Google Cloud all provide two ways to manage resources: a web-based portal (GUI) and a command-line interface (CLI). The portal is visual and good for exploring, but the CLI is where the real power is.
Why? Because CLI commands are scriptable and repeatable. When you create a database through a portal, you click through forms and hope you remember the settings next time. When you do it with a CLI command, you have an exact record of what you did. You can put that command in a script, run it again in a different environment, or share it with a colleague. This is the same principle behind Infrastructure as Code, which professional teams use to manage cloud environments, and we will cover in the last week of the track. For now, we will use the cli tool.
Every major cloud has its own CLI: AWS has the aws CLI, Google Cloud has gcloud, and Azure has az. The patterns are similar across all of them: <cli> <service> <action> --flags. If you learn to work with az, picking up aws or gcloud later will feel familiar. For the history of how cloud management went from portals to APIs to CLIs, see History of Cloud Computing.
As a Data Engineer, you will spend more time in the terminal than in the portal. Pipelines run on servers that have no screen, CI/CD systems execute CLI commands, and debugging often means checking logs from the command line. Building CLI fluency now pays off throughout your career.
Before you can do anything, authenticate with Azure:
az login
This opens a browser window for authentication. After login, the CLI knows who you are and which subscriptions you can access.
If you have multiple subscriptions, set the one you want to work with:
# List subscriptions
az account list --output table
# Set the active subscription
az account set --subscription "<SUBSCRIPTION_ID>"
Your teacher will tell you which subscription to use.
Azure services cost money. In a real project, every resource you create adds to the bill. For this track, your teacher has set up a custom role that prevents you from creating infrastructure like resource groups, storage accounts, databases, or environments. You work with what is already provisioned.
The one exception is Container App Jobs, which you can create yourself. Jobs do have cost implications, so chapter 5 explains how to keep them under control. Chapter 6 covers cost awareness more broadly.
A resource group is a logical container for related services. Your teacher has created a shared resource group for the class. Everything you use this week lives there.
# List resource groups
az group list --output table
Resource groups follow a naming pattern like rg-<project>-<env> (for example rg-weather-dev). You will see this in the shared resource group your teacher set up.
The CLI supports several output formats. Use the one that fits your task:
# Human-readable table
az group list --output table
# Machine-readable JSON (default)
az group list --output json
# Tab-separated values (easy to pipe)
az group list --output tsv
--output table is best for quick inspection. --output json is best when you need to parse values in a script.
Every command has a --help flag:
az containerapp job create --help
If you are unsure which command to use, start broad:
az find "create container app job"
<aside>
💡 Using AI to help: If you are stuck, paste your goal into an LLM and ask it to suggest the az command. (⚠️ Ensure no PII or sensitive company data is included!)
</aside>
Verify the suggested command with --help before running it.
The Azure portal (portal.azure.com) is a web-based dashboard. Use it to:
The portal is useful for visual inspection, but the CLI is faster for repeated tasks.
<aside> ⌨️ Hands on: Log in to the Azure portal, navigate to your subscription, and find the resource group your teacher created. List the resources inside it.
</aside>
Here is a preview of the CLI commands you will run in later chapters:
# Blob storage
az storage account list --resource-group rg-weather-dev --output table
# PostgreSQL
az postgres flexible-server list --resource-group rg-weather-dev --output table
# Container Apps
az containerapp job list --resource-group rg-weather-dev --output table
# Container Registry (from Week 5)
az acr repository list --name hyfregistry --output table
You do not need to memorize these. Each chapter introduces the commands you need. The point is: the CLI follows a consistent pattern of az <service> <action> --flags.
<aside> 📘 Core program connection: You used the CLI in the Core program. The same command-line habits apply here. Refresher: https://www.notion.so/hackyourfuture/Intro-to-the-CLI-2ab50f64ffc9814e8096d8a20725a23f
</aside>
Every Azure resource has a unique ID. You will see these in logs, error messages, and permission settings.
/subscriptions/<SUB_ID>/resourceGroups/<RG>/providers/Microsoft.Storage/storageAccounts/<NAME>
You do not need to construct these yourself, but recognizing the pattern helps when debugging.
<aside> 🤓 Curious Geek: Azure CLI under the hood
The Azure CLI is a Python application that wraps the Azure REST API. Every az command translates to one or more HTTP calls. You can see the raw requests by adding --debug to any command. If you want, you can inspect the code since it is open source.
</aside>
az group list --output table and identify the shared resource group.az resource list --resource-group <name> --output table to see what the teacher pre-provisioned.az containerapp job create help page and list three required flags.az containerapp job create and get an error you do not understand. What are two ways to get more information about the command and its flags?--output table and --output json, and when would you choose each?az --query, if you want to filter JSON output laterThe HackYourFuture curriculum is licensed under CC BY-NC-SA 4.0 *https://hackyourfuture.net/*
Built with ❤️ by the HackYourFuture community · Thank you, contributors
Found a mistake or have a suggestion? Let us know in the feedback form.