Introduction to Cloud and Azure

Your pipeline is containerized. Now you need a place to run it reliably and at scale. The cloud gives you compute, storage, and managed services without maintaining servers yourself.

By the end of this chapter, you should understand what cloud services are, how Azure organizes them, and why this week builds directly on Week 5.

Concepts

What the cloud gives data pipelines

Cloud services give you:

Scalability: run bigger jobs without buying hardware
Reliability: managed services with backups and monitoring
Speed: provision resources in minutes

Cloud service types

Cloud providers offer services at different levels of abstraction:

IaaS (Infrastructure as a Service): you rent virtual machines and manage the operating system, runtime, and application yourself. Example: Azure Virtual Machines.
PaaS (Platform as a Service): the provider manages the infrastructure and runtime. You deploy your code or container. Example: Azure Container Apps, Azure Database for PostgreSQL.
SaaS (Software as a Service): ready-to-use applications. Example: Microsoft 365, Slack.

<aside> 🖼️ Visual: Cloud Service Types

</aside>

This week you will use mostly PaaS services: Azure Database for PostgreSQL and Azure Container Apps. PaaS is the sweet spot for data engineering because you focus on your pipeline, not on patching servers.

Why this course uses Azure

The three major cloud providers are Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform (GCP). All three offer compute, storage, networking, and managed databases. The skills you learn this week (working with CLIs, deploying containers, connecting to managed databases) transfer between clouds; each provider has its own naming and quirks, but the concepts are the same.

This course uses Azure because it has a dominant market share in business and enterprise in the Netherlands. The ACM (Authority for Consumers and Markets) estimates that Microsoft and AWS each hold 35% to 40% of the market. Many Dutch companies already have Microsoft contracts (Office 365, Active Directory), which makes adopting Azure easier than bringing in a new vendor. If your future employer uses AWS or GCP instead, you will recognize the patterns.

<aside> 📘 Core program connection: In the Core program Systems week you learned the basics of cloud computing. This week focuses on Azure services for data pipelines. Refresh here: https://www.notion.so/hackyourfuture/Cloud-computing-2d750f64ffc9801ca8a7f14dfe7d513d

</aside>

How Azure is organized

Azure has a clear hierarchy for organizing resources:

Tenant (your organization)
└── Subscription (billing boundary)
    └── Resource group (logical container for a project)
        ├── Azure Database for PostgreSQL
        ├── Azure Container Apps environment
        └── Azure Container Registry

<aside> 🖼️ Visual: Azure Resource Hierarchy

</aside>

Tenant: the top-level identity for your organization. In this course, HackYourFuture is the tenant.
Subscription: controls billing and access. Your Azure for Students subscription lives under the shared tenant.
Resource group: a logical container that groups related resources. When you delete a resource group, everything inside it is deleted too.
Region: each resource group and resource is created in a specific Azure region (like westeurope). Keep all resources for a project in the same region to reduce latency and cost.

<aside> ⌨️ Hands on: Open the Azure portal, navigate to your subscription, and find the resource group your teacher created. Identify which region it uses.

</aside>

Shared responsibility

Azure manages the infrastructure, but you still manage:

Your data quality: Azure ensures the database server is running, but if your Python script inserts duplicated or null rows into PostgreSQL, Azure won't fix it.
Access permissions: Azure provides the login system, but you must explicitly grant your Container App permission to read from the database using identities or connection strings.
Configuration values: Azure runs your container, but you must provide the correct environment variables (like API keys) so your code knows what to do.
Cost visibility: Azure bills you for what you use. If you accidentally leave a heavy process running in a loop all weekend, you are responsible for the bill.

<aside> 🤓 Curious Geek: The first cloud launch

Amazon launched EC2 in 2006, which popularized pay-as-you-go compute. Azure followed soon after and now offers hundreds of managed services. For a deeper look at how cloud computing evolved -- from bare metal to VMs to serverless, and from CapEx to pay-per-second billing -- see History of Cloud Computing.

</aside>

Exercises

Explain the difference between IaaS and PaaS with one example each.
List three reasons cloud platforms are useful for data pipelines.
Describe how Week 5 containers fit into Week 6 deployments.

🧠 Knowledge Check

Your pipeline needs to run once a day and write results to a database. Would you choose IaaS or PaaS for the database, and why?
What happens to all the resources inside a resource group when you delete the resource group?
A colleague says "the cloud is just someone else's computer." What nuance does this miss about managed services like Azure Database for PostgreSQL?

Extra reading

Azure Architecture Guide: reference for designing cloud solutions
What is Azure?: official overview of Azure services

The HackYourFuture curriculum is licensed under CC BY-NC-SA 4.0 *https://hackyourfuture.net/*

CC BY-NC-SA 4.0 Icons

Built with ❤️ by the HackYourFuture community · Thank you, contributors

Found a mistake or have a suggestion? Let us know in the feedback form.