Week 6 - Cloud and Azure Essentials

Introduction to Cloud and Azure

Azure CLI and the Portal

Azure Blob Storage

Azure PostgreSQL Databases

Azure Container Apps Jobs

Cost Awareness

History of Cloud Computing

Week 6 Gotchas & Pitfalls

Practice

Week 6 Assignment: Deploy Your Pipeline to Azure

Week 6 Lesson Plan

🎒 Week 6 Assignment: Deploy Your Pipeline to Azure

This week you will run your containerized pipeline in Azure and store its outputs in blob storage and Postgres. The result should be a cloud-native data pipeline that produces stored outputs you can verify.

Task 1: Upload Raw Data to Blob Storage

Write a Python script that uploads your pipeline's raw output (JSON or CSV) to the shared Azure Blob Storage account. Verify the blob appeared using the CLI or portal.

<aside> 💡 Get the connection string from your teacher or retrieve it from Key Vault (Chapter 5). Follow the naming convention: raw/<source>/<date>.json.

</aside>

Task 2: Connect to Azure Postgres

Connect to the shared Postgres server your teacher provided. Create a table for your pipeline output and verify you can insert and query rows from Python.

<aside> 💡 Get the connection string from your teacher or retrieve it from Key Vault (Chapter 5). Do not use the admin account for your pipeline.

</aside>

Task 3: Update Your Pipeline for Cloud Storage

Update your pipeline to write to both blob storage and Postgres. Your pipeline should:

• Read POSTGRES_URL and AZURE_STORAGE_CONNECTION_STRING from environment variables
• Upload raw output to blob storage
• Create the output table if it does not exist
• Insert rows into Postgres using the closing() pattern from Chapter 4 and log the count
• Silence Azure SDK logging noise with logging.getLogger("azure").setLevel(logging.WARNING) (see Chapter 5)

<aside> ⚠️ Do not commit connection strings or passwords. Use environment variables.

</aside>

Task 4: Deploy as a Container App Job

Rebuild your Docker image with the cloud storage changes from Task 3 and push it to ACR (either with docker push or through CI). Verify the image is in the registry with az acr repository show-tags.

Create a Container App Job in the shared environment and resource group your teacher provided. Remember to include:

• --registry-server pointing to your ACR
• --replica-timeout 300 so stuck jobs are killed automatically
• --env-vars for POSTGRES_URL, AZURE_STORAGE_CONNECTION_STRING, and LOG_LEVEL

Confirm the job starts and logs are visible.

Task 5: Run and Verify

Start the job and confirm it writes to both storage targets:

1. Check the job execution history for a successful run.
2. Verify a blob appeared in storage.
3. Query Postgres and verify the row count matches your log output.

<aside> ⌨️ Hands on: Add a log line that prints the row count written and the blob name uploaded.

</aside>

Task 6: Clean Up

Delete your Container App Job when you are done:

az containerapp job delete --name <your-job-name> --resource-group <rg> --yes

Do not leave jobs running after the assignment. See Chapter 5 and Chapter 6 for why this matters.

Task 7: AI Assist Report

Create AI_ASSIST.md and describe:

• The prompt you gave an LLM
• The output it provided
• What you changed or verified

Deliverables

You must submit:

• A deployed Container App Job in Azure
• Raw output uploaded to Azure Blob Storage
• A Postgres table populated by your pipeline
• AI_ASSIST.md
• A short README describing how to trigger a run and verify results

Technical Requirements

• Use a clear project structure, for example:

week6-azure-assignment/
├── src/
│   └── pipeline.py
├── Dockerfile
├── requirements.txt
├── README.md
└── AI_ASSIST.md

• Use logging instead of print for pipeline status.
• Fail fast if required env vars are missing.
• Use pinned dependency versions.

Evaluation Criteria

Your assignment will be evaluated on:

• Deployment: container job runs without manual fixes
• Data storage: outputs exist in both blob storage and Postgres with correct data
• Security: secrets are not committed and least privilege is used
• Observability: logs show row counts, blob names, and status messages
• Clarity: README explains how to run and verify outputs

Extra reading

• Azure Container Apps Jobs: official guide for container jobs
• Azure Blob Storage Python quickstart: uploading blobs from Python
• Azure Database for PostgreSQL: managed Postgres documentation

Submission

1. Create a git branch week6/your-name.
2. Commit your work with a clear message.
3. Push the branch and open a Pull Request.

The HackYourFuture curriculum is licensed under CC BY-NC-SA 4.0 *https://hackyourfuture.net/*

Built with ❤️ by the HackYourFuture community · Thank you, contributors

Found a mistake or have a suggestion? Let us know in the feedback form.