Assignment: Deploy to Azure

The Scenario

Your team has been running the weather-data pipeline on your laptop for testing, which means nobody else can see its output and it stops when you close the lid. The product owner needs it to run on Azure every morning and write to the team's shared storage so the analytics dashboard refreshes overnight. This week you take the containerized pipeline from Week 5 and deploy it as an Azure Container App Job that writes raw output to Blob Storage and structured rows to managed Postgres.

By the end you will have a cloud-native data pipeline producing stored outputs you can verify.

<aside> 📝 If a task feels unfamiliar, the Practice chapter has paired drills (Ex2 for blob upload + CLI verify, Ex3 for Postgres schema + ingest, Ex4 for the container job) and three risk-free Python widgets (parse_postgres_url, validate_create_command, monthly_cost_eur) to pre-flight the trickiest steps before you spend Azure credits.

</aside>

Task 1: Upload Raw Data to Blob Storage

Write a Python script that uploads your pipeline's raw output (JSON or CSV) to the shared Azure Blob Storage account. Verify the blob appeared using the CLI or portal.

<aside> 💡 Get the connection string from your teacher or retrieve it from Key Vault (Azure Container Apps Jobs). Follow the naming convention: raw/<source>/<date>.json.

</aside>

Task 2: Connect to Azure Postgres

Connect to the shared Postgres server your teacher provided. To prevent overwriting other students' tables, create an isolated schema named dev_<your_name> (e.g. dev_lasse) and set the search_path to it. Create a weather_readings table inside your schema, and verify you can insert and query rows from Python.

<aside> 💡 Get the connection string from your teacher or retrieve it from Key Vault (Azure Container Apps Jobs). Do not use the admin account for your pipeline.

</aside>

Task 3: Update Your Pipeline for Cloud Storage

Update your pipeline to write to both blob storage and Postgres. Your pipeline should:

Read POSTGRES_URL and AZURE_STORAGE_CONNECTION_STRING from environment variables
Upload raw output to blob storage
Create your personal database schema and the output table if they do not exist (using the search_path schema redirection pattern from Azure PostgreSQL Databases)
Insert rows into Postgres using the closing() pattern from Azure PostgreSQL Databases and log the count
Silence Azure SDK logging noise with logging.getLogger("azure").setLevel(logging.WARNING) (see Azure Container Apps Jobs)

<aside> ⚠️ Do not commit connection strings or passwords. Use environment variables.

</aside>

Task 4: Deploy as a Container App Job

Rebuild your Docker image with the cloud storage changes from Task 3 and push it to ACR (either with docker push or through CI). Verify your image is in the registry:

az acr repository show-tags \
  --name hyfregistry \
  --repository <your-handle>-weather-pipeline \
  --output table

Create a Container App Job in the shared environment (env-hyf-data) and resource group (rg-hyf-data). Deploy your Week 5 image (<your-handle>-weather-pipeline:<tag>). If your image fails to pull or crash-loops while you debug, you may temporarily use the teacher-maintained fallback hyfregistry.azurecr.io/weather-pipeline:assignment, but the assignment deliverable must eventually run your image.

Run the az containerapp job create command to deploy your job, retrieving the connection string secrets directly from Azure Key Vault:

az containerapp job create \
  --name <your-handle>-weather-job \
  --resource-group rg-hyf-data \
  --environment env-hyf-data \
  --image hyfregistry.azurecr.io/<your-handle>-weather-pipeline:<tag> \
  --registry-server hyfregistry.azurecr.io \
  --trigger-type Manual \
  --replica-timeout 300 \
  --replica-retry-limit 0 \
  --env-vars \
    POSTGRES_URL="$(az keyvault secret show --vault-name kv-hyf-data --name postgres-url --query value -o tsv)" \
    AZURE_STORAGE_CONNECTION_STRING="$(az keyvault secret show --vault-name kv-hyf-data --name storage-connection-string --query value -o tsv)" \
    LOG_LEVEL=INFO

Confirm the job starts and logs are visible.

Task 5: Run and Verify

Start the job and confirm it writes to both storage targets:

Check the job execution history for a successful run.
Verify a blob appeared in storage.
Query Postgres and verify the row count matches your log output.
Capture the most recent successful run as evidence in your README. Prefer CLI output (no PII): az containerapp job execution list --name <your-job-name> --resource-group rg-hyf-data --output table > docs/execution_history.txt. If you prefer a portal screenshot, crop it tightly to the execution-history rows and redact your account email, tenant ID, and subscription ID before committing. Reference the file under a ## Verification heading in your README.

<aside> ⌨️ Hands on: Add a log line that prints the row count written and the blob name uploaded.

</aside>

Task 6: Clean Up

Delete your Container App Job when you are done:

az containerapp job delete --name <your-job-name> --resource-group <rg> --yes

Do not leave jobs running after the assignment. See Azure Container Apps Jobs and Cost Awareness for why this matters.

Task 7: AI Assist Report

Create AI_ASSIST.md and describe:

The prompt you gave an LLM
The output it provided
What you changed or verified

Deliverables

You must submit:

A deployed Container App Job in Azure
Raw output uploaded to Azure Blob Storage
A Postgres table populated by your pipeline
AI_ASSIST.md
A short README describing how to trigger a run and verify results, including the execution-history evidence from Task 5 (CLI output saved to docs/execution_history.txt preferred, a cropped + redacted portal screenshot accepted as a fallback)

Technical Requirements

Use a clear project structure, for example:

week6-azure-assignment/
├── src/
│   └── pipeline.py
├── Dockerfile
├── requirements.txt
├── README.md
└── AI_ASSIST.md

Use logging instead of print for pipeline status.
Fail fast if required env vars are missing.
Use pinned dependency versions.

Evaluation Criteria

Your assignment will be evaluated on:

Deployment: container job runs without manual fixes
Data storage: outputs exist in both blob storage and Postgres with correct data
Security: secrets are not committed and least privilege is used
Observability: logs show row counts, blob names, and status messages
Clarity: README explains how to run and verify outputs

Extra reading

Azure Container Apps Jobs: official guide for container jobs
Azure Blob Storage Python quickstart: uploading blobs from Python
Azure Database for PostgreSQL: managed Postgres documentation

Submission

Create a git branch week6/your-name.
Commit your work with a clear message.
Push the branch and open a Pull Request.

The HackYourFuture curriculum is licensed under CC BY-NC-SA 4.0 *https://hackyourfuture.net/*

CC BY-NC-SA 4.0 Icons

Built with ❤️ by the HackYourFuture community · Thank you, contributors

Found a mistake or have a suggestion? Let us know in the feedback form.