Week 6 - Cloud and Azure Essentials
Introduction to Cloud and Azure
Week 6 Assignment: Deploy Your Pipeline to Azure
Cloud deployments introduce new failure modes. Use this list to debug faster.
Each gotcha starts with what you might assume (the misconception), followed by what actually happens (the reality).
"Region does not matter for Azure services. Azure is the cloud, so everything is connected anyway."
Cross-region traffic is slower and costs more. A container job in West Europe talking to a database in North Europe adds latency to every query and may incur egress charges.
The Fix: Keep storage account, database, and container job in the same region.
"I can pick any Azure region. Data is data, and the cloud is global, so it does not matter where my database runs."
In the EU, the General Data Protection Regulation (GDPR), known in the Netherlands as the AVG (Algemene Verordening Gegevensbescherming), regulates where personal data can be stored and processed. If your pipeline handles personal data (names, emails, IP addresses, location data), storing it in a US region may violate GDPR. Even within Europe, some organizations require data to stay in a specific country.
Beyond legal compliance, your employer or client may have contractual data residency requirements that restrict which regions you can use.
The Fix: For this course, use westeurope (Netherlands) for all resources. In a real project, check your organization's data residency policy before creating any resource. When in doubt, keep data in the EU and ask your team lead.
"If the database exists, the app can connect. I created the server, so my code should be able to reach it."
Azure blocks all traffic by default unless you explicitly allow your IP or Azure services. This is a security feature, not a bug.
The Fix: Add firewall rules and enable access for Azure services.
"Any Postgres connection string will work. I copied one from a tutorial and it should be fine."
Azure requires SSL and a specific host format (<server>.postgres.database.azure.com). A connection string that works for a local Postgres install will fail on Azure because sslmode is not set.
The Fix: Use sslmode=require and the full Azure host name.
"My container can read .env files in the cloud. It worked locally with docker run --env-file, so it should work the same way in Azure."
The container job only sees variables you define in Azure with --env-vars. There is no .env file on the server. Your container starts with an empty environment and immediately crashes on os.environ["POSTGRES_URL"].
The Fix: Set env vars when creating or updating the Container App Job.
"Admin credentials are fine for my app. I used them during setup and they work, so why create another user?"
Admin credentials have full access: drop tables, delete databases, read other teams' data. If they leak through logs, git history, or a compromised container, the blast radius is everything on the server.
The Fix: Create a least-privilege user for the pipeline.
"A successful deploy means the pipeline ran. The az containerapp job create command returned without errors, so everything is working."
Job creation and job execution are separate steps. The job can be created successfully while the container fails on startup due to missing env vars, a bad image, or a runtime error in your code.
The Fix: Check execution history and logs. Verify rows exist in Postgres and blobs exist in storage.
"The default timeout is enough for any pipeline. My pipeline runs in 30 seconds locally, so the default should be fine."
Cloud environments are slower than your laptop: cold starts, network latency, and image pulls all add time. If your pipeline takes longer than --replica-timeout, Azure kills it mid-run without finishing the database writes or blob upload.
The Fix: Set --replica-timeout to a value that covers your longest expected run, with some margin. 300 seconds is a good default.
"If the script finishes, the blob was uploaded. Python did not throw an exception, so the upload must have worked."
Network errors or wrong container names can cause the upload to fail without a clear error if you do not check the return value. The SDK may retry silently and eventually give up.
The Fix: Verify the blob exists after upload using container_client.list_blobs() or the CLI. Log the blob name on success.
--registry-server when creating a job"Specifying --image hyfregistry.azurecr.io/weather-pipeline:1.0 is enough for Container Apps to find the image. The full registry URL is right there in the image name."
Container Apps does not parse the registry from the image name. Without --registry-server, it defaults to Docker Hub and fails to pull your private ACR image. The job creation succeeds, but every execution fails with an image pull error.
The Fix: Always include --registry-server hyfregistry.azurecr.io when creating a job from ACR.
--container name when viewing logs"The container name is the same as my Docker image name. I named my image weather-pipeline, so the container should have the same name."
By default, Container Apps sets the container name to the job name, not the image name. If your job is called weather-job but you pass --container weather-pipeline, you get an empty or error response.
The Fix: Use your job name as the --container value: az containerapp job logs show --container weather-job.
"Setting LOG_LEVEL=INFO gives me clean pipeline logs. I configured logging properly, so I should only see my own messages."
Chapter 5 mentions this in passing during the deployment walkthrough. It appears here as a standalone gotcha because this is the issue students hit most often after deploying: the container runs, but the logs are unreadable because Azure SDK libraries (azure-storage-blob, azure-identity) emit HTTP headers, auth details, and retry metadata at the INFO level. Your pipeline output gets buried.
The Fix: Add this line early in your pipeline, before any Azure SDK calls:
logging.getLogger("azure").setLevel(logging.WARNING)
This silences the SDK while keeping your own logging.info() messages visible.
"I can upload to any container name and Azure will create it automatically. Object storage is supposed to be simple: just put objects in and get them out."
Blob containers must exist before you upload. Unlike some S3 configurations, Azure Blob Storage does not auto-create containers. If your code targets raw but nobody created that container, the upload fails with a 404 error that does not clearly say "container not found".
The Fix: Verify the container exists with az storage container list --account-name <name>. Your teacher has pre-created raw and processed containers.