Exercise - Diagnose and fix a failing deployment

In this exercise, you troubleshoot a failing container app and apply targeted fixes. You use revision status, logs, and the Azure CLI to isolate deployment issues. This workflow is common in AI solutions because startup behavior changes frequently when you update models and dependencies.

Tasks performed in this exercise:

• Deploy a mock AI document processing API as a container app
• Introduce and diagnose a missing environment variable error
• Introduce and diagnose an ingress configuration issue
• Query Log Analytics for historical troubleshooting data

Important: Azure Container Registry task runs are temporarily paused from Azure free credits. This exercise requires a Pay-As-You-Go, or another paid plan.

Before you start

To complete the exercise, you need:

• An Azure subscription with the permissions to deploy the necessary Azure services.
• Visual Studio Code on one of the supported platforms.
• The latest version of the Azure CLI.
• Python 3.12 or greater.

Download project starter files and deploy Azure services

You can find the starter files for this exercise in the labs/lab-06-manage-azure-container-apps/ folder of this project.

1. Open a terminal and navigate to the lab folder:

   cd labs/lab-06-manage-azure-container-apps

2. Open the azdeploy.py deployment script and change the two values at the top of the script to meet your needs, then save your changes.

   "<your-resource-group-name>" # Resource Group name 
   "<your-azure-region>" # Azure region for the resources

3. Login to your Azure account:

   az login

4. Ensure you have the necessary extensions and providers:

   az extension add --name containerapp 
   az extension add --name log-analytics
   az provider register --namespace Microsoft.App 
   az provider register --namespace Microsoft.OperationalInsights 
   az provider register --namespace Microsoft.ContainerRegistry

Create resources in Azure

1. Run the deployment script to deploy the necessary services.

   python azdeploy.py

2. When the script is running:

   • Enter 1 to launch the Create Azure Container Registry and build container image option.
   • Enter 2 to launch the Create Container Apps environment option.
   • Enter 3 to launch the Deploy the container app and configure secrets option.
   • Enter 5 to exit the deployment script.

3. Load the environment variables generated by the script into your terminal session:

   source .env

   (Keep the terminal open. If you close it, you must run this command again).

4. Retrieve the app FQDN and store the result to a variable:

   FQDN=$(az containerapp show -n $CONTAINER_APP_NAME -g $RESOURCE_GROUP \
     --query properties.configuration.ingress.fqdn -o tsv) 
   echo "$FQDN"

5. Call the default endpoint to verify the app is running:

   curl -s "https://$FQDN/"

   Look for the model.name field, it should be set to gpt-5.4-mini.

Diagnose a missing environment variable

1. Remove the MODEL_NAME environment variable:

   az containerapp update -n $CONTAINER_APP_NAME -g $RESOURCE_GROUP \
     --remove-env-vars MODEL_NAME

2. Confirm a new revision was created and is receiving 100% traffic:

   az containerapp revision list -n $CONTAINER_APP_NAME -g $RESOURCE_GROUP -o table

3. Check the root endpoint. The model.name field now shows the default value of not-configured:

   curl -s "https://$FQDN/" | jq .model

4. Diagnose the root cause by confirming the environment variable is missing in the configuration:

   az containerapp show -n $CONTAINER_APP_NAME -g $RESOURCE_GROUP \
     --query "properties.template.containers[0].env" -o table

5. Fix the issue by adding the variable back:

   az containerapp update -n $CONTAINER_APP_NAME -g $RESOURCE_GROUP \
     --set-env-vars MODEL_NAME=$MODEL_NAME

6. Verify the fix:

   curl -s "https://$FQDN/" | jq .model

Diagnose an ingress configuration issue

1. Update the container app to use the wrong target port (3000):

   az containerapp ingress update -n $CONTAINER_APP_NAME -g $RESOURCE_GROUP \
     --target-port 3000

2. Access the health endpoint (it will fail/timeout):

   curl -s "https://$FQDN/health"

3. Diagnose the root cause by checking the ingress configuration:

   az containerapp show -n $CONTAINER_APP_NAME -g $RESOURCE_GROUP \
     --query "properties.configuration.ingress" -o yaml

4. Check the container logs. You should see gunicorn startup messages indicating the app is listening on port 8000, confirming the mismatch:

   az containerapp logs show -n $CONTAINER_APP_NAME -g $RESOURCE_GROUP

5. Fix the ingress configuration by setting the correct target port:

   az containerapp ingress update -n $CONTAINER_APP_NAME -g $RESOURCE_GROUP \
     --target-port 8000

6. Verify the fix. You should see {"status":"healthy"}:

   curl -s "https://$FQDN/health"

Query Log Analytics for historical troubleshooting

Console logs are recent only. For historical troubleshooting, logs persist in Log Analytics.

1. Get the Log Analytics workspace ID:

   WORKSPACE_ID=$(az containerapp env show -n $ACA_ENVIRONMENT -g $RESOURCE_GROUP \
     --query properties.appLogsConfiguration.logAnalyticsConfiguration.customerId -o tsv) 
   echo "Workspace ID: $WORKSPACE_ID"

2. Query the console logs for the last 20 entries:

   az monitor log-analytics query -w $WORKSPACE_ID \
     --analytics-query "ContainerAppConsoleLogs_CL | where ContainerAppName_s == '$CONTAINER_APP_NAME' | project TimeGenerated, Log_s | order by TimeGenerated desc | take 20" \
     -o table

3. Query for error-level logs specifically:

   az monitor log-analytics query -w $WORKSPACE_ID \
     --analytics-query "ContainerAppConsoleLogs_CL | where ContainerAppName_s == '$CONTAINER_APP_NAME' and Log_s contains 'error' | order by TimeGenerated desc | take 20" \
     -o table

Clean up resources

Run the following command to delete the resource group (replace <rg-name> with your actual resource group name):

az group delete --name <rg-name> --no-wait --yes