Azure Container Apps Monitoring and Observability with Application Insights – Part 8

This is the eighth part of Building Microservice Applications with Azure Container Apps and Dapr. The topics we’ll cover are:

• Tutorial for building Microservice Applications with Azure Container Apps and Dapr – Part 1
• Deploy backend API Microservice to Azure Container Apps – Part 2
• Communication between Microservices in Azure Container Apps – Part 3
• Dapr Integration with Azure Container Apps – Part 4
• Azure Container Apps State Store With Dapr State Management API – Part 5
• Azure Container Apps Async Communication with Dapr Pub/Sub API – Part 6
• Azure Container Apps with Dapr Bindings Building Block – Part 7
• Azure Container Apps Monitoring and Observability with Application Insights – (This Post)
• Continuous Deployment for Azure Container Apps using GitHub Actions – Part 9
• Use Bicep to Deploy Dapr Microservices Apps to Azure Container Apps – Part 10
• Azure Container Apps Auto Scaling with KEDA – Part 11
• Azure Container Apps Volume Mounts using Azure Files – Part 12
• Integrate Health probes in Azure Container Apps – Part 13

Azure Container Apps Monitoring and Observability with Application Insights

When building a microservices application that consists of many distributed services that communicate with each other across different processes and use different infrastructure services (different message brokers, different databases, different 3rd party components, etc…) then having a mechanism to observe and trace from end to end is inevitable.

The good news that Azure Container Apps provides various built-in Monitoring and Observability features that help in understanding how are the various services are operating and performing. Those features will help to monitor and diagnose the state of your distributed services to improve performance and respond to critical problems.

The Azure Container Apps official documentation page talks thoroughly about observability features, so I will not cover them here.
From my personal experience while building this tutorial and during the development and test phases, the 2 features I relied on most were the Log streaming and Azure Monitor Log Analytics.

The official documentation page didn’t talk about how to configure Azure Container Apps and Azure Container Apps Environment with Application Insights, as well Azure Container Apps do not support Auto-Instrumentation for Application Insights, so in this post, I will be focusing on how we can integrate Application Insights into our microservices application.

The source code for this tutorial is available on GitHub. You can check the demo application too.

Application Insights overview

Application Insights is an offering from Azure Monitor that will help us to monitor all Azure Container Apps under the same Container App Environment and collect telemetry about the services within the solution, as well as understand the usage of the services and users’ engagement via integrated analytics tools.
What I mean by Telemetry here is the data collected to observe our application, it can be broken into three categories :

1. Distributed Tracing: provides insights into the traffic between services involved in distributed transactions, think of when the Frontend Web App talks with the Backend Api App to insert or retrieve data. An application map of how calls are flowing between services is very important for any distributed application.
2. Metrics: provide insights into the performance of a service and its resource utilization, think of the CPU and Memory utilization of the Backend Background processor, and how we can understand when we need to increase the number of replicas.
3. Logging: provides insights into how code is executing and if errors have occurred.

So let’s get started and add Application Insights to our solution.

Provision a Workspace-based Application Insights Instance

We need to create one single Application Insights instance to be used for all the services telemetry within our application, to create the instance using Azure CLI script below:

We are creating here a Workspace-based Application Insights resource as the “Classic” one will be out of support in 2024. So we are going to use the same Workspace we created when we provisioned the Azure Container App Environment in this previous post.

Next, we need to update the Azure Container App Environment we have created in this previous post, if you remember we used the CLI command az containerapp env create  (link for command properties) as the below script to create the Environment:

You can see that we didn’t provide the property --dapr-instrumentation-key when creating it, and unfortunately we can’t update this from the Azure CLI, there is an open issue on GitHub asking to add support for updating the Azure Container Apps Environment via the CLI.
So my recommendation is to set the property “dapr-instrumentation-key” when creating the ACA Environment as it is the property responsible for setting the instrumentation key of the Application Insights instance used by Dapr to export Service to Service communication telemetry, without it you will not be able to see the relation between microservices.

In our case we need to do the update, in order to do this we are going to use Bicep/ARM template, I’m going to cover in the future post in full detail how we are going to use Bicep to provision every component in the solution including the external components such as Azure Storage, Cosmos DB, Service bus, etc… it will be a detailed post to show you how Bicep will simplify the creation of Infrastructure as a Code (IaC) and will enable you to create and redeploy the entire solution with all configuration we have done manually by executing one command or running 1 single action from GitHub actions.

For this post, I will keep the description minimal so follow along with me to create a simple Bicep file used to update the Azure Container Apps Environment, to do so, from VS code add a new folder named “deploy” on the same root folder of our solution “TasksTracker.ContainerApps/deploy” and then add a new file named “updateacaEnvironment.bicep”.
You need to install a VS code extension named Bicep, so go ahead and install it as we are going to use it heavily in the next post.
Now open the file named “updateacaEnvironment.bicep” and paste the code below

What we’ve done here is simple, we are getting a reference for the existing Application Insights and Log Analytics Workspace instances, then, we are updating the existing Azure Container Apps Environment, see how we are setting the property “daprAIInstrumentationKey” on the highlighted line above, and providing the Application Insights Instrumentation Key.

To execute this file, use the PowerShell terminal and run the command below:

Installing Application Insights SDK into the 3 Microservices apps

Now we need to add Application Insights SDK to the 3 services we have, this is an identical operation, so I will describe how we can do it on the Backend API and you do the same on the remaining services.

Step 1: Install the Application Insights SDK using NuGet

To add the SDK, open the file “TasksTracker.TasksManager.Backend.Api.csproj” and add the below NuGet reference

Step 2: Set RoleName property for all the services

Next and on each project, we will add a new file named “AppInsightsTelemetryInitializer.cs” on the root directory of the project, so add a file named “AppInsightsTelemetryInitializer.cs” under project “TasksTracker.TasksManager.Backend.Api.Svc” and paste the below:

The only difference between each file on the 3 projects is the RoleName property value, this property will be used by Application Insights to identify the components on the application map, as well for example it will be useful for us if we need to filter on all the warning logs generated from the Backend API project, so we will use the value “tasksmanager-backend-api” when we filter.
You can check the RoleName value used in the project “TasksTracker.WebPortal.Frontend.Ui” and project “TasksTracker.Processor.Backend.Svc“.

Next, we need to register this “AppInsightsTelemetryInitializer” class, to do this, open the file “Program.cs” and add the code highlighted below (lines 5-9), don’t forget that you need to do the same for the remaining 2 projects.

Step 3: Set the Application Insights instrumentation key in appsettings.json file

Now we need to set the Application Insights Instrumentation Key so the projects are able to send telemetry data to the AI instance, to do this open file “appsettings.json” and paste the code below, we are going to set this via secrets and environment variables once we redeploy the Container Apps and create new revisions.

With this step we have completed the changes on the projects, let’s now deploy the changes and create new revisions.

Deploy the 3 services to Azure Container Apps and create new Revisions

Step 1: Add Application Insights Insrumnation key as a secret

Let’s create a secret named “appinsights-key” on each Container App which contains the value of the AI instrumentation key, remember that we can obtain this value from Azure Portal by going to AI instance we created or we can get it from Azure CLI as we did above. To create the secret use PowerShell console and paste the code below:

Step 2: Build new images and push them to ACR

As we have done previously we need to build and push the 3 apps images to ACR so they are ready to be deployed to Azure Container Apps, to do so, continue using the same PowerShell console and paste the code below (Make sure you are on directory “TasksTracker.ContainerApps”):

Step 3: Deploy new revisions of the 3 services to Azure Container Apps and set a new environment variable

As we’ve done multiple times, we need to update the Azure Container App hosting the 3 services with a new revision so our code changes are available for end users, to do so run the below PowerShell script, n and notice how we used the property --set-env-vars to set new environment variable named “ApplicationInsights__InstrumentationKey” and its value is a secret reference coming from the secret “appinsights-key” we added in step 1.

With those changes in place, you should start seeing telemetry coming to the Application Insights instance provisioned, let’s review Application Insights key dashboards and panels in Azure Portal.

Distributed Tracing via Application Map

Application Map will help us to spot any performance bottlenecks or failure hotspots across all our services of our distributed microservices application. Each node on the map represents an application component (service) or its dependencies and has a health KPI and alerts status.

Looking at the image above, you will see for example how the Backend Api with could RoleName “tasksmanager-backend-api” is depending on the Cosmos DB instance, showing the number of calls and average time to service these calls. The application map is interactive so you can select a service/ component and drill down into details. For example, when I drill down into the Dapr State node to understand how many times my backend API invoked the Dapr Sidecar state service to Save/Delete state, you will see results similar to the below image:

Mintor production application using Live Metrics

This is one of my favorite monitoring panels, it provides you with near real-time (1-second latency) status of your entire distributed application, we can see performance and failures count, we can watch exceptions and traces as they happened, and we can see live servers (replicas in our case) and the CPU and Memory utilization and the number of requests they are handling.
These live metrics provide very powerful diagnostics for our production microservice application. Check the image below and see the server names and some requests coming to the system.

Logs search using Transaction Search

Transaction search in Application Insights will help us to find and explore individual telemetry items, such as exceptions, web requests, or dependencies. As well as any log traces and events that we’ve added to the application.
For example, if I want to see all the event types of type “Request” for the cloud RoleName “tasksmanager-backend-api” in the past 24 hours, I can use the transaction search dashboard to do this, see how the filters are set and the results are displayed nicely, you can drill down on each result to have more details and what telemetry was captured before and after. A very useful feature when troubleshooting exceptions and reading logs.

Failures and Performance Panels

The failure panel allows us to view the frequency of failures across different operations to help us to focus our efforts on those with the highest impact.

The Performance panel displays performance details for the different operations in our system. By identifying those operations with the longest duration, we can diagnose potential problems or best target our ongoing development to improve the overall performance of the system

In the next post, we will be implementing continuous integration and deployment so we automate the process of building images, pushing them to ACR, and then creating new revisions.

The source code for this tutorial is available on GitHub. You can check the demo application too.

Follow me on Twitter @tjoudeh

References:

• The Dapr observability building block
• Opentelemtry Overview
• Application Map: Triage distributed applications
• Find and diagnose performance issues with Azure Application Insights
• Featured image credit by Bethany Drouin