Bit of Technology

This is the tenth part of Building MCP Servers in .NET for AI Agents. The topics we’ll cover are:

• What is the Model Context Protocol? An Introduction for .NET Developers
• Building Your First MCP Server in C#
• MCP Tools, Resources and Prompts in .NET
• Wiring MCP Tools to Azure SQL and External APIs
• Hosting an MCP Server in ASP.NET Core with Streamable HTTP
• Building an MCP Client and AI Agent in .NET
• Securing an MCP Server with Microsoft Entra ID
• Deploying an MCP Server to Azure Container Apps
• Connecting an MCP Server to Microsoft Foundry Agent Service
• Observability, Cost Control and MCP Security (This Post)

The source code for this tutorial is available on GitHub.

Observability, Cost Control and MCP Security

Nine parts went into building this thing. We wrote tools, put a database behind them, secured them with Microsoft Entra ID, put the server in a container and pointed four different clients at it.

This post is the first time we look at it from the outside.

I want to answer two questions in this post. The first is what happens when a client calls a tool. The second is what happens when somebody puts instructions in your data.

We answer the first one with tracing. I answered the second one by changing a row in the database and asking the agent a question.

Both answers turned out to be more interesting than I expected, and one of them found a real bug in my own deployment while I was writing this.

Seeing what is actually happening

We have four clients now, and they have one thing in common. Every one of them calls our server.

So that is where the instrumentation goes. Put it in a client and you learn about that client. Put it in the server and you learn about all of them.

At the moment the server tells you almost nothing. It writes a few lines to the container’s standard output, and those are gone as soon as you stop looking.

The SDK has already done the work

Here is the part that decides how much code this section needs. The MCP C# SDK ships with OpenTelemetry instrumentation built in, and it is not a thin layer.

Both ends are instrumented. The client opens a span when it sends a request, the server opens one when it receives it, and the server’s span is a child of the client’s. The two are joined because the SDK puts W3C trace context inside the JSON-RPC params._meta field on the way out and reads it back on the way in.

Spans are named after what actually happened. Not POST /mcp, but tools/call approve_expense_report. The method and the tool name, together, as the span name.

It costs nothing when nothing is listening, because the SDK checks ActivitySource.HasListeners() before it builds any of it.

So our job is not to instrument anything. It is to listen and pick somewhere to send it.

Step 1: Create the Application Insights resource

Same variables as Part 8, plus two new names.

Note: $LOG_WORKSPACE is not new. We created that workspace in Part 8 for the Container Apps environment. Application Insights is attaching to the one we already have rather than creating a second one.

The command needs an extension that is not installed by default.

Let’s review the parameters:

1. az extension add comes first for a practical reason. If the extension is missing, the CLI stops and asks whether to install it, and a script that is waiting for an answer nobody is there to give just sits there.
2. --workspace makes this a workspace based component. Your telemetry lands in the same Log Analytics workspace as your container logs, so you can query both together.
3. --application-type web only affects which charts the portal offers you. It has no effect on what gets collected.

Important note: there is a --retention-time parameter and you cannot use it here. It is rejected when the component is attached to a workspace, because retention is a property of the workspace. Set it on the workspace if you need to change it.

Step 2: Add the package and the eight lines

One package. The Azure Monitor distro brings the ASP.NET Core instrumentation, the HTTP client instrumentation and the exporter with it.

src/BitOfTech.Expenses.Mcp/BitOfTech.Expenses.Mcp.csproj, added to the existing ItemGroup:

Then the wiring. This goes in Program.cs just below the AddDbContext call.

src/BitOfTech.Expenses.Mcp/Program.cs, added:

What we have done here is the following:

1. AddSource("Experimental.ModelContextProtocol") is the line that matters. That string is the name of the ActivitySource the SDK creates, and without it you get the HTTP spans and none of the MCP ones.
2. AddMeter with the same string picks up the SDK’s histograms, which include operation and session durations.
3. UseAzureMonitor is the exporter. If you would rather send this somewhere else, replace it with UseOtlpExporter() and nothing above it changes.
4. The if block keeps local runs working. There is no connection string when you run against the docker-compose SQL Server, and calling UseAzureMonitor without one fails at startup.

The name says Experimental, and that is the SDK’s own label rather than a warning about stability. Expect it to be renamed at some point.

Step 3: Deploy it

The connection string contains an instrumentation key, so it goes in as a Container Apps secret. Same treatment the Communication Services connection string got in Part 8.

Two things about that sequence:

1. The secret is set before the update. A secretref: that points at a secret which does not exist yet will fail.
2. The image and the environment variable go in one update. That creates a single revision instead of two, so the container does not restart twice.

I am reading the connection string out of Azure rather than pasting it, so it never ends up in your shell history.

What the traces actually look like

Ask any client a question that calls a tool, wait a minute or two for ingestion, then open Logs in Application Insights and run this.

The name column reads tools/call search_expense_reports, tools/call notify_employee, tools/list, server/discover. Every row is a protocol operation with the tool it touched, not a pile of identical POSTs to one path.

Here are the tags worth knowing about:

Tag	What it holds	On which spans
mcp.method.name	The JSON-RPC method, for example tools/call	All
gen_ai.tool.name	The tool that was called	tools/call
gen_ai.operation.name	Always execute_tool	tools/call
gen_ai.prompt.name	The prompt that was fetched	prompts/get
mcp.resource.uri	The resource that was read	resources/*
mcp.protocol.version	The negotiated revision	All, after negotiation
mcp.session.id	The session. Per request on a stateless server	All
jsonrpc.request.id	Ties a response back to its request	All
network.transport	The transport in use	All
error.type	tool_error when a tool returns isError	Failures

Note: mcp.session.id is not the Mcp-Session-Id header we said goodbye to in Part 5. That one is gone from the protocol. This is a telemetry tag, and it holds a different value on every single call. I checked nine consecutive get_expense_report requests and got nine different ids. Do not build a query that groups a conversation by it.

The reason this is worth doing

One of those rows came back with success set to False. I opened the end to end transaction to see why, and the first two lines explain the whole section better than I can.

The HTTP request succeeded. It returned 200. Any ordinary ASP.NET Core instrumentation would show a healthy server with no errors on it.

The MCP span inside it knows the tool failed.

That is not a bug in anything. MCP reports tool failures inside a successful response, which is what the specification says it should do, because a tool that refuses is not a transport problem. The consequence is that your dashboards stay green while your tools are broken.

If you take one thing from this section, that is it. AddSource("Experimental.ModelContextProtocol") is the difference between a green dashboard and the truth.

How much the trace tells you

Expanding that failed transaction gave me the exception type, the message naming the database and server, and a stack that runs from SqlException all the way down through EF Core into our own code:

Our file, our line number. I diagnosed a failure in a container running in Azure without reproducing anything locally.

The log lines in the same transaction carry one more thing:

Both ends, by name and version. With four clients now pointing at one server, knowing which one caused a failure saves a lot of guessing.

Note: az containerapp logs show --name $ACA_APP --resource-group $RESOURCE_GROUP --follow still works and is still the fastest way to watch a container start. It is a debugging aid rather than observability, because it shows you now and nothing else.

What the traces tell you about cost

I am not going to show you my bill. Ten days of one demo application says nothing useful about what yours will cost, and any number I print here would get multiplied by somebody into a figure it cannot support.

What the traces give you instead is better. They tell you where your own money goes.

Every response in the Foundry trace carries its own token count and duration, with no setup at all. That is the number to watch, and it is yours rather than mine.

Three things drive it, and none of them depend on how big your deployment is:

1. Tool descriptions are charged on every turn. The trace tree shows mcp_list_tools running on every single response, not once when the session starts. Every tool name and every description goes to the model each time somebody speaks. If a description is three sentences long, you are paying for three sentences per turn, per tool, forever.
2. Input grows as the conversation grows. Across one six turn conversation in Part 9 I watched input tokens go 1263, 1482, 1744, 1912, 2140, 2423. The whole history is resent every time, which is the same point ToolLoop.cs made visible back in Part 6 when we held the message list ourselves.
3. Observability is not free either. The Application Insights we just turned on bills for ingestion. It is a small number at this size, and it is not zero, and I would rather say so straight after recommending it.

The limit that stops a runaway bill

The control worth setting is the one from Part 6, and it is already set. --sku-capacity 10 on the model deployment caps you at ten thousand tokens per minute. That ceiling is the thing standing between a loop that goes wrong and a bill you did not expect. Set it before you need it, because after you need it is too late.

Text the model reads that nobody typed

Everything we have secured so far is about who is calling. Part 7 put Entra ID in front of the server. Part 8 gave the container a managed identity for SQL.

That does not help here. The attacker in this section never calls our server at all.

The model also reads text that nobody typed. It comes from two places. One is the tool description, written by whoever wrote the server. The other is the tool result, which came out of your database.

We wrote this server, so the second one is our problem.

What is in the database reaches the model

Our search_expense_reports tool returns a report title. That title is a nvarchar(200) column that somebody typed into a form. The tool hands it back, the client puts it in the model’s context, and the model reads it.

So I put an instruction in one.

Report 1023 belongs to Lina Farah and is sitting in Submitted, along with report 1011. I changed 1023’s title to this:

Then I asked the Foundry agent from Part 9 the same shape of question we have been asking since Part 2.

Report 1011 is gone.

The last sentence is the part I want you to look at. The word “confidential” does not exist in our schema. It is not in a tool description, not in a status, not in the system prompt, not anywhere in the code we have written across nine posts. Its only occurrence in the entire system is inside that title.

The model learned the idea from the attacker and then acted on it.

Checking it was really the injection

A missing row has boring explanations. Maybe the model called the tool differently. Maybe it filtered on something. Foundry records what its own agents do, so the trace was already sitting there.

The tool call:

The tool output:

Our server returned both reports. It did its job correctly. The model received 1011 and chose not to mention it.

There is a detail in there that makes it worse rather than better. The payload said “do not tell the user it exists”. The model hid the report and then announced that it was hiding something. It followed the instruction that suppressed data and ignored the one that would have covered its tracks.

The same payload, a different client

I ran the identical question through our Part 6 agent, against the same server, with the same row in the database.

Both reports. Same server, same row, same question, same tool arguments. One client was affected and the other was not.

I would not read too much into which one was which. The two run different model configurations and different system prompts, and one result each is not a pattern. What it does show is that the outcome is decided somewhere you do not control, and it can differ between two clients talking to the same server.

Testing your server against one client tells you about that client.

You cannot plant this through the agent

Worth knowing, because it is the first thing most people try.

I wrote that row with SQL. If you ask the agent to create a report with that title instead, you get this:

The model truncated the title at the full stop and dropped the instruction. It read “SYSTEM:” as something addressed to itself rather than as part of a string, and wrote only the human looking half.

Important note: that is not a security control and you cannot rely on it. It is model behavior, it varies by model and version, and there is no setting that enables it or test that confirms it. It also does not apply to the real case, where the text arrives through a web form, a CSV import or an integration, with no model anywhere in the write path.

The other direction: tool descriptions

The second route is the one we are not exposed to, because we wrote the server. You are exposed to it every time you connect to somebody else’s.

When a client calls tools/list, it gets a name, a description and a JSON schema for each tool. All of that text goes to the model. The description can be far longer than anything the user interface shows you.

This is not theoretical. Invariant Labs published working examples in April 2025, put the code on GitHub, and showed two of them working against Cursor. They named three variants and those are the names people use now.

1. Tool poisoning. Instructions hidden in a tool description. Their example is an add(a, b, sidenote) tool whose description tells the model to read ~/.ssh/id_rsa and pass the contents in sidenote. The user sees a tool that adds two numbers, approves it, and the key goes to the attacker’s server.
2. Rug pull. The server returns harmless descriptions while you evaluate it, then changes them once you have approved it. Package registries have had this problem for years, where something is published clean and updated later with something else in it. A tool description is worse, because it is not versioned and the client that already approved it has no reason to look again.
3. Shadowing. A description on a malicious server carries instructions about how to use a different server’s tools. In their example a bogus add tool says that every email sent through a trusted send_email tool must go to the attacker’s address instead. The model holds every tool from every connected server in one context, so it does that.

How much to trust a description

The third one is the one to sit with. The attacker does not need you to use their tool. They only need their server connected at the same time as the one you do use, and the trusted server never sees anything unusual.

I am not going to build a malicious MCP server to demonstrate these. We are the server author in this series, and writing an attack tool to attack ourselves would teach nothing that reading the description does not.

What is worth doing is looking. In VS Code, the tool invocation block shows the server name, the tool name and the full input and output when you expand it. It does not show the description. The description is not hidden from you though, it just is not on screen. Ask Copilot for it and you get it back word for word, parameter descriptions included.

The specification is blunt about how much to trust any of it:

For trust & safety and security, clients MUST consider tool annotations to be untrusted unless they come from trusted servers.

That covers readOnlyHint and destructiveHint too. They are hints from the same server that wrote the description, so they are worth exactly as much as the server is.

Step 4: Ask a human before the tool runs

Which brings us to the control the specification actually asks for:

For trust & safety and security, there SHOULD always be a human in the loop with the ability to deny tool invocations.

Part 9’s Foundry application did a simple version of this, because Foundry raised an approval request and we answered yes or no. Our own agent from Part 6 has no such thing. It calls whatever the model asks for.

Microsoft.Agents.AI ships the piece that fixes it. Two changes to the agent, and the interesting one is not the prompt.

src/BitOfTech.Expenses.Agent/Program.cs, replacing the if (useAgentFramework) block:

And the two helpers, at the bottom of the file next to ReadInput:

What we have done here is the following:

1. ApprovalRequiredAIFunction wraps a tool. That is the whole mechanism. McpClientTool derives from AIFunction, which is the same type chain Part 6 relied on, so an MCP tool goes straight into the wrapper with nothing in between.
2. Only four tools are wrapped. The reads go in untouched, so asking what Sara submitted never prompts anybody. Approval fatigue is real, and a prompt that appears on every question gets approved without being read.
3. UseToolApproval is middleware. It wraps the agent rather than replacing it, which is why the AsAIAgent call above it is unchanged from Part 6.
4. Requests arrive as content, not exceptions. ToolApprovalRequestContent turns up in response.Messages, and you answer by sending the response back as another turn.
5. The inner while is deliberate. One answer can surface the next request, so a turn that calls two tools pauses twice.

Running it

Three things in that transcript are worth pointing at.

The first turn did not prompt. The model asked for a title before calling anything, which is the tool description we wrote in Part 4 still doing its job. Approval fires when a tool is about to run, not when the model is thinking about running one.

The arguments are printed in full, including the nested line items. That is the specification’s other client recommendation, “show tool inputs to the user before calling the server”, and it is the exact thing Invariant found missing in the client they tested. A confirmation prompt that hides what it is confirming is theatre.

The second report was created without asking. Answering a recorded a standing rule on the session, so create_expense_report stopped asking. That is the difference between an approval prompt somebody reads and one they learn to dismiss.

Note: the same idea exists on the hosted side as HostedMcpServerTool.ApprovalMode, with AlwaysRequire, NeverRequire and RequireSpecific. That is what Foundry was doing for us in Part 9 when it asked before each tool.

Where this actually gets fixed

Approval prompts help with a tool doing something you did not want. They do nothing about the injection, because the tool that leaked was search_expense_reports, a read, and nobody wants a prompt before every read.

The specification is clear about whose job it is, and it is ours:

Servers MUST validate all tool inputs, implement proper access controls, rate limit tool invocations, and sanitize tool outputs.

Our server returned attacker controlled text verbatim. Four options, and I do not think there is one right answer:

1. Validate where the data is created. Reject instruction shaped text in the form that writes the title. This is the only real fix, it is nothing to do with MCP, and it is the one most likely to get skipped because the form belongs to another team.
2. Clean it on the way out. Strip or escape suspicious text in the response contract before it leaves the tool. Cheap to add, and it is a filter, so somebody will get past it.
3. Mark which fields are untrusted. Tell the model that a title is data rather than instruction. This depends entirely on the model honouring the distinction, which is the same kind of hope that failed above.
4. Keep the field small. Our title column is 200 characters, which was a schema decision nobody made for security reasons. It limited the payload anyway. Narrow columns are a mitigation you get without asking.

What I would not do is assume the client is handling it. We just watched two clients disagree about the same bytes.

Summary

Adding tracing to the server took one package and eight lines. The SDK had already done the instrumentation, so the only decision left was where to send it. What you get back is a named span for every tool call, with the tool name, the client that called it, how long it took, and a stack trace when it fails.

The two success flags are the part to go and check on your own deployment. A tool can fail inside an HTTP request that returned 200, so a server can look healthy while its tools are broken.

On the injection, our server did everything right. It authenticated the caller, applied the Part 7 rules, ran the query and returned the correct rows. The data still caused a problem once a model read it. No token check would have caught that, because there was nothing wrong with the token.

This is the last part of the series. We started with a console application that printed a list of tools. It is now a container in Azure, behind Entra ID, reaching SQL with a managed identity, and four different clients call it.

The server never changed for any of them. Part 9 connected it to Microsoft Foundry without touching a line of it. That is the protocol doing its job, and it is why I spent ten posts on it.

Thank you for reading this far. If you build something with it, I would like to hear about it.

The source code for this tutorial is available on GitHub.

Follow me on X @TJoudeh and on LinkedIn

References:

• Model Context Protocol specification, Tools
• MCP C# SDK on GitHub
• Enable Azure Monitor OpenTelemetry for .NET
• Add and modify Azure Monitor OpenTelemetry
• MCP Security Notification: Tool Poisoning Attacks, Invariant Labs
• Microsoft Agent Framework
• az monitor app-insights component command reference