Skip to content

Chapter 17: Profiling in Production - Using dotnet-trace and dotnet-counters

Theoretical Foundations

In the realm of high-performance C# for AI, we have meticulously engineered our token processing pipelines using Span<T> to minimize memory allocations and leveraged SIMD intrinsics to accelerate vector operations. However, the transition from a development environment to a production setting introduces a layer of complexity that cannot be fully replicated in a lab. The theoretical foundation of profiling in production is built upon the understanding that real-world data distributions, user concurrency patterns, and hardware heterogeneity create dynamic performance characteristics that static analysis or synthetic benchmarks fail to capture.

To understand why dotnet-counters and dotnet-trace are indispensable, we must first appreciate the lifecycle of a request in an AI application. Imagine a high-frequency trading algorithm. In a controlled test, it might execute trades in microseconds. But in production, it must contend with network latency spikes, garbage collection pauses, and just-in-time (JIT) compilation overhead. Similarly, an AI model processing tokens—whether for a chatbot or a code generator—faces variable input lengths and unpredictable memory pressure.

The Critical Role of the Garbage Collector (GC) in AI Pipelines

In previous chapters, we explored how Span<T> and ArrayPool<T> drastically reduce pressure on the Large Object Heap (LOH). However, even with these optimizations, the GC remains a central actor in the application's performance profile. The .NET GC is generational, and understanding its behavior in production is vital.

When an AI application processes a stream of tokens, it often creates short-lived objects (e.g., intermediate tensors, token objects, or string fragments). In a high-throughput scenario, these objects accumulate rapidly in Generation 0. If the allocation rate exceeds the GC's ability to collect, a Gen 0 collection is triggered. While Gen 0 collections are generally fast (stopping the world for only a few milliseconds), in a latency-sensitive AI application—such as real-time voice synthesis—these pauses can introduce jitter.

The dotnet-counters tool allows us to monitor gc-heap-size and gen-0-gc-count in real-time. This is analogous to watching the fuel gauge and engine temperature of a race car while it is on the track. You don't want to wait until the engine overheats (Out of Memory exception) or the fuel runs out (memory fragmentation) to take action. By correlating high Gen 0 collection counts with increased CPU usage, we can infer that our token processing logic is allocating too many temporary objects, necessitating a shift to more aggressive use of ref struct types or object pooling.

JIT Compilation and "Cold Start" Latency

Just-In-Time compilation is another hidden variable in production. When a complex method involving SIMD intrinsics is executed for the first time, the JIT compiler translates the Intermediate Language (IL) into machine code. This introduces a "warm-up" cost.

In the context of AI, consider a dynamic routing mechanism that switches between different model heads based on the input type. The first time a specific branch is taken, the JIT overhead can be significant. dotnet-trace captures JIT compilation events, allowing us to visualize exactly when methods are being compiled. If we observe a spike in CPU usage that correlates with a specific user request pattern, we might identify that a critical path is being JIT-compiled on the hot path.

This theoretical understanding drives the architectural decision to pre-compile hot paths using Tiered Compilation settings or to use ReadyToRun (R2R) binaries, ensuring that the heavy lifting of tokenization and model inference is not hampered by runtime translation overhead.

The Anatomy of a Bottleneck: CPU vs. I/O

AI applications are often CPU-bound due to the mathematical intensity of matrix multiplications. However, they are also frequently I/O-bound when fetching model weights or streaming tokens over a network.

Using dotnet-counters, we monitor cpu-usage and thread-thread-count. A sudden drop in CPU usage while the application is ostensibly "busy" processing a request is a classic sign of an I/O bottleneck. In a token processing pipeline, this might occur when the application waits for a database to retrieve embedding vectors or for a network stream to deliver the next chunk of data.

The theoretical implication here is the "Little's Law" of concurrency: the number of concurrent requests is determined by the arrival rate and the service time. If the service time is inflated by I/O waits, the thread pool must inject more threads to maintain throughput. However, excessive threads lead to context switching overhead. dotnet-trace helps us visualize the thread pool state, revealing if we are suffering from thread starvation or excessive context switching.

Visualizing the Execution Flow

To visualize the flow of data and the points of observation, consider the following diagram. It illustrates how a request traverses the system and where dotnet-counters and dotnet-trace intercept the flow for analysis.

This diagram illustrates the complete execution flow of a request as it traverses the .NET system, highlighting the specific interception points where dotnet-counters and dotnet-trace capture data for performance analysis.
Hold "Ctrl" to enable pan & zoom

This diagram illustrates the complete execution flow of a request as it traverses the .NET system, highlighting the specific interception points where `dotnet-counters` and `dotnet-trace` capture data for performance analysis.

Interpreting the "What If" Scenarios

The theoretical foundation of profiling is not just about observation, but about hypothesis testing.

  1. What if the GC pause time exceeds 50ms? In a synchronous processing model, this blocks the thread. The theoretical solution involves analyzing the allocation profile. If the allocations are rooted in large buffers (e.g., processing a massive context window), we might switch to unmanaged memory allocation via NativeMemory.Alloc or utilize ArrayPool more aggressively to keep objects in Gen 0.

  2. What if the JIT time is dominant? If dotnet-trace shows that the method ProcessTokens is being JIT-compiled repeatedly (perhaps due to dynamic generic specialization), we might refactor the code to use concrete types or enable AggressiveInlining to reduce the overhead of method calls, though this must be balanced against code size.

  3. What if CPU usage is low but throughput is poor? This is the classic "hidden wait" scenario. It implies that threads are blocked, likely on I/O or locks. In AI applications, this often happens when multiple threads compete for access to a shared model instance (locking) or when reading from a slow stream. The theoretical fix involves moving to asynchronous I/O (async/await) or lock-free data structures using Interlocked operations, ensuring that threads are not parked while waiting for data.

The Connection to Previous Optimizations

This chapter relies heavily on the concepts introduced in Book 10, specifically regarding Span<T> and SIMD.

  • Span and Allocation Profiling: In "Chapter 12: Memory Management with Span", we learned that Span<T> is a stack-only type that creates a view over memory. When profiling with dotnet-counters, a reduction in the gc-heap-size metric after refactoring a loop to use Span<T> instead of IEnumerable<string> validates the optimization. The theoretical link is direct: Span<T> reduces GC pressure, which reduces pause times, leading to lower tail latency (p99).
  • SIMD and CPU Profiling: In "Chapter 14: Vectorization with SIMD", we utilized Vector<T> to process data in parallel. When using dotnet-counters, we expect to see higher CPU utilization but lower execution time for the same workload. If the CPU usage remains low while using SIMD, it indicates that the operation is memory-bound (waiting for data to load into the CPU registers) rather than compute-bound. This insight directs us to optimize memory layout (e.g., ensuring arrays are contiguous) rather than further algorithmic tuning.

Theoretical Foundations

dotnet-trace does not just capture data; it captures events from specific providers. The two most critical providers for AI workloads are the Microsoft-Windows-DotNETRuntime provider (for GC and JIT events) and the Microsoft-Windows-Kernel-Thread provider (for thread activity).

When we capture a trace, we are essentially recording a flight data recorder log of the application. The theoretical challenge is filtering the noise. In an AI application, the "noise" might be the telemetry of the application itself. We must configure the trace to focus on the specific process ID (PID) and filter events to those relevant to performance (e.g., GCStart, GCEnd, Method/JITCompilationStart).

For example, a "stop-the-world" GC event is recorded as a GCPause event. By analyzing the duration of these events in the trace, we can calculate the "percent of time in GC." If this exceeds 5-10% in a high-performance AI pipeline, it indicates excessive allocation pressure, requiring a revisit of the memory strategies discussed in earlier chapters.

Theoretical Foundations

The theoretical foundation of profiling in production rests on the premise that optimization is a feedback loop. We hypothesize that a change (e.g., using SIMD) will improve performance, we deploy it, and we use dotnet-counters and dotnet-trace to validate the hypothesis against the chaotic reality of production data.

This approach moves AI development from an art to a science. It replaces intuition with data, allowing us to build robust, high-performance systems that can handle the rigorous demands of real-world AI inference. The tools provided by the .NET runtime are not merely diagnostic utilities; they are the lenses through which we view the complex interactions between our C# code, the runtime, and the underlying hardware.

Basic Code Example

Here is a self-contained example demonstrating how to instrument a C# application for profiling, specifically tailored for an AI token processing scenario.

using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Diagnostics.Metrics;
using System.Linq;
using System.Threading;
using System.Threading.Tasks;

namespace TokenProcessingProfiler
{
    // 1. Define a custom Meter for AI-specific metrics.
    // This adheres to the OpenTelemetry standard and allows dotnet-counters to track these specifically.
    public static class AiMetrics
    {
        public static readonly Meter Meter = new("TokenProcessing.AI", "1.0.0");

        // Counter: Represents a monotonically increasing value (total tokens processed).
        public static readonly Counter<long> TotalTokensProcessed = 
            Meter.CreateCounter<long>("ai.tokens.total", "tokens", "Total number of tokens processed");

        // Histogram: Used for measuring the latency of tokenization operations.
        public static readonly Histogram<double> TokenizationLatency = 
            Meter.CreateHistogram<double>("ai.tokenization.latency", "ms", "Time taken to tokenize a prompt");
    }

    class Program
    {
        // 2. Simulate a realistic AI workload.
        // In a real scenario, this might involve calling an LLM or running a local model.
        // Here, we simulate the latency and CPU load to generate profiling data.
        static async Task Main(string[] args)
        {
            Console.WriteLine("Starting AI Token Processing Simulation...");
            Console.WriteLine("Run the following commands in a separate terminal to observe metrics:");
            Console.WriteLine("  > dotnet-counters monitor --process-id <PID> TokenProcessing.AI");
            Console.WriteLine("  > dotnet-trace collect --process-id <PID> --providers Microsoft-Windows-DotNETRuntime");
            Console.WriteLine("\nPress any key to start processing...");
            Console.ReadKey();

            var cts = new CancellationTokenSource();
            var processingTask = StartProcessingPipeline(cts.Token);

            Console.WriteLine("\nProcessing running. Press 'q' to quit.");
            while (Console.ReadKey().Key != ConsoleKey.Q)
            {
                // Keep running
            }

            cts.Cancel();
            await processingTask;
            Console.WriteLine("\nSimulation stopped.");
        }

        static async Task StartProcessingPipeline(CancellationToken token)
        {
            // 3. Create a background task to simulate continuous incoming requests.
            var pipelineTasks = new List<Task>();

            // We spawn multiple workers to simulate concurrent API requests.
            for (int i = 0; i < 4; i++)
            {
                pipelineTasks.Add(Task.Run(async () => 
                {
                    while (!token.IsCancellationRequested)
                    {
                        await ProcessTokenBatch();
                    }
                }, token));
            }

            await Task.WhenAll(pipelineTasks);
        }

        static async Task ProcessTokenBatch()
        {
            // 4. Start a high-resolution timer to measure latency.
            // Stopwatch is crucial for precise timing in profiling.
            var sw = Stopwatch.StartNew();

            // 5. Simulate the "Tokenization" phase.
            // In a real app, this involves string manipulation or model inference.
            // We add a random delay to mimic network/processing variance.
            var randomDelay = Random.Shared.Next(50, 200);
            await Task.Delay(randomDelay);

            // 6. Simulate the "Inference" phase (CPU intensive).
            // We perform a dummy calculation to spike CPU usage, 
            // allowing dotnet-trace to capture JIT/GC activity.
            double result = 0;
            for (int i = 0; i < 10000; i++)
            {
                result += Math.Sqrt(i) * Math.Sin(i);
            }

            sw.Stop();

            // 7. Record metrics using the custom Meter.
            // This data is exposed to dotnet-counters and OpenTelemetry exporters.
            var tokenCount = Random.Shared.Next(50, 150);
            AiMetrics.TotalTokensProcessed.Add(tokenCount);
            AiMetrics.TokenizationLatency.Record(sw.ElapsedMilliseconds);

            // 8. Simulate occasional GC pressure.
            // Allocating objects forces the Garbage Collector to run.
            // Profiling this helps identify memory bottlenecks in token pipelines.
            if (Random.Shared.Next(0, 10) > 8) 
            {
                // Allocate a moderately large list to trigger Gen0/Gen1 collections
                var _ = new List<byte>(1024 * 100); 
            }
        }
    }
}

Explanation

This example simulates a microservice that processes tokens for an AI application. It is designed specifically to generate observable data for dotnet-counters and dotnet-trace.

1. The Metrics Infrastructure (AiMetrics Class)

In modern .NET, we use the System.Diagnostics.Metrics API, which is the standard for OpenTelemetry and .NET performance counters.

  • Meter: Acts as a factory for metrics. It groups related metrics together under a name and version.
  • Counter: Used for values that only go up (e.g., total requests). This is ideal for dotnet-counters because it requires no state reset.
  • Histogram: Used to capture distributions of values (e.g., request latency). This is essential for understanding the P50, P95, and P99 latencies of your token processing.

2. The Simulation Logic (ProcessTokenBatch)

To profile effectively, we need a workload that mimics real-world behavior:

  • CPU Bound: The for loop calculating Math.Sqrt simulates the mathematical operations involved in model inference or token encoding.
  • I/O Bound: Task.Delay simulates network latency when calling an external LLM API.
  • Memory Allocation: The conditional allocation of List<byte> simulates object creation during string manipulation or result parsing. This forces the Garbage Collector (GC) to work, which we can observe in the trace.

3. The Observation Loop (Main)

The Main method sets up a continuous loop of processing tasks. By running multiple tasks concurrently, we simulate a realistic load scenario where multiple users are hitting the API simultaneously. This concurrency is critical for spotting thread pool starvation or lock contention issues.

Common Pitfalls

  1. Polling Counters vs. Event Counters:

    • Mistake: Using dotnet-counters without understanding that it polls the application. If your application exits immediately (like a console app without a Console.ReadKey or loop), dotnet-counters will show no data because the process lifecycle is too short to attach and poll.
    • Fix: Ensure your application has a long-running lifecycle (e.g., a BackgroundService, a web API, or a loop as shown in the example) to allow the monitoring tools to attach and sample data.

  2. Misinterpreting Histograms:

    • Mistake: Treating dotnet-counters output for histograms as exact point-in-time measurements.
    • Fix: dotnet-counters shows an aggregation (average, percentile) over the polling interval. For deep analysis of specific request traces (e.g., "Why did request X take 500ms?"), you must use dotnet-trace to capture EventSource events and analyze them in PerfView or Visual Studio.

  3. GC Pressure in Production:

    • Mistake: Allocating excessively in the "hot path" (the token processing loop).
    • Fix: In the example, we allocate List<byte> to demonstrate profiling. In real high-performance code, you should use Span<T> or ArrayPool<T> to avoid Gen2 allocations, which cause long GC pauses. Use the captured trace to identify which methods are responsible for the most allocations.

Visualizing the Workflow

The following diagram illustrates the flow of data from the application to the profiling tools.

Diagram: G
Hold "Ctrl" to enable pan & zoom

Step-by-Step Execution Guide

  1. Build the Application: Compile the code using the .NET 7 or .NET 8 SDK. 

    dotnet build

  2. Run dotnet-counters: Open a terminal. Find the Process ID (PID) of the running application. Then run: 

    dotnet-counters monitor --process-id <PID> TokenProcessing.AI

    • What to look for: You will see a live-updating table showing ai.tokens.total increasing and ai.tokenization.latency calculating averages. This confirms that your custom metrics are being emitted correctly.

  3. Run dotnet-trace: In a third terminal (or while dotnet-counters is running), start collecting a trace file: 

    dotnet-trace collect --process-id <PID> --providers Microsoft-Windows-DotNETRuntime:4

    • What to look for: This captures CLR events (GC collections, JIT compilations, Exceptions). Let it run for 30 seconds, then press Enter to stop. You will get a .nettrace file.

  4. Analyze the Trace: Open the .nettrace file in Visual Studio or PerfView.

    • GC Stats: Look for "GC Heap Size" and "Pause Time". If the "Pause Time" spikes, your List<byte> allocation logic (or real memory usage) is causing Garbage Collection pauses.
    • CPU Stacks: Look at the CPU samples. You should see heavy usage in Math.Sqrt and Math.Sin, confirming the simulation is CPU-bound.

  5. Optimization Validation: If you were to optimize the code (e.g., replacing the Math.Sqrt loop with a SIMD vector operation), you would repeat steps 2 and 3. A successful optimization would show:

    • dotnet-counters: Higher throughput (tokens/sec).
    • dotnet-trace: Fewer CPU samples spent in the processing method and potentially reduced GC frequency.

The chapter continues with advanced code, exercises and solutions with analysis, you can find them on the ebook on Leanpub.com or Amazon

Loading knowledge check...

Code License: All code examples are released under the MIT License. Github repo.

Content Copyright: Copyright © 2026 Edgar Milvus | Privacy & Cookie Policy. All rights reserved.

All textual explanations, original diagrams, and illustrations are the intellectual property of the author. To support the maintenance of this site via AdSense, please read this content exclusively online. Copying, redistribution, or reproduction is strictly prohibited.