Skip to content

Chapter 5: Scaling Principles: Latency, Throughput, and State Management

Theoretical Foundations

The theoretical foundation for deploying containerized AI agents at scale rests upon three pillars: elastic scaling, state persistence, and throughput optimization. In a cloud-native environment, an AI agent is not a static executable; it is a dynamic service that must react to fluctuating demand while maintaining conversational context and minimizing inference latency. This requires moving beyond simple container orchestration to a sophisticated interplay of runtime metrics, distributed data structures, and memory management techniques.

The Analogy: The High-Volume Restaurant Kitchen

To understand these concepts, consider a high-volume restaurant kitchen during the dinner rush.

  1. The Agent Pod (The Chef): A single instance of an AI agent running in a container is like a chef. The chef has a limited capacity to process orders (inference requests).
  2. Horizontal Pod Autoscaling (HPA) (The Kitchen Manager): The manager watches the chefs. If the queue of orders grows or the chefs are sweating (high GPU/CPU usage), the manager hires more chefs (scales out pods). If the rush ends, the manager sends chefs home (scales in) to save money (resources).
  3. Distributed Caching (The Recipe Book & Prep Station): A chef cannot remember every order for every table perfectly while cooking. They rely on a shared recipe book (distributed cache like Redis) and a prep station where ingredients are pre-arranged. If Chef A starts a complex sauce for Table 5 but gets reassigned, Chef B can step in, look at the prep station (cache), and continue without starting over.
  4. Request Batching (The Plating Station): Instead of plating one dish at a time as soon as it's ready, the kitchen gathers several dishes that finish at similar times and plates them together in one swift motion. This reduces the number of trips to the pass (GPU memory bandwidth) and maximizes the use of the garnish spoon (compute cycle).

Horizontal Pod Autoscaling (HPA) and Custom Metrics

Standard HPA in Kubernetes typically scales based on CPU or memory usage. However, for AI agents, these are poor proxies for load. A GPU might be at 100% utilization processing a single, massive batch of requests, or it might be idle waiting for a network response.

Why Custom Metrics Matter: In AI inference, the true bottleneck is often the queue depth of requests waiting for the GPU or the GPU memory allocation. We need to scale based on the metric that directly correlates with the agent's ability to respond: inference latency or GPU utilization.

The C# Context: While C# does not natively manage Kubernetes HPA configurations, it is the language of the agent's runtime environment. The agent must expose these metrics to the Kubernetes Metrics Server. In modern .NET, this is achieved using System.Diagnostics.Metrics (introduced in .NET 6), which provides a high-performance, low-allocation API for emitting metrics.

We use an IMeter to track the "Time to First Token" (TTFT) or "Tokens Per Second" (TPS). If the average TTFT exceeds a threshold (e.g., 500ms), the C# application emits a custom metric. An adapter (like Prometheus) scrapes this metric, and the HPA controller uses it to calculate the replica count.

using System.Diagnostics;
using System.Diagnostics.Metrics;

// The meter is the source of truth for the agent's health.
public class InferenceMetrics
{
    private static readonly Meter _meter = new("AI.Agent.Inference");

    // Tracks the latency of generating a response.
    private static readonly Histogram<double> _generationLatency = _meter.CreateHistogram<double>(
        "agent.generation.latency.ms", 
        "ms", 
        "Time taken to generate a response");

    // Tracks the queue depth (how many requests are waiting for the GPU).
    private static readonly ObservableGauge<int> _queueDepth = _meter.CreateObservableGauge<int>(
        "agent.queue.depth",
        () => RequestQueue.Count, // Callback to read current queue size
        "requests",
        "Number of requests waiting for inference");

    public void RecordLatency(double latencyMs)
    {
        _generationLatency.Record(latencyMs);
    }
}

Architectural Implication: By decoupling the scaling trigger from generic CPU usage to domain-specific metrics (latency/queue depth), we achieve intent-based scaling. The system scales not because the CPU is busy, but because the user experience is degrading. This requires a deep integration with the .NET runtime's instrumentation capabilities.

Distributed Caching for Agent State

AI agents are inherently stateful during a session. A conversation requires context (previous messages, tool outputs, memory). However, in a horizontally scaled environment, a user's request might land on Pod A, while their next request lands on Pod B. If Pod A crashes or scales down, the conversation history is lost if it resides solely in that pod's RAM.

The Problem of Ephemeral State: Containers are ephemeral. Relying on InMemoryCache or static variables for session state is a critical failure point in distributed systems. We need a mechanism to externalize state so that any pod can pick up a conversation exactly where the previous one left off.

The Solution: Distributed Caching (Redis/Memcached) We treat the distributed cache as the "short-term memory" of the agent cluster. The C# IDistributedCache interface abstracts the underlying storage provider, allowing us to swap between Redis, SQL Server, or NCache without changing the agent logic.

Why C# Interfaces are Crucial here: In Book 6, we discussed Dependency Injection (DI) and Interfaces. We defined an IConversationStore. In this chapter, we implement that interface using IDistributedCache. This allows us to inject a Redis-backed implementation in production and a memory-backed implementation in unit tests.

Serialization and Memory Efficiency: Storing complex agent state (which might include System.Text.Json objects or binary tensors) requires efficient serialization. .NET 8 introduced System.Text.Json source generators, which allow for high-performance, reflection-free serialization. This is critical because the overhead of serializing/deserializing state on every request can negate the benefits of caching.

using Microsoft.Extensions.Caching.Distributed;
using System.Text.Json;
using System.Text.Json.Serialization.Metadata;

public interface IAgentStateStore
{
    Task<T?> GetStateAsync<T>(string sessionId, CancellationToken ct);
    Task SetStateAsync<T>(string sessionId, T state, CancellationToken ct);
}

public class RedisAgentStateStore : IAgentStateStore
{
    private readonly IDistributedCache _cache;

    public RedisAgentStateStore(IDistributedCache cache)
    {
        _cache = cache;
    }

    public async Task<T?> GetStateAsync<T>(string sessionId, CancellationToken ct)
    {
        byte[]? data = await _cache.GetAsync(sessionId, ct);
        if (data == null) return default;

        // Using Source Generators for zero-reflection deserialization
        // This is a modern C# feature that drastically improves performance.
        return JsonSerializer.Deserialize(data, typeof(T)) as T;
    }

    public async Task SetStateAsync<T>(string sessionId, T state, CancellationToken ct)
    {
        byte[] data = JsonSerializer.SerializeToUtf8Bytes(state);

        var options = new DistributedCacheEntryOptions
        {
            SlidingExpiration = TimeSpan.FromMinutes(30) // Evict inactive sessions
        };

        await _cache.SetAsync(sessionId, data, options, ct);
    }
}

Architectural Implication: By externalizing state, we enable stateless pods. The pods contain only the logic and the loaded AI model weights; the data flows in and out. This allows for aggressive scaling. If a pod becomes unresponsive, the Kubernetes controller terminates it, and the next request simply retrieves the state from the cache and initializes a new pod. This pattern is essential for high-availability AI agents.

Request Batching and Memory Management

AI models, particularly Large Language Models (LLMs), are computationally expensive. The cost of a single inference is dominated by memory bandwidth and matrix multiplication. Processing requests one by one (synchronous processing) leaves the GPU underutilized because memory access patterns are not optimized.

The Concept of Batching: Batching involves collecting multiple inference requests and processing them simultaneously in a single forward pass of the model.

  • Static Batching: Grouping requests that arrive within a fixed time window.
  • Dynamic Batching: Grouping requests of varying lengths (token counts) efficiently to maximize GPU utilization without exceeding memory limits.

The C# Context: System.Threading.Channels In C#, we use System.Threading.Channels to implement a producer-consumer pattern for request batching. This is a modern, high-performance alternative to BlockingCollection or ConcurrentQueue.

  1. The Producer: The HTTP endpoint receives a request. Instead of calling the model immediately, it writes the request (payload + completion source) into a Channel.
  2. The Consumer: A background service (a BackgroundService in .NET) reads from the channel. It accumulates requests until a threshold is met (e.g., batch size = 32, or timeout = 10ms).
  3. The Batcher: Once the threshold is met, the consumer takes the batch, formats the input tensors, and sends them to the inference engine (e.g., ONNX Runtime, TorchSharp, or an external Python process via gRPC).

Why Channels? Channels are "async-ready" and highly efficient. They handle backpressure automatically. If the consumer is slow (GPU is busy), the channel's internal buffer fills up, and the producer's write operation awaits, naturally throttling the incoming HTTP requests without crashing the server.

using System.Threading.Channels;

public class BatchingService
{
    private readonly Channel<InferenceRequest> _channel;
    private readonly ModelRunner _modelRunner;

    public BatchingService(ModelRunner modelRunner)
    {
        // Create a bounded channel to prevent memory exhaustion.
        _channel = Channel.CreateBounded<InferenceRequest>(new BoundedChannelOptions(1000)
        {
            FullMode = BoundedChannelFullMode.Wait // Apply backpressure
        });
        _modelRunner = modelRunner;
    }

    public async ValueTask EnqueueAsync(InferenceRequest request)
    {
        await _channel.Writer.WriteAsync(request);
    }

    public async Task ProcessBatchesAsync(CancellationToken stoppingToken)
    {
        await foreach (var batch in ReadBatchesAsync(stoppingToken))
        {
            // Execute the batch on the GPU
            await _modelRunner.ExecuteBatchAsync(batch);
        }
    }

    private async IAsyncEnumerable<IReadOnlyList<InferenceRequest>> ReadBatchesAsync(
        [EnumeratorCancellation] CancellationToken ct)
    {
        var batch = new List<InferenceRequest>(capacity: 32);
        var timer = Task.Delay(TimeSpan.FromMilliseconds(10), ct);

        await foreach (var request in _channel.Reader.ReadAllAsync(ct))
        {
            batch.Add(request);

            // Condition 1: Batch is full
            if (batch.Count >= 32)
            {
                yield return batch;
                batch = new List<InferenceRequest>(32);
                timer = Task.Delay(TimeSpan.FromMilliseconds(10), ct);
            }
            // Condition 2: Timeout (latency optimization)
            else if (batch.Count > 0 && await Task.WhenAny(timer, Task.CompletedTask) == timer)
            {
                yield return batch;
                batch = new List<InferenceRequest>(32);
                timer = Task.Delay(TimeSpan.FromMilliseconds(10), ct);
            }
        }
    }
}

Architectural Implication: Batching transforms the cost curve of inference. By increasing throughput per GPU cycle, we reduce the number of pods required to handle the same load, directly impacting cloud costs. However, it introduces latency variance: a request arriving just after a batch is dispatched must wait for the next batch. This is a trade-off between throughput (requests per second) and latency (time to first token).

Integration: The Complete Picture

These three concepts form a feedback loop:

  1. Traffic enters and is enqueued via System.Threading.Channels (Batching).
  2. The Batching Service groups requests and retrieves Agent State from the IDistributedCache (Redis).
  3. The model processes the batch. The Inference Metrics meter records the latency.
  4. The HPA Controller reads the custom metric (e.g., agent.generation.latency.ms). If latency spikes, it scales out the number of pods.
  5. New pods start up, connect to the same Redis cluster, and immediately begin processing from the shared queue.

Visualizing the Architecture

The following diagram illustrates the flow of data and control signals within the cluster.

This diagram illustrates a distributed system where new application pods dynamically connect to a shared Redis cluster to process tasks from a common queue.
Hold "Ctrl" to enable pan & zoom

This diagram illustrates a distributed system where new application pods dynamically connect to a shared Redis cluster to process tasks from a common queue.

Theoretical Foundations

By leveraging modern C# features like System.Threading.Channels for backpressure management, IDistributedCache for state abstraction, and System.Diagnostics.Metrics for observability, we transform a simple AI model into a resilient, cloud-native service. The theoretical goal is to decouple the lifecycle of the compute resources (pods) from the lifecycle of the user session (state), allowing them to scale independently based on real-time demand signals.

Basic Code Example

Here is a simple, self-contained C# example demonstrating a request batching pattern for an AI inference service. This pattern is fundamental for optimizing resource usage (GPU/CPU) in containerized environments, as it allows aggregating individual requests into a single batch to maximize hardware parallelism.

using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;
using System.Threading;
using System.Threading.Tasks;

namespace CloudNativeAI.Microservices.Inference
{
    /// <summary>
    /// Represents a single inference request containing input data and a completion source
    /// to signal the result back to the caller.
    /// </summary>
    /// <typeparam name="TInput">The type of input data (e.g., a string or tensor).</typeparam>
    /// <typeparam name="TOutput">The type of output data (e.g., a prediction or embedding).</typeparam>
    public class InferenceRequest<TInput, TOutput>
    {
        public TInput Input { get; init; }
        public TaskCompletionSource<TOutput> CompletionSource { get; init; }
    }

    /// <summary>
    /// Simulates an AI Model Inference Engine that processes requests in batches.
    /// In a real scenario, this would wrap a TensorFlow.NET or Torch.NET model.
    /// </summary>
    public class InferenceEngine<TInput, TOutput>
    {
        private readonly BlockingCollection<InferenceRequest<TInput, TOutput>> _requestQueue;
        private readonly int _batchSize;
        private readonly int _batchTimeoutMs;
        private readonly CancellationTokenSource _cancellation;
        private readonly Task _processingTask;

        // Simulate a model that takes time to run
        private readonly Func<List<TInput>, List<TOutput>> _modelInferenceFunc;

        public InferenceEngine(int batchSize, int batchTimeoutMs, Func<List<TInput>, List<TOutput>> modelInferenceFunc)
        {
            _batchSize = batchSize;
            _batchTimeoutMs = batchTimeoutMs;
            _modelInferenceFunc = modelInferenceFunc;

            // Bounded capacity prevents memory exhaustion if the queue grows too large
            _requestQueue = new BlockingCollection<InferenceRequest<TInput, TOutput>>(boundedCapacity: 1000);
            _cancellation = new CancellationTokenSource();

            // Start the background processor
            _processingTask = Task.Run(() => ProcessBatchesAsync(_cancellation.Token));
        }

        /// <summary>
        /// Enqueues a request to be processed.
        /// </summary>
        public Task<TOutput> InferAsync(TInput input)
        {
            var tcs = new TaskCompletionSource<TOutput>();
            var request = new InferenceRequest<TInput, TOutput>
            {
                Input = input,
                CompletionSource = tcs
            };

            // Non-blocking addition to the queue
            _requestQueue.Add(request);

            return tcs.Task;
        }

        /// <summary>
        /// Background loop that collects requests and executes them in batches.
        /// </summary>
        private async Task ProcessBatchesAsync(CancellationToken token)
        {
            while (!token.IsCancellationRequested)
            {
                // 1. Wait for the first request to arrive (blocking)
                if (!_requestQueue.TryTake(out var firstRequest, TimeSpan.FromMilliseconds(100), token))
                {
                    continue;
                }

                var batch = new List<InferenceRequest<TInput, TOutput>> { firstRequest };
                var sw = Stopwatch.StartNew();

                // 2. Try to fill the batch until size limit or timeout is reached
                while (batch.Count < _batchSize && sw.ElapsedMilliseconds < _batchTimeoutMs)
                {
                    // Non-blocking peek/take with short timeout
                    if (_requestQueue.TryTake(out var nextRequest, TimeSpan.FromMilliseconds(10)))
                    {
                        batch.Add(nextRequest);
                    }
                }

                // 3. Execute the batch inference
                await ExecuteBatchAsync(batch);
            }
        }

        private async Task ExecuteBatchAsync(List<InferenceRequest<TInput, TOutput>> batch)
        {
            try
            {
                // Extract inputs
                var inputs = batch.Select(r => r.Input).ToList();

                // Simulate network/IO latency or heavy GPU computation
                await Task.Delay(50); 

                // Run the model inference (CPU/GPU bound)
                // In a real container, this is where the GPU memory is utilized.
                var results = _modelInferenceFunc(inputs);

                // Map results back to individual requests
                for (int i = 0; i < batch.Count; i++)
                {
                    batch[i].CompletionSource.TrySetResult(results[i]);
                }
            }
            catch (Exception ex)
            {
                // Propagate errors to all requests in the batch
                foreach (var req in batch)
                {
                    req.CompletionSource.TrySetException(ex);
                }
            }
        }

        public void Stop()
        {
            _cancellation.Cancel();
            _requestQueue.CompleteAdding();
            _processingTask.Wait();
        }
    }

    // --- Main Program to Demonstrate Usage ---
    class Program
    {
        static async Task Main(string[] args)
        {
            Console.WriteLine("Starting Cloud-Native Inference Batching Demo...");

            // 1. Initialize the Engine with a batch size of 4 and a timeout of 200ms
            // This simulates a model that processes 4 items at once, or waits max 200ms.
            var engine = new InferenceEngine<string, string>(
                batchSize: 4,
                batchTimeoutMs: 200,
                modelInferenceFunc: inputs =>
                {
                    // Simulate AI Model processing (e.g., Sentiment Analysis)
                    Console.WriteLine($"[Model] Processing Batch of {inputs.Count} inputs...");
                    return inputs.Select(i => $"Processed: {i}").ToList();
                });

            // 2. Simulate multiple clients sending requests concurrently
            var tasks = new List<Task>();
            for (int i = 1; i <= 10; i++)
            {
                int id = i;
                tasks.Add(Task.Run(async () =>
                {
                    var sw = Stopwatch.StartNew();
                    Console.WriteLine($"[Client {id}] Sending request...");

                    var result = await engine.InferAsync($"Input_{id}");

                    sw.Stop();
                    Console.WriteLine($"[Client {id}] Received: {result} in {sw.ElapsedMilliseconds}ms");
                }));

                // Small delay to stagger requests slightly
                await Task.Delay(20); 
            }

            await Task.WhenAll(tasks);
            engine.Stop();

            Console.WriteLine("Demo Complete.");
        }
    }
}

Line-by-Line Explanation

1. Data Structures and Definitions

  • InferenceRequest<TInput, TOutput>: This is a simple POCO (Plain Old CLR Object) used to encapsulate a single request. It holds the Input data and a TaskCompletionSource. The TaskCompletionSource is the mechanism that allows asynchronous code to "wait" for a result that will be produced later by the batch processor.
  • BlockingCollection<T>: This is a thread-safe collection provided by .NET's System.Collections.Concurrent namespace. It is ideal for producer-consumer scenarios. When we call .Add(), it adds items; when we call .Take(), it removes items. If the collection is empty, .Take() blocks until an item is available, which is perfect for our background processing loop.

2. The InferenceEngine Class

  • Constructor:
    • boundedCapacity: 1000: This sets a limit on the queue size. In a real microservice, if the request rate exceeds processing capacity, the queue grows indefinitely until OutOfMemory. This limit acts as a backpressure mechanism.
    • Task.Run(...): We immediately spin up a background thread (or pool thread) to run ProcessBatchesAsync. This decouples the request ingestion from the processing logic.
  • InferAsync:
    • This is the public API method called by clients.
    • It creates a TaskCompletionSource, wraps it in our request object, and pushes it into the queue.
    • Crucially, it immediately returns the Task from the TCS. The caller can await this task, but the execution won't complete until the background processor finishes the batch.
  • ProcessBatchesAsync (The Batching Logic):
    • Step 1 (Wait for First Request): The loop starts with _requestQueue.TryTake. This is a blocking call (with a timeout) that waits for at least one request to arrive. Without this, the loop would spin idly consuming CPU.
    • Step 2 (Filling the Batch):
      • Once we have one request, we start a Stopwatch.
      • We enter a while loop that continues as long as the batch count is less than _batchSize AND the elapsed time is less than _batchTimeoutMs.
      • This is a critical logic block: It balances Latency (don't wait too long if we have few requests) vs Throughput (wait to fill the batch to maximize hardware utilization).
    • Step 3 (Execution): Once the batch is full or the timeout hits, we call ExecuteBatchAsync.
  • ExecuteBatchAsync:
    • This method extracts the raw inputs from the list of requests.
    • Simulation: Task.Delay(50) simulates the latency of a GPU inference call. In a real scenario, this would be an expensive matrix multiplication.
    • Result Mapping: The model returns a list of results. We iterate through the batch and pair each result with its specific TaskCompletionSource to unblock the waiting client.

3. The Main Program (Simulation)

  • We instantiate the engine with a batchSize of 4.
  • We spawn 10 client tasks with a small stagger (Task.Delay(20)).
  • Expected Behavior:
    • Clients 1-4 arrive quickly. The engine picks up Client 1, fills the batch with 2, 3, 4, and executes.
    • Client 5 arrives just as the first batch finishes or slightly after.
    • Because of the batchTimeoutMs (200ms), if requests 5, 6, and 7 arrive slowly, the engine won't wait indefinitely; it will process them as a smaller batch (size 3) once the timeout triggers.

Visualizing the Flow

The following diagram illustrates the flow of data from the client through the queue to the model and back.

Diagram: G
Hold "Ctrl" to enable pan & zoom

Common Pitfalls

  1. Unbounded Queue Growth (Memory Leaks):

    • Mistake: Initializing BlockingCollection without a bounded capacity.
    • Consequence: If the inference model is slow or fails, requests pile up in memory. In a container with a memory limit (e.g., 2GB), this causes an OutOfMemoryException and kills the pod, leading to a restart loop.
    • Fix: Always set boundedCapacity. When the queue is full, the Add method blocks or throws, forcing the caller to handle backpressure (e.g., return HTTP 503 Service Unavailable).

  2. Tail Latency Amplification:

    • Mistake: Setting a static batchTimeoutMs that is too high (e.g., 1000ms) for real-time applications.
    • Consequence: If a request arrives just after a batch starts forming, it must wait for the timeout to trigger or the batch to fill. This adds artificial latency to the request.
    • Fix: Use dynamic batching windows or prioritize low-latency paths for critical requests.

  3. Exception Handling in Batches:

    • Mistake: Letting an exception in the model inference crash the background processor loop.
    • Consequence: The entire batching engine stops, and no further requests are processed until the container restarts.
    • Fix: Wrap the _modelInferenceFunc call in a try/catch block (as shown in the example). Ensure that TrySetException is called on all pending requests in the batch so clients know the request failed, rather than hanging indefinitely.

  4. Thread Starvation:

    • Mistake: Performing synchronous, blocking CPU work inside ExecuteBatchAsync (e.g., a heavy for loop calculating matrix math).
    • Consequence: Since the batching loop runs on a Task thread, blocking it prevents the loop from picking up new requests, reducing the effective throughput to 1 batch at a time.
    • Fix: Always offload heavy computation to a background thread or use await Task.Run(() => heavyWork). In the example, we simulate this with Task.Delay, but in reality, the model call should be asynchronous if possible.

The chapter continues with advanced code, exercises and solutions with analysis, you can find them on the ebook on Leanpub.com or Amazon

Loading knowledge check...

Code License: All code examples are released under the MIT License. Github repo.

Content Copyright: Copyright © 2026 Edgar Milvus | Privacy & Cookie Policy. All rights reserved.

All textual explanations, original diagrams, and illustrations are the intellectual property of the author. To support the maintenance of this site via AdSense, please read this content exclusively online. Copying, redistribution, or reproduction is strictly prohibited.