Skip to content

Chapter 18: The Service Mesh: Resilience and Observability with Istio

Theoretical Foundations

The orchestration of distributed AI agents within a cloud-native ecosystem represents a paradigm shift from monolithic application design to fluid, resilient, and scalable systems. To understand this, we must first establish the theoretical bedrock: the convergence of containerization, orchestration, and intelligent workload management. This is not merely about running Python scripts in Docker containers; it is about constructing a self-healing, adaptive nervous system for AI inference.

The Containerized Agent: Encapsulation and Dependency Isolation

At the atomic level of this architecture lies the Containerized Agent. In previous chapters, specifically within Book 4 ("Modern .NET & Cloud Architecture"), we established the principles of dependency isolation and immutable infrastructure. We learned that an application, regardless of its language, must be packaged with its runtime, libraries, and configuration to ensure consistency across environments.

In the context of AI agents, this concept is paramount. An AI agent is rarely a standalone executable; it is a composite entity comprising:

  1. The Inference Engine: The core logic (often Python-based for ML libraries, but increasingly C# via ML.NET or TorchSharp for high-performance .NET workloads).
  2. The Model Weights: Gigabytes of binary data representing the trained neural network.
  3. The Communication Layer: gRPC or HTTP clients to speak with other agents or the orchestrator.
  4. System Dependencies: Specific versions of CUDA, cuDNN, or other GPU-accelerated libraries.

Why is containerization critical here? Consider the "Dependency Hell" analogy. Imagine a library containing every book ever written, but each book requires a unique, specialized magnifying glass to read it. If you try to read two incompatible books simultaneously, the library collapses. In traditional deployments, installing multiple AI agents on a single host leads to conflicting library versions (e.g., Agent A needs TensorFlow 1.x, Agent B needs TensorFlow 2.x).

Containerization solves this by providing virtual walls. Each agent lives in its own room with its own specialized tools. The host operating system (the "building") provides the foundation, but the agents cannot interfere with each other’s tools. This isolation ensures that an update to the inference engine of one agent does not break the stability of the entire pipeline.

The Orchestrator: Kubernetes as the Conductor

Once agents are containerized, they need a manager. This is where Kubernetes (K8s) enters the theoretical framework. Kubernetes is not just a scheduler; it is a control plane that continuously strives to match the desired state of the system with the actual state.

In an AI inference pipeline, the "desired state" is dynamic. We might declare: "I want 5 instances of the Sentiment Analysis Agent and 3 instances of the Summarization Agent." Kubernetes acts as a conductor in an orchestra. If a violinist (a pod/agent) faints (crashes), the conductor immediately signals a replacement to step in. If the music (traffic) gets louder, the conductor signals more instruments to play.

The Critical Role of Statefulness: While web servers are often stateless (each request is independent), AI agents often require state. This state might be:

  • Session State: Maintaining context in a multi-turn conversation.
  • Model State: The loaded weights in GPU memory (which are expensive to reload).
  • Data State: Caching intermediate results for downstream processing.

Kubernetes manages this through StatefulSets. Unlike a Deployment (which treats pods as interchangeable cattle), a StatefulSet treats pods as pets. They have stable, unique identifiers (e.g., agent-0, agent-1). This is crucial for distributed agents that need to discover each other reliably or maintain persistent connections to a vector database.

Dynamic Scaling: The Horizontal Pod Autoscaler (HPA) and Custom Metrics

The theoretical core of this chapter is Dynamic Scaling. In traditional software, we scale based on CPU or Memory usage. However, AI inference is unique. A model might be computationally idle (low CPU) while waiting for a batch to fill up, or it might be memory-bound (loading a large model) without high CPU utilization.

Therefore, relying on standard metrics is insufficient. We must implement the Horizontal Pod Autoscaler (HPA) with Custom Metrics.

The Analogy: The Restaurant Kitchen Imagine a restaurant kitchen (the inference service).

  • CPU/Memory Scaling: This is like measuring how hot the stoves are. A stove might be burning hot (high CPU), but if the chefs are waiting for ingredients, the kitchen isn't actually working efficiently.
  • Queue-Based Scaling (Custom Metrics): This measures the number of orders on the rail (the request queue). If the rail is full, you call in more chefs. This is the "why" of custom metrics.

In Kubernetes, we query a metrics server (like Prometheus) to get the requests_per_second or queue_length. We then instruct HPA to adjust the replica count based on these business-level signals, not just infrastructure signals.

Traffic Shifting and Inter-Agent Communication with Istio

AI agents rarely operate in isolation. They form a graph of dependencies. An "Orchestrator Agent" might route a user query to a "Router Agent," which then delegates to specific "Specialist Agents" (e.g., a Code Generation Agent or a Math Solver Agent).

Managing this traffic flow is complex. We need to ensure that:

  1. If the Code Generation Agent is overwhelmed, traffic is retried or routed elsewhere.
  2. We can perform A/B Testing (sending 10% of traffic to a new, experimental model version).
  3. We can implement Circuit Breaking (stopping traffic to an agent that is consistently failing).

This is achieved via a Service Mesh, specifically Istio. Istio injects a sidecar proxy (Envoy) into every agent pod. This proxy intercepts all incoming and outgoing network traffic.

The Analogy: The Intelligent Mailroom Imagine a building with many offices (Agents). Without a service mesh, each office handles its own mail. If an office moves, everyone needs to update their address book. With Istio (the Intelligent Mailroom), every piece of mail goes to the mailroom first. The mailroom knows:

  • Which offices are currently open.
  • Which offices are specialized in specific topics.
  • How to route a letter to ensure it arrives quickly (Load Balancing).
  • When to return a letter because the office is closed (Circuit Breaking).

This allows us to decouple the agents. The sender doesn't need to know the exact IP address of the receiver; it just asks the service mesh for "The Summarization Service," and the mesh handles the routing, retries, and security (mTLS).

Resource Quotas and Cost Efficiency

Finally, we must address the economic reality of AI. GPU resources are expensive and scarce. In a shared cluster, an aggressive text-generation agent could consume all available VRAM, starving a critical image-recognition agent.

Kubernetes Resource Quotas and LimitRanges enforce boundaries.

  • Requests: The minimum resources guaranteed to a pod (e.g., 4GB VRAM). This ensures the agent has enough memory to load the model.
  • Limits: The maximum resources a pod can use (e.g., 8GB VRAM). This prevents a memory leak or runaway process from crashing the node.

The Analogy: Hotel Room Allocation Think of the GPU node as a hotel floor.

  • Requests are like booking a room. You guarantee the room is yours, even if you are just sleeping (idle).
  • Limits are the fire code. You cannot expand your furniture into the hallway or adjacent rooms, regardless of how many guests you have.

By strictly defining these, we ensure fair sharing and cost predictability. We can even use Node Selectors to route high-priority agents to "Premium GPU Nodes" (e.g., A100s) and background batch-processing agents to "Standard CPU Nodes."

Architectural Visualization

The following diagram illustrates the flow of a request through this theoretical architecture, highlighting the interaction between the Service Mesh, the Orchestrator, and the Agents.

The diagram depicts a request's journey through a service mesh, orchestrated by a manager that dynamically routes high-priority tasks to powerful GPU nodes and background batch jobs to standard CPU nodes.
Hold "Ctrl" to enable pan & zoom

The diagram depicts a request's journey through a service mesh, orchestrated by a manager that dynamically routes high-priority tasks to powerful GPU nodes and background batch jobs to standard CPU nodes.

Deep Dive: The Role of C# in Modern AI Orchestration

While Python dominates the ML training space, C# has emerged as a first-class citizen in the orchestration and high-performance inference layer, particularly within the .NET ecosystem. The transition from "scripting" to "engineering" is where C# features shine.

1. Interfaces and Dependency Injection for Model Swapping

In the theoretical model of an AI agent, the specific implementation of the model (e.g., OpenAI API vs. a local ONNX runtime) should be abstract. This is crucial for building flexible systems.

Why this matters: Imagine you have a ChatAgent. In development, you might use a local, lightweight model for speed. In production, you might switch to a cloud-based GPT-4 model for quality. Hardcoding the client makes this risky.

C# Interfaces allow us to define a contract. The IInferenceEngine interface guarantees that any implementation will have a GenerateAsync method.

using System.Threading.Tasks;

namespace AI.Agents.Core
{
    // The contract defined in a shared library (referencing previous chapters on Clean Architecture)
    public interface IInferenceEngine
    {
        Task<string> GenerateAsync(string prompt);
    }

    // Implementation 1: Local high-performance model (e.g., using TorchSharp or ML.NET)
    public class LocalLlamaEngine : IInferenceEngine
    {
        public async Task<string> GenerateAsync(string prompt)
        {
            // Logic to run inference on local GPU
            return await Task.FromResult("Local response: " + prompt);
        }
    }

    // Implementation 2: Cloud-based API
    public class OpenAiEngine : IInferenceEngine
    {
        public async Task<string> GenerateAsync(string prompt)
        {
            // Logic to call REST API
            return await Task.FromResult("Cloud response: " + prompt);
        }
    }
}

Dependency Injection (DI): In a cloud-native app, we configure the DI container at startup. This allows the Kubernetes deployment to inject the correct engine based on a configuration map (ConfigMap) without changing the code.

2. IAsyncEnumerable<T> for Streaming Inference

AI inference is latency-sensitive. Users expect tokens to appear as they are generated, not all at once. In HTTP/1.1, this was difficult. With HTTP/2 and gRPC, streaming is native.

C#’s IAsyncEnumerable<T> (introduced in C# 8.0) is the perfect abstraction for this. It allows the agent to yield return tokens as they are generated by the model, back-pressureing the system naturally.

using System.Collections.Generic;
using System.Threading.Tasks;

namespace AI.Agents.Streaming
{
    public class StreamingAgent
    {
        private readonly IInferenceEngine _engine;

        public StreamingAgent(IInferenceEngine engine)
        {
            _engine = engine;
        }

        // Returning a stream of tokens rather than a single string
        public async IAsyncEnumerable<string> StreamResponseAsync(string prompt)
        {
            // Hypothetical method that yields tokens as they are computed
            await foreach (var token in _engine.GenerateStreamAsync(prompt))
            {
                yield return token;
            }
        }
    }
}

Why this is critical for scaling: When using Kubernetes HPA, we often scale based on "concurrent requests." If a request holds a connection open for 30 seconds while streaming, the load balancer sees a long-lived connection. IAsyncEnumerable ensures that the connection remains active but responsive, allowing the orchestrator to accurately gauge the load on the agent.

3. record Types for Immutable State Management

AI agents often need to pass context between steps. This context (history, metadata, intermediate results) should be immutable to prevent side effects in a distributed system.

C# record types provide value-based equality and immutability out of the box. This is ideal for representing the state of a conversation or the configuration of a model.

namespace AI.Agents.State
{
    // Immutable state representation
    public record InferenceContext(
        string SessionId, 
        int MaxTokens, 
        float Temperature, 
        Dictionary<string, object> Metadata
    );

    public class ContextManager
    {
        public InferenceContext UpdateContext(InferenceContext current, string newKey, object newValue)
        {
            // Records support non-destructive mutation (with)
            return current with { Metadata = current.Metadata.Concat(new[]{new KeyValuePair<string, object>(newKey, newValue)}).ToDictionary() };
        }
    }
}

Architectural Implication: In a distributed trace (e.g., via OpenTelemetry), passing these immutable records ensures that logs across different microservices (Orchestrator -> Specialist) correlate perfectly without shared mutable memory.

4. Span<T> and Memory Management for High-Throughput Inference

When dealing with massive model weights or high-throughput tokenization, memory allocation becomes a bottleneck. The Garbage Collector (GC) in .NET can cause pauses (latency spikes) if too many short-lived objects are created.

C#’s Span<T> and Memory<T> allow us to work with slices of memory without allocating new objects. In an AI agent processing large batches of text tokens, using Span<T> to slice a large array of integers (tokens) reduces pressure on the GC, ensuring the inference loop remains smooth.

using System;

namespace AI.Agents.Performance
{
    public class Tokenizer
    {
        // Processing a batch of tokens without allocating new arrays
        public void ProcessBatch(ReadOnlyMemory<int> tokenBatch)
        {
            ReadOnlySpan<int> span = tokenBatch.Span;

            // Iterate over the slice of memory
            for (int i = 0; i < span.Length; i++)
            {
                // Process token logic here
                var token = span[i];
            }
        }
    }
}

Theoretical Foundations

By combining these elements—containerization, Kubernetes orchestration, service mesh traffic management, and C#’s modern concurrency and memory features—we achieve a system that is:

  1. Resilient: Agents fail and restart automatically (K8s Self-Healing).
  2. Scalable: Agents multiply or shrink based on actual demand (HPA with Custom Metrics).
  3. Efficient: Code utilizes modern language features to minimize latency and resource usage (C# Async/Streaming).
  4. Decoupled: Agents communicate via abstract contracts, allowing independent evolution (Service Mesh & Interfaces).

This theoretical foundation sets the stage for the practical implementation of deploying these agents into a live Kubernetes cluster, which we will explore in the subsequent sections.

Basic Code Example

Here is a comprehensive guide to containerizing a distributed AI agent using modern .NET (C#) and Docker, specifically tailored for a Kubernetes environment.

The Real-World Context: The "Sentinel" Anomaly Detector

Imagine a manufacturing plant where hundreds of sensors stream telemetry data (vibration, temperature, pressure) to a central system. We need a lightweight Edge AI Agent that runs in a container on the factory floor. Its job is to:

  1. Receive a stream of sensor readings.
  2. Run a lightweight inference model (e.g., a decision tree or a small neural network) locally to detect anomalies.
  3. If an anomaly is detected, it immediately sends an alert to a central command queue.
  4. It must be resilient, scalable, and easy to deploy across multiple edge nodes.

We will build a C# console application that simulates this agent, containerize it using a multi-stage Docker build for optimization, and prepare it for Kubernetes orchestration.

The C# Code: Program.cs

This code uses modern .NET 6+ features, including Top-Level Statements, IHostedService for background processing, and the System.Threading.Channels library for high-performance, asynchronous data streaming.

// System namespaces for core functionality
using System;
using System.Threading;
using System.Threading.Tasks;
using System.Threading.Channels;

// Microsoft.Extensions namespaces for Dependency Injection and Hosting
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;

namespace SentinelAgent
{
    // 1. Domain Model: Represents a sensor reading
    public record SensorData(string SensorId, double Value, DateTime Timestamp);

    // 2. Domain Model: Represents an anomaly alert
    public record AnomalyAlert(string SensorId, double Value, string Reason);

    // 3. The Core Agent Logic: Inference Engine
    // In a real scenario, this would load a ONNX or ML.NET model.
    public interface IInferenceEngine
    {
        bool IsAnomaly(SensorData data);
    }

    public class SimpleThresholdEngine : IInferenceEngine
    {
        // Simulating a model threshold. In production, this comes from a config map or model file.
        private const double Threshold = 90.0;

        public bool IsAnomaly(SensorData data)
        {
            // Simulate inference latency (e.g., matrix multiplication)
            Thread.Sleep(10); 
            return data.Value > Threshold;
        }
    }

    // 4. The Alerting Service (Output)
    public interface IAlertDispatcher
    {
        Task SendAlertAsync(AnomalyAlert alert, CancellationToken cancellationToken);
    }

    public class ConsoleAlertDispatcher : IAlertDispatcher
    {
        private readonly ILogger<ConsoleAlertDispatcher> _logger;

        public ConsoleAlertDispatcher(ILogger<ConsoleAlertDispatcher> logger)
        {
            _logger = logger;
        }

        public Task SendAlertAsync(AnomalyAlert alert, CancellationToken cancellationToken)
        {
            // In production, this would push to RabbitMQ, Azure Service Bus, or Kafka
            _logger.LogWarning("ALERT TRIGGERED: Sensor {Id} reported {Value}. Reason: {Reason}", 
                alert.SensorId, alert.Value, alert.Reason);
            return Task.CompletedTask;
        }
    }

    // 5. The Data Ingestion Service (Input)
    // Uses Channels for high-throughput, non-blocking data streaming
    public class SensorIngestionService
    {
        private readonly Channel<SensorData> _channel;

        public SensorIngestionService()
        {
            // Bounded channel prevents memory overflows if ingestion outpaces processing
            var options = new BoundedChannelOptions(1000)
            {
                FullMode = BoundedChannelFullMode.Wait
            };
            _channel = Channel.CreateBounded<SensorData>(options);
        }

        public ChannelWriter<SensorData> Writer => _channel.Writer;
        public ChannelReader<SensorData> Reader => _channel.Reader;
    }

    // 6. The Background Worker (The Agent Host)
    public class AgentWorker : BackgroundService
    {
        private readonly SensorIngestionService _ingestionService;
        private readonly IInferenceEngine _inferenceEngine;
        private readonly IAlertDispatcher _dispatcher;
        private readonly ILogger<AgentWorker> _logger;

        public AgentWorker(
            SensorIngestionService ingestionService,
            IInferenceEngine inferenceEngine,
            IAlertDispatcher dispatcher,
            ILogger<AgentWorker> logger)
        {
            _ingestionService = ingestionService;
            _inferenceEngine = inferenceEngine;
            _dispatcher = dispatcher;
            _logger = logger;
        }

        protected override async Task ExecuteAsync(CancellationToken stoppingToken)
        {
            _logger.LogInformation("Agent Worker started. Listening for sensor data...");

            // Read from the channel continuously
            await foreach (var data in _ingestionService.Reader.ReadAllAsync(stoppingToken))
            {
                // Run inference
                if (_inferenceEngine.IsAnomaly(data))
                {
                    var alert = new AnomalyAlert(data.SensorId, data.Value, "Threshold Exceeded");
                    await _dispatcher.SendAlertAsync(alert, stoppingToken);
                }
                else
                {
                    _logger.LogDebug("Sensor {Id} reading {Value} is normal.", data.SensorId, data.Value);
                }
            }
        }
    }

    // 7. Simulated Data Generator (To make the example runnable)
    public class DataGenerator : BackgroundService
    {
        private readonly SensorIngestionService _ingestionService;
        private readonly Random _random = new();
        private readonly ILogger<DataGenerator> _logger;

        public DataGenerator(SensorIngestionService ingestionService, ILogger<DataGenerator> logger)
        {
            _ingestionService = ingestionService;
            _logger = logger;
        }

        protected override async Task ExecuteAsync(CancellationToken stoppingToken)
        {
            int iteration = 0;
            while (!stoppingToken.IsCancellationRequested)
            {
                iteration++;

                // Simulate varying sensor values. 
                // Occasionally generate a high value (>90) to trigger an anomaly.
                double value = _random.NextDouble() * 100; 
                if (iteration % 20 == 0) value = 95.0; // Force anomaly every 20 ticks

                var data = new SensorData($"Sensor-{_random.Next(1, 5)}", value, DateTime.UtcNow);

                // Write to the channel (non-blocking)
                await _ingestionService.Writer.WriteAsync(data, stoppingToken);

                _logger.LogDebug("Generated data: {Id} = {Value}", data.SensorId, data.Value);

                // Simulate sensor polling rate
                await Task.Delay(500, stoppingToken);
            }
        }
    }

    // 8. Main Entry Point (DI Configuration)
    public class Program
    {
        public static async Task Main(string[] args)
        {
            var host = Host.CreateDefaultBuilder(args)
                .ConfigureServices((context, services) =>
                {
                    // Register Singleton for the channel wrapper
                    services.AddSingleton<SensorIngestionService>();

                    // Register Transient/Scoped implementations
                    services.AddSingleton<IInferenceEngine, SimpleThresholdEngine>();
                    services.AddSingleton<IAlertDispatcher, ConsoleAlertDispatcher>();

                    // Register Hosted Services (Background Tasks)
                    services.AddHostedService<AgentWorker>();
                    services.AddHostedService<DataGenerator>(); // Simulates external input
                })
                .ConfigureLogging(logging =>
                {
                    // Clear default providers to simplify console output for Docker logs
                    logging.ClearProviders();
                    // Add simple console logging
                    logging.AddConsole();
                    // Set minimum level to Information to see alerts, Debug to see data flow
                    logging.SetMinimumLevel(LogLevel.Information);
                })
                .Build();

            await host.RunAsync();
        }
    }
}

Detailed Line-by-Line Explanation

1. Domain Models (SensorData, AnomalyAlert)

  • Lines 12-13: We define SensorData as a record. In modern C#, record types are immutable by default and provide value-based equality. This is crucial in distributed systems to ensure that data passed between agents or threads isn't accidentally modified downstream.
  • Line 15: AnomalyAlert is also a record. It encapsulates the output payload.

2. The Inference Engine (IInferenceEngine, SimpleThresholdEngine)

  • Lines 18-28: We define an interface IInferenceEngine. This abstraction allows us to swap the underlying model (e.g., from a simple threshold to a complex ONNX runtime) without changing the worker logic.
  • Line 26: Thread.Sleep(10) simulates the computational cost of running a machine learning inference. In a real scenario, this is where predictionEngine.Predict() would be called.
  • Line 27: The logic checks if the sensor value exceeds 90.0. This is our "model."

3. The Alert Dispatcher (IAlertDispatcher)

  • Lines 31-45: This interface decouples the detection logic from the notification mechanism.
  • Line 43: We use ILogger to output the alert. In a Kubernetes environment, this writes to stdout/stderr, which is captured by the K8s logging driver (e.g., Fluentd) and sent to a central log store (like Elasticsearch or Azure Monitor).

4. The Ingestion Service (SensorIngestionService)

  • Lines 48-61: This is the backbone of the data pipeline. We use System.Threading.Channels.
  • Line 54: BoundedChannelOptions sets a limit of 1000 messages. If the producer (sensor generator) fills the buffer faster than the consumer (agent worker) can process, the channel will apply backpressure (wait) rather than crashing the app with an OutOfMemoryException. This is vital for stability in Kubernetes.

5. The Agent Worker (AgentWorker)

  • Lines 64-82: This inherits from BackgroundService, the standard .NET way to implement long-running processes.
  • Line 76: ReadAllAsync is an async iterator. It efficiently pulls data from the channel as soon as it's available, yielding control when the channel is empty (zero CPU waste).
  • Line 78-81: The core business logic: Check for anomaly -> Create Alert -> Dispatch.

6. The Data Generator (DataGenerator)

  • Lines 85-108: Since we don't have physical sensors connected to our dev machine, this service simulates them.
  • Line 99: It writes to the ChannelWriter. This is non-blocking and thread-safe.
  • Line 104: Task.Delay(500) simulates a sensor that sends data every 500ms.

7. Main Entry Point & DI (Program)

  • Lines 112-131: We use the Generic Host (IHost), which provides dependency injection, configuration, and logging setup automatically.
  • Line 119: services.AddSingleton<SensorIngestionService>() ensures that the channel is shared between the DataGenerator (writer) and the AgentWorker (reader) within the same process.
  • Line 125: We register both background services. The host will start them in parallel when RunAsync is called.

Containerization: The Dockerfile

To run this in Kubernetes, we need a container image. We use a Multi-Stage Build to keep the final image small and secure.

# Stage 1: Build Environment
# Uses the .NET SDK to restore dependencies and compile the app
FROM mcr.microsoft.com/dotnet/sdk:8.0 AS build
WORKDIR /src

# Copy the project file and restore (caching layer)
COPY SentinelAgent.csproj .
RUN dotnet restore

# Copy the rest of the source code
COPY . .
RUN dotnet publish -c Release -o /app/publish

# Stage 2: Runtime Environment
# Uses the smaller ASP.NET runtime image (includes ASP.NET Core, but we only use the base runtime)
FROM mcr.microsoft.com/dotnet/aspnet:8.0 AS final
WORKDIR /app

# Copy the published artifacts from the build stage
COPY --from=build /app/publish .

# Set the entry point
ENTRYPOINT ["dotnet", "SentinelAgent.dll"]

Dockerfile Explanation:

  1. Stage 1 (Build): We use the full SDK image to compile. This image is large (~700MB) but contains compilers.
  2. Stage 2 (Runtime): We switch to the aspnet:8.0 image. This is optimized for running .NET apps and is much smaller (~200MB).
  3. Multi-Stage Benefit: The final image contains only the compiled DLLs and the runtime, not the C# compiler or NuGet caches. This reduces attack surface and download times in Kubernetes.

Kubernetes Deployment Manifest

Here is how you would deploy this agent to a Kubernetes cluster. This manifests includes a Deployment (to manage pods) and a Service (to expose metrics, though our app doesn't expose HTTP endpoints, in a real scenario it would expose a /metrics endpoint for Prometheus).

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sentinel-agent
  labels:
    app: sentinel-agent
spec:
  replicas: 3  # Start with 3 instances for high availability
  selector:
    matchLabels:
      app: sentinel-agent
  template:
    metadata:
      labels:
        app: sentinel-agent
    spec:
      containers:

      - name: agent
        image: myregistry/sentinel-agent:v1.0
        imagePullPolicy: Always
        resources:
          requests:
            memory: "64Mi"
            cpu: "50m" # 0.05 CPU cores
          limits:
            memory: "128Mi"
            cpu: "200m" # 0.2 CPU cores
        # Liveness Probe: Restarts the container if the app crashes
        livenessProbe:
          exec:
            command:

            - /bin/sh
            - -c
            - "ps aux | grep dotnet"
          initialDelaySeconds: 10
          periodSeconds: 5
        # Readiness Probe: Ensures traffic isn't sent until the app is ready
        readinessProbe:
          tcpSocket:
            port: 80 # Assuming the app eventually listens on port 80, otherwise remove
          initialDelaySeconds: 5
          periodSeconds: 10
apiVersion: v1
kind: Service
metadata:
  name: sentinel-agent-service
spec:
  selector:
    app: sentinel-agent
  ports:

    - protocol: TCP
      port: 80
      targetPort: 80
  type: ClusterIP

Kubernetes Manifest Explanation:

  1. Replicas (Line 7): We set replicas: 3. Kubernetes ensures 3 copies of your agent are always running. If one node fails, K8s reschedules the pod elsewhere.
  2. Resources (Lines 18-23): We define Requests (guaranteed resources) and Limits (hard caps). This is critical for cost efficiency. We request only 50m CPU (1/20th of a core) and 64Mi RAM.
  3. Liveness Probe (Lines 24-31): Kubernetes runs this command periodically. If it fails (e.g., the C# app crashes), K8s restarts the container.
  4. Readiness Probe (Lines 32-38): Tells K8s when the pod is ready to accept traffic. If this fails, K8s removes the pod from the Service load balancer.

Visualizing the Architecture

The following diagram illustrates how the C# Agent fits into the Kubernetes ecosystem, handling the flow of data from sensors to alerts.

<Sensors> (External) 
    |
    v
[Kubernetes Service / Ingress]
    |
    v
+-------------------------------------------------------+
|  Kubernetes Cluster (Node 1)                          |
|                                                       |
|  +-------------------+       +-------------------+    |
|  |   Pod (Agent)     |       |   Pod (Agent)     |    |
|  |   [C# Process]    |       |   [C# Process]    |    |
|  |                   |       |                   |    |
|  |  Data Ingestion   |       |  Data Ingestion   |    |
|  |  -> Channel       |       |  -> Channel       |    |
|  |  -> Inference     |       |  -> Inference     |    |
|  |  -> Alerting      |       |  -> Alerting      |    |
|  +-------------------+       +-------------------+    |
|           ^                             ^             |
|           | (Write)                     | (Write)     |
|           |                             |             |
|  +-------------------------------------------------+  |
|  |           Shared Message Broker (Kafka/RabbitMQ)|  |
|  |           (Central Command Queue)              |  |
|  +-------------------------------------------------+  |
+-------------------------------------------------------+

Common Pitfalls

  1. Blocking the Event Loop:

    • Mistake: Using Thread.Sleep() or synchronous I/O (e.g., HttpClient.Send()) directly inside the ExecuteAsync method without await.
    • Consequence: In C#, the main thread is single-threaded for logic execution (until I/O occurs). Blocking it prevents the application from processing other events or responding to shutdown signals gracefully. This causes high latency and unresponsiveness in K8s.
    • Fix: Always use await Task.Delay() and async/await for all I/O operations.

  2. Ignoring Channel Capacity:

    • Mistake: Creating an UnboundedChannel for high-throughput data.
    • Consequence: If the consumer (inference engine) slows down (e.g., model is complex), the unbounded channel will grow indefinitely, consuming all available RAM and causing the Pod to be OOMKilled (Out of Memory) by Kubernetes.
    • Fix: Always use BoundedChannel with a reasonable limit. Handle the FullMode strategy (e.g., Wait or DropOldest) based on your business logic.

  3. Missing Resource Limits in K8s:

    • Mistake: Deploying the container without defining resources.limits.
    • Consequence: If the agent has a memory leak or a traffic spike, it will consume all RAM on the Kubernetes Node. This can crash the Node itself, affecting other unrelated applications running on the same hardware.
    • Fix: Always set requests and limits. Use the .NET dotnet-counters tool to monitor actual usage and tune these values.

  4. Baking Secrets into the Image:

    • Mistake: Putting connection strings or API keys in appsettings.json and committing them to the Docker image.
    • Consequence: Anyone with access to the Docker image can extract the secrets.
    • Fix: Use Kubernetes Secrets and mount them as environment variables or files at runtime. The C# app should read them via IConfiguration.

The chapter continues with advanced code, exercises and solutions with analysis, you can find them on the ebook on Leanpub.com or Amazon

Loading knowledge check...

Code License: All code examples are released under the MIT License. Github repo.

Content Copyright: Copyright © 2026 Edgar Milvus | Privacy & Cookie Policy. All rights reserved.

All textual explanations, original diagrams, and illustrations are the intellectual property of the author. To support the maintenance of this site via AdSense, please read this content exclusively online. Copying, redistribution, or reproduction is strictly prohibited.