Chapter 6: Server-Sent Events (SSE) for Streaming LLM Tokens

Theoretical Foundations

Server-Sent Events (SSE) represents a fundamental architectural shift in how we design interactive AI interfaces. When building AI web APIs with ASP.NET Core, specifically for serving Large Language Model (LLM) outputs, the traditional request-response cycle is often insufficient. In previous chapters, we established the baseline for creating non-streaming endpoints where the entire LLM response is generated and returned in a single HTTP payload. However, the nature of LLMs is inherently token-by-token; they generate text incrementally, word by word, or even character by character. To mimic the fluidity of human conversation and maintain user engagement, we must expose this incremental generation process to the client in real-time. This is where Server-Sent Events (SSE) becomes not just a feature, but a necessity.

The Limitations of Standard HTTP for AI Streaming

To understand why SSE is the optimal choice for streaming LLM tokens, we must first analyze the limitations of the standard HTTP request-response model used in previous chapters. In a standard model, a client sends a request, and the server processes it. During this processing, the client is effectively in a "waiting" state, often represented by a loading spinner. The server cannot send partial data; it must buffer the entire LLM response before it can transmit anything back over the network.

Imagine a scenario where an LLM takes 15 seconds to generate a paragraph. In a standard request-response cycle:

1. t=0s: Client sends request.
2. t=0s to 15s: Server generates tokens internally (e.g., "The", " quick", " brown", " fox..."). The client sees nothing but a loading indicator.
3. t=15s: Server finishes generation, serializes the full string to JSON, and sends the HTTP response.
4. t=15s + Network Latency: Client receives the payload and renders the entire paragraph instantly.

While functional, this creates a poor user experience. The user has no feedback that the system is working, leading to perceived latency and potential frustration. Furthermore, buffering the entire response consumes significant server memory, especially for long-form content like code generation or detailed summaries.

The Mechanics of Server-Sent Events (SSE)

SSE is a standard defined in the HTML5 specification that allows a server to push data to a client over a single, long-lived HTTP connection. Unlike WebSockets, which are bidirectional, SSE is strictly unidirectional (server-to-client). This simplicity makes it ideal for LLM streaming, where the client initiates the request and the server simply pushes tokens as they are generated.

The Protocol: SSE operates over a standard HTTP connection. The server must set specific headers to instruct the client to treat the response as an event stream:

• Content-Type: text/event-stream: This tells the browser to parse the incoming stream as SSE events.
• Cache-Control: no-cache: Ensures intermediaries (like proxies or CDNs) do not buffer the response.
• Connection: keep-alive: Keeps the TCP connection open.

The Data Format: The data itself is a simple, line-based text format. Each message is separated by two newline characters (\n\n). The format typically looks like this:

event: token
data: The

event: token
data: quick

event: token
data: brown

event: done
data: {}

In the context of LLMs, we usually define custom event types. For example, we might emit an event of type token for every generated token, and an event of type done or error when the stream concludes or fails.

The Role of IAsyncEnumerable in Modern C

In previous chapters, we utilized Task<IActionResult> for synchronous or fully asynchronous operations. However, for streaming, C# provides a powerful interface introduced in .NET Core 3.0: IAsyncEnumerable<T>.

IAsyncEnumerable<T> is to asynchronous streams what IEnumerable<T> is to synchronous collections. It represents a stream of data that can be enumerated asynchronously, meaning the iterator can await the production of the next item.

Why is this crucial for LLMs? LLMs are computationally intensive. Generating the next token involves running a forward pass through a massive neural network. This operation is inherently asynchronous and time-consuming. IAsyncEnumerable<T> allows the C# runtime to yield control back to the web server framework (Kestrel) after each token is generated, allowing the server to handle other requests or flush the network buffer, rather than blocking the thread waiting for the entire generation to complete.

Consider the analogy of a Faucet vs. a Water Tank.

• Standard HTTP (Water Tank): The server fills a massive tank (memory) with water (tokens). Once the tank is full, the server opens the valve, and the client receives the entire volume at once. If the tank is large, this takes a long time to fill and requires significant storage.
• SSE with IAsyncEnumerable (Faucet): The server acts as a faucet. As water (tokens) is produced, it drips out immediately. The client receives a continuous flow. The server doesn't need to store the water; it just passes it through.

Architectural Implications: The Producer-Consumer Pattern

When implementing SSE in ASP.NET Core, we are essentially building a Producer-Consumer pipeline. The LLM (running on the GPU or CPU) is the Producer. The ASP.NET Core controller action is the intermediary, and the HTTP response stream is the Consumer.

1. The Producer (LLM): Generates tokens asynchronously.
2. The Intermediary (Controller): Iterates over the LLM's output. In modern C#, this is done using the await foreach syntax.
3. The Consumer (Client): Reads the SSE stream via the EventSource API.

One of the most critical architectural considerations here is backpressure. If the LLM generates tokens faster than the network can transmit them to the client, the server's memory buffer will fill up, potentially leading to an OutOfMemoryException or high latency. IAsyncEnumerable combined with the Channel<T> class (often used under the hood by ASP.NET Core's streaming mechanisms) helps manage this flow, ensuring that the producer doesn't overwhelm the consumer.

The Frontend Integration: The EventSource API

While the backend handles the stream generation, the frontend must be capable of consuming it. The native browser API for this is EventSource. Unlike fetch or XMLHttpRequest, EventSource is designed specifically to handle long-lived connections and automatically reconnects if the connection drops (a common occurrence with mobile networks or unstable Wi-Fi).

The Analogy: A Radio Broadcast Think of SSE as a radio station. The client (radio receiver) tunes into a specific frequency (the API endpoint). Once tuned in, the client passively listens. The server (radio tower) broadcasts continuously. If the signal is lost (network blip), the radio automatically attempts to retune. The client doesn't need to ask "Did I miss anything?"—the server simply pushes the next relevant segment.

In the context of our AI application, this means the frontend can update the UI incrementally. As each token arrives via the onmessage event handler, the application can append it to the DOM, creating the illusion of instant typing.

Comparison with Alternatives: gRPC Streaming and WebSockets

It is vital to understand why SSE is chosen over other protocols in this specific context of AI Web APIs.

gRPC Streaming: gRPC offers robust bidirectional streaming capabilities. However, it requires HTTP/2 and often introduces complexity regarding browser support (requiring proxies or specific configurations) and serialization (Protobuf vs. JSON). For a standard web API intended to be consumed by browsers directly, SSE over HTTP/1.1 or HTTP/2 is simpler and universally supported without additional libraries.

WebSockets: WebSockets provide full-duplex communication. While powerful, they are overkill for LLM streaming. LLMs are generally one-way streams of data from server to client during generation. The client sends a request, and the server responds with a stream. WebSockets require a handshake and maintain a more complex state. SSE is lightweight, stateless (from the connection perspective), and integrates seamlessly with standard HTTP middleware (authentication, logging, etc.) because it is just HTTP.

Handling Edge Cases in SSE

When building AI APIs, we must account for specific edge cases inherent to LLMs and network communication:

1. Context Length Exceeded: If the LLM hits its maximum token limit mid-stream, the server must gracefully close the stream with an error event.
2. Model Hallucinations/Filtering: If the LLM generates content that violates safety filters, the stream must be aborted immediately. This requires the server to have the ability to terminate the IAsyncEnumerable iteration and send a specific error event to the client.
3. Network Interruptions: If a user's internet connection drops while receiving a stream, the server might continue generating tokens, wasting compute resources. While SSE doesn't solve this natively, we can implement "heartbeat" messages (empty comments sent every 15 seconds) to keep the connection alive and detect dead connections faster.

Visualizing the Data Flow

To better visualize the flow of data from the LLM to the client via SSE in ASP.NET Core, consider the following diagram:

Hold "Ctrl" to enable pan & zoom

A diagram illustrating how Server-Sent Events (SSE) maintain a persistent connection between an ASP.NET Core server and a client, where heartbeat messages are periodically sent to keep the line alive and detect failures.

Deep Dive: The yield return Keyword and State Management

In C#, the yield return keyword is the backbone of iterators. When used with IAsyncEnumerable, it allows the function to pause its execution at a specific point, return a value to the caller, and wait to be resumed.

In the context of an LLM API, the controller method might look conceptually like this (abstracted from actual code):

public async IAsyncEnumerable<string> StreamTokensAsync(string prompt)
{
    // Initialize the LLM session
    var llmSession = _modelService.CreateSession(prompt);

    // Loop until generation is complete
    while (!llmSession.IsComplete)
    {
        // Asynchronously wait for the next token
        string token = await llmSession.GetNextTokenAsync();

        // Yield the token back to the HTTP response stream
        yield return token;

        // The function pauses here until the HTTP infrastructure
        // is ready to accept more data (Backpressure handling).
    }
}

This mechanism is profound because it decouples the generation logic from the transmission logic. The LLM service doesn't need to know it's being served over HTTP. It simply exposes an async stream of strings. This adheres to the Dependency Inversion Principle (a concept from SOLID principles, referenced in earlier architectural chapters), allowing us to swap the underlying LLM implementation (e.g., from OpenAI to a local HuggingFace model) without changing the controller code, as long as both implement the same async streaming interface.

Conclusion

The theoretical foundation of SSE in AI applications rests on the need for latency masking and memory efficiency. By streaming tokens as they are generated, we transform a blocking operation into a fluid, real-time experience. Modern C# features like IAsyncEnumerable provide the syntactic and runtime support necessary to handle these asynchronous streams elegantly, while the native EventSource API in browsers ensures robust, automatic consumption of these events. This combination forms the backbone of modern, responsive AI interfaces.

Basic Code Example

Here is a simple, self-contained "Hello World" example for streaming tokens using Server-Sent Events (SSE) in ASP.NET Core.

Real-World Context: The Conversational AI Chatbot

Imagine you are building a chatbot interface similar to ChatGPT. When a user asks a question, the Large Language Model (LLM) doesn't reply instantly. Instead, it generates text word-by-word (or token-by-token). If you wait for the entire response to finish before sending it to the browser, the user sees a blank screen for several seconds, leading to a poor user experience.

To solve this, we use Server-Sent Events (SSE). This allows the server to push individual tokens to the client as soon as they are generated, creating the illusion of "live" typing.

The Code Example

This example demonstrates an ASP.NET Core Web API endpoint that simulates generating a response and streams it to the client using IAsyncEnumerable and the text/event-stream content type.

using Microsoft.AspNetCore.Mvc;
using System.Text;
using System.Text.Json;

namespace SseStreamingDemo.Controllers;

[ApiController]
[Route("api/[controller]")]
public class ChatController : ControllerBase
{
    /// <summary>
    /// Simulates a streaming LLM response using Server-Sent Events (SSE).
    /// </summary>
    /// <param name="prompt">The user input (not used in this simulation).</param>
    /// <returns>A stream of text/event-stream formatted data.</returns>
    [HttpGet("stream")]
    public async Task StreamChat([FromQuery] string prompt = "Hello")
    {
        // 1. Set the standard SSE content type.
        // This tells the browser to treat the connection as an event stream.
        Response.ContentType = "text/event-stream";

        // 2. Create a simulated sequence of tokens.
        // In a real app, this would come from an LLM library (e.g., Azure OpenAI SDK).
        var tokens = new List<string> { "Hello", " ", "World", "!", " This", " is", " a", " streaming", " demo." };

        // 3. Iterate over the tokens asynchronously.
        foreach (var token in tokens)
        {
            // 4. Format the token according to the SSE specification.
            // Format: "data: {json_payload}\n\n"
            // We wrap the token in JSON to send metadata (e.g., timestamp, role) if needed.
            var jsonPayload = JsonSerializer.Serialize(new { token = token });
            var sseMessage = $"data: {jsonPayload}\n\n";

            // 5. Convert to bytes and write to the response body.
            var bytes = Encoding.UTF8.GetBytes(sseMessage);
            await Response.Body.WriteAsync(bytes, 0, bytes.Length);

            // 6. Flush the stream immediately.
            // Without this, the buffer might hold the data until the connection closes.
            await Response.Body.FlushAsync();

            // 7. Simulate network/processing delay.
            await Task.Delay(100); 
        }
    }
}

Frontend Integration (JavaScript)

To visualize the result, here is how a client consumes this stream using the native EventSource API:

// Connect to the SSE endpoint
const eventSource = new EventSource('/api/chat/stream?prompt=Hello');

// Listen for messages (the default event type)
eventSource.onmessage = (event) => {
    // Parse the JSON data sent by the server
    const data = JSON.parse(event.data);

    // Append the token to the UI
    const outputDiv = document.getElementById('output');
    outputDiv.innerHTML += data.token;

    console.log('Received token:', data.token);
};

// Handle errors
eventSource.onerror = (err) => {
    console.error('EventSource failed:', err);
    eventSource.close();
};

Detailed Line-by-Line Explanation

1. The Controller Setup

[ApiController]
[Route("api/[controller]")]
public class ChatController : ControllerBase

• [ApiController]: Enables automatic model validation and HTTP 400 responses.
• [Route("api/[controller]")]: Defines the URL pattern. For this controller, the base URL is /api/Chat.

2. The Endpoint Signature

[HttpGet("stream")]
public async Task StreamChat([FromQuery] string prompt = "Hello")

• [HttpGet("stream")]: Maps this method to GET /api/Chat/stream.
• async Task: The method is asynchronous. This is crucial for I/O operations like writing to the network stream without blocking the server thread.
• [FromQuery]: Explicitly tells ASP.NET Core to look for the prompt parameter in the URL query string (e.g., ?prompt=Hello).

3. Setting the Content Type

Response.ContentType = "text/event-stream";

• Why this matters: This HTTP header is the contract between the server and the client. It instructs the browser (or any HTTP client) to switch from standard HTTP response processing to SSE mode. The client will now listen for incoming data indefinitely until the connection is closed.

4. Simulating Data Generation

var tokens = new List<string> { "Hello", " ", "World", "!", " This", " is", " a", " streaming", " demo." };

• In a production environment, you would likely use IAsyncEnumerable<T> yielded by an AI SDK. Here, we simulate that behavior with a simple list to keep the example self-contained.

5. The Message Loop (SSE Protocol)

foreach (var token in tokens)
{
    var jsonPayload = JsonSerializer.Serialize(new { token = token });
    var sseMessage = $"data: {jsonPayload}\n\n";
    // ...
}

• SSE Format: The protocol requires strict formatting.
  • data:: The prefix indicating the payload follows.
  • \n\n: Two newlines mark the end of a single event.
• JSON Payload: While SSE can send raw text, wrapping the token in JSON allows you to send structured data (e.g., { "token": "Hello", "id": 1 }) without breaking the protocol.

6. Writing and Flushing

var bytes = Encoding.UTF8.GetBytes(sseMessage);
await Response.Body.WriteAsync(bytes, 0, bytes.Length);
await Response.Body.FlushAsync();

• WriteAsync: Writes the bytes to the underlying TCP stream.
• FlushAsync: This is the most critical step. By default, web servers buffer responses to optimize throughput. If you do not flush, the client might not receive the token until the buffer fills up or the request finishes. Flushing ensures the token is sent immediately over the network.

7. Simulating Latency

await Task.Delay(100);

• Without a delay, the loop would execute so fast that all tokens would be sent in a single TCP packet, effectively "batching" the response. A delay simulates the natural latency of an LLM generating tokens.

Common Pitfalls

1. Forgetting to Flush the Stream

• The Issue: The client receives nothing until the entire request finishes, defeating the purpose of streaming.
• The Fix: Always call await Response.Body.FlushAsync() after writing to the stream inside the loop.

2. Buffering Middleware

• The Issue: Even if you flush your controller code, other middleware components (like logging or compression middleware) might buffer the response.
• The Fix: Ensure you disable buffering if it's enabled elsewhere. You can do this in Program.cs or on the response itself:

  Response.Headers.Add("X-Accel-Buffering", "no"); // Nginx specific hint
  // Or ensure no middleware buffers the request body
  

3. JSON Serialization Overhead

• The Issue: Serializing a large JSON object for every single token adds CPU overhead.
• The Fix: For high-throughput scenarios, consider sending raw strings if you don't need metadata, or use System.Text.Json source generators for faster serialization.

4. Connection Closure

• The Issue: If the client closes the connection (e.g., user navigates away), the server loop continues running, wasting resources.
• The Fix: Check HttpContext.RequestAborted.IsCancellationRequested inside the loop and break if true.

  foreach (var token in tokens)
  {
      if (HttpContext.RequestAborted.IsCancellationRequested)
          break;
      // ... logic
  }
  

Visualizing the Data Flow

The following diagram illustrates the flow of data from the Server (ASP.NET Core) to the Client (Browser) using SSE.

Hold "Ctrl" to enable pan & zoom

This diagram illustrates the unidirectional data flow from the server to the client, where the ASP.NET Core backend pushes updates to the browser via a persistent HTTP connection using Server-Sent Events (SSE).

The chapter continues with advanced code, exercises and solutions with analysis, you can find them on the ebook on Leanpub.com or Amazon

---

Loading knowledge check...

---

---

Code License: All code examples are released under the MIT License. Github repo.

Content Copyright: Copyright © 2026 Edgar Milvus | Privacy & Cookie Policy. All rights reserved.

All textual explanations, original diagrams, and illustrations are the intellectual property of the author. To support the maintenance of this site via AdSense, please read this content exclusively online. Copying, redistribution, or reproduction is strictly prohibited.