Skip to content

Chapter 6: From Lists to Streams - Introduction to IAsyncEnumerable

Theoretical Foundations

The fundamental challenge in building responsive, scalable AI applications is handling data that arrives over time, rather than all at once. Traditional data structures, like List<T>, are eager; they require the entire dataset to be materialized in memory before processing can begin. This model breaks down when dealing with asynchronous operations, particularly when consuming Large Language Model (LLM) responses, where tokens are generated and streamed sequentially.

To solve this, C# introduced the IAsyncEnumerable<T> interface. This is not merely a collection; it is a contract for a pull-based asynchronous stream. It allows a consumer to request the next item in a sequence, awaiting its arrival, without blocking the thread. This is the cornerstone of modern, non-blocking data pipelines.

The Problem with Eager Loading in AI Pipelines

In previous chapters, we discussed the Task<T> pattern for handling single asynchronous operations. However, AI pipelines often involve continuous data flows. Consider a scenario where you are generating a response from a local Llama model via an API. The model does not return a complete string instantly; it streams tokens one by one.

If we were to use a Task<List<string>> approach, the application would:

  1. Send the request.
  2. Wait for the entire response to finish generating.
  3. Receive a massive list of tokens.
  4. Begin processing.

This introduces latency and memory pressure. The user sees nothing until the generation is complete. In a real-time chat interface, this is unacceptable. We need to process data as it arrives, rendering tokens to the UI immediately.

The Analogy: The Book vs. The Magazine Subscription

To understand the shift from List<T> to IAsyncEnumerable<T>, consider the difference between buying a book and subscribing to a magazine series.

The List<T> Approach (The Book): You go to a store and buy a complete book. You cannot read page 1 until the entire book has been printed, bound, and shipped to the store. If the book is 1,000 pages long, you must wait for the entire production process to finish. This is synchronous and eager. You hold the entire object in your hands before you can consume any part of it.

The IAsyncEnumerable<T> Approach (The Magazine Subscription): You subscribe to a monthly magazine series. You receive the first issue immediately. You can read it while the authors are still writing the second issue. When you finish the first issue, you "pull" the next one (or it is delivered). The production of the next issue happens asynchronously relative to your reading. You do not need to wait for the entire series to be written to enjoy the first installment. This is asynchronous and lazy.

In AI, we are the magazine subscriber. We want to read the "text" (process the tokens) as they are "published" by the model, without waiting for the entire "book" to be finished.

Theoretical Foundations

IAsyncEnumerable<T> implements a pull-based model. The consumer drives the flow of data.

  1. The Producer (Iterator): The method that generates data (e.g., an AI model inference loop). It uses yield return to emit a value and then suspends execution, waiting for the consumer to request the next value.
  2. The Consumer (Caller): The code iterating over the stream (usually via await foreach). It requests the next item, awaiting the ValueTask<T> returned by the iterator.

This contrasts with push-based models (like Reactive Extensions IObservable<T>), where the producer pushes data to the consumer regardless of whether the consumer is ready. In high-throughput AI scenarios, the pull model is often preferred because it naturally implements backpressure—if the consumer is slow (e.g., rendering to a UI), the producer pauses generation, preventing memory overflow.

Deep Dive: The Interface and State Machine

The IAsyncEnumerable<T> interface is deceptively simple. It is defined in the System.Collections.Generic namespace:

public interface IAsyncEnumerable<out T>
{
    IAsyncEnumerator<T> GetAsyncEnumerator(CancellationToken cancellationToken = default);
}

It returns an IAsyncEnumerator<T>, which itself looks like this:

public interface IAsyncEnumerator<out T> : IAsyncDisposable
{
    T Current { get; }
    ValueTask<bool> MoveNextAsync();
}

The Mechanics: When you write an async iterator method using yield return, the C# compiler transforms your code into a state machine (similar to how it handles async/await).

  • State 0: Initial state.
  • State 1: After the first yield return.
  • State N: After the Nth yield return.

When MoveNextAsync() is called:

  1. The state machine resumes execution from where it suspended.
  2. It runs until the next yield return.
  3. It updates Current with the yielded value.
  4. It returns true to indicate data is available.
  5. If the method completes, it returns false.

Crucially, because the method is async, the state machine can handle await keywords inside the loop. This allows the iterator to await external asynchronous events (like an HTTP response stream) between yielding items.

Integration with AI Pipelines: Handling Infinite Streams

In the context of AI, IAsyncEnumerable<T> is the standard for handling streaming endpoints. Most modern LLM APIs (OpenAI, Azure OpenAI, Anthropic) support Server-Sent Events (SSE). SSE is a protocol where the server sends data as a series of text lines separated by double newlines.

An IAsyncEnumerable<string> wrapper around an HttpClient response stream allows us to parse these events incrementally.

Why this matters for Model Swapping: As discussed in Book 2 regarding Abstraction Layers, we often build a IChatModel interface to swap between providers (e.g., OpenAI vs. Local Llama). The return type of a streaming method in this interface is critical.

public interface IChatModel
{
    // Returning a List<string> forces buffering and breaks streaming UI.
    // IAsyncEnumerable<string> allows real-time updates regardless of the backend.
    IAsyncEnumerable<string> StreamCompletionAsync(string prompt);
}

Whether the backend is a cloud API or a local quantized model, the consumer (UI layer) iterates the same way. The IAsyncEnumerable abstracts away the latency and buffering differences between the providers.

Managing Backpressure and Cancellation

Two critical edge cases in streaming are backpressure and cancellation.

Backpressure: Imagine a scenario where the AI model generates text at 1000 tokens/second, but the UI can only render at 30 frames/second. Without backpressure, the application buffer would grow indefinitely, leading to an OutOfMemoryException. With IAsyncEnumerable, the loop pauses at await foreach. The iterator does not produce the next token until the loop body (the UI render) completes. This synchronization point naturally throttles the producer.

Cancellation: AI generation can be expensive and time-consuming. A user might click "Stop" mid-stream. IAsyncEnumerable supports CancellationToken natively. The iterator checks the token at every suspension point (every yield return or await). If cancellation is requested, the state machine transitions to a disposal state, releasing resources (like the HTTP connection or GPU context) immediately.

Visualizing the Data Flow

The following diagram illustrates the lifecycle of an asynchronous stream in an AI application.

The diagram illustrates the lifecycle of an asynchronous stream in an AI application, depicting how the state machine transitions to a disposal state to immediately release resources like HTTP connections or GPU contexts when cancellation is requested.
Hold "Ctrl" to enable pan & zoom

The diagram illustrates the lifecycle of an asynchronous stream in an AI application, depicting how the state machine transitions to a disposal state to immediately release resources like HTTP connections or GPU contexts when cancellation is requested.

Comparison: IEnumerable<T> vs IAsyncEnumerable<T>

It is vital to distinguish between synchronous and asynchronous enumeration.

  • IEnumerable<T> / foreach: Used for in-memory collections. Iterating foreach (var item in list) is synchronous. If the data source requires I/O (like reading a file line by line), IEnumerable forces blocking calls or complex workarounds.
  • IAsyncEnumerable<T> / await foreach: Designed for I/O-bound sequences. The loop yields control to the event loop while waiting for data, allowing other tasks to run.

Architectural Implication: In a server-side ASP.NET Core application handling multiple concurrent AI requests, using IEnumerable for I/O operations would block threads from the thread pool. This reduces scalability (increasing thread count). Using IAsyncEnumerable ensures that threads are free to handle other requests while waiting for AI responses, maximizing throughput.

The Role of yield return

The yield return keyword is the syntactic sugar that enables this lazy evaluation. It instructs the compiler to generate a state machine that preserves the local variables and execution position.

Without yield return, implementing IAsyncEnumerable manually requires creating a class that implements the interface explicitly, managing state integers, and handling MoveNextAsync logic manually. This is error-prone and verbose. yield return encapsulates this complexity, allowing developers to write linear-looking code that executes asynchronously and lazily.

Theoretical Foundations

IAsyncEnumerable<T> is the bridge between the synchronous world of collections and the asynchronous world of I/O. For AI pipelines, it is non-negotiable. It enables:

  1. Low Latency: Processing begins before the entire payload is received.
  2. Memory Efficiency: Only one item (or a small buffer) is held in memory at a time.
  3. Scalability: Non-blocking iteration allows the application to handle thousands of concurrent streams.
  4. Abstraction: It provides a uniform interface for streaming data, whether from a remote API or a local computation.

By mastering this pattern, we move from building applications that "wait for data" to applications that "react to data," which is the essence of modern real-time AI systems.

Basic Code Example

Here is a basic "Hello World" example demonstrating IAsyncEnumerable<T> to simulate streaming a response from an AI model.

Real-World Context

Imagine you are building a chat application that interfaces with a Large Language Model (LLM). When a user asks a question, the LLM does not return the entire answer instantly. Instead, it generates the text token by token (word by word). If you used a standard List<string> or string, your application would wait for the entire response to download before showing anything to the user. This results in a "loading" spinner and a poor user experience.

By using IAsyncEnumerable<T>, we can process the data as it arrives. We "yield" each token immediately as it is generated, allowing the UI to update in real-time, mimicking the typing effect of a human or the streaming nature of an AI.

Code Example

using System;
using System.Collections.Generic;
using System.Threading;
using System.Threading.Tasks;

public class Program
{
    public static async Task Main(string[] args)
    {
        Console.WriteLine("User: What is the capital of France?");
        Console.Write("AI: ");

        // 1. Consume the async stream
        // The 'await foreach' loop retrieves items one by one as they become available.
        await foreach (var token in GetStreamingResponseAsync("France"))
        {
            Console.Write(token);
            // Simulate human-like typing speed
            await Task.Delay(100); 
        }

        Console.WriteLine("\n\n[End of Stream]");
    }

    /// <summary>
    /// Simulates an AI model generating a response token by token.
    /// </summary>
    /// <param name="topic">The topic to generate text about.</param>
    /// <returns>An asynchronous stream of strings (tokens).</returns>
    public static async IAsyncEnumerable<string> GetStreamingResponseAsync(string topic)
    {
        // Simulated response data
        string[] tokens = { "The", " capital", " of", " ", topic, " is", " Paris." };

        foreach (string token in tokens)
        {
            // 2. Yield the current token immediately
            // This passes the data to the caller without blocking the loop.
            yield return token;

            // 3. Simulate asynchronous work (e.g., network latency or LLM inference time)
            // In a real scenario, this delay represents waiting for the next token 
            // from the API.
            await Task.Delay(200); 
        }
    }
}

Line-by-Line Explanation

  1. using System.Collections.Generic;

    • This namespace contains the definition for IAsyncEnumerable<T>, which is the core interface required for creating asynchronous streams.

  2. public static async Task Main(string[] args)

    • The entry point of the application. It is marked async to allow the use of await inside the method, which is necessary for consuming asynchronous streams.

  3. await foreach (var token in GetStreamingResponseAsync("France"))

    • This is the consumption side of the stream.
    • await foreach: Introduced in C# 8.0, this construct iterates over an IAsyncEnumerable<T>. Unlike a standard foreach, it awaits the retrieval of each item. It suspends execution until the next item is available, but it does not block the thread indefinitely; it yields control back to the system.
    • GetStreamingResponseAsync("France"): Calls the generator method. Note that the method returns immediately with an IAsyncEnumerable<string> object, not a fully populated collection.

  4. Console.Write(token);

    • Prints the received token immediately. Because this happens inside the loop, the user sees the text appear incrementally (e.g., "The" appears, then " capital", etc.).

  5. await Task.Delay(100);

    • Simulates the UI rendering time or user perception delay. In a real WPF or Blazor app, this might be unnecessary as the UI update itself takes time, but here it ensures the console output is readable.

  6. public static async IAsyncEnumerable<string> GetStreamingResponseAsync(string topic)

    • This is the generator method signature.
    • async IAsyncEnumerable<string>: The return type indicates that this method will produce a sequence of strings asynchronously.
    • [EnumeratorCancellation]: (Implicitly handled by the compiler, though you can add the attribute explicitly if you need to pass a CancellationToken).

  7. foreach (string token in tokens)

    • A standard synchronous loop iterating over a local array of strings representing the "AI response."

  8. yield return token;

    • The Magic Keyword: This is the most critical line.
    • When the compiler sees yield return, it transforms the method into a state machine.
    • Execution pauses here, the current token is handed off to the await foreach loop in Main, and the method "suspends" its state (local variables like token and the loop index are preserved).
    • It resumes execution from this exact spot only when the consumer requests the next item.

  9. await Task.Delay(200);

    • Simulates the latency of a real API call. Because this is an async method, we can await here without freezing the application. In a real-world scenario, this represents the time it takes for the AI model to compute the next token.

Visualizing the Flow

The following diagram illustrates the "Ping-Pong" nature of IAsyncEnumerable. The Consumer requests data, the Producer generates one item, yields it, and suspends until requested again.

A diagram illustrating the Ping-Pong flow of IAsyncEnumerable, where the Consumer requests data, the Producer generates and yields one item, and then suspends until the next request is made.
Hold "Ctrl" to enable pan & zoom

A diagram illustrating the Ping-Pong flow of `IAsyncEnumerable`, where the Consumer requests data, the Producer generates and yields one item, and then suspends until the next request is made.

Common Pitfalls

1. Forgetting the await in await foreach A common mistake is trying to use a standard foreach loop on an IAsyncEnumerable<T>: 

// ❌ WRONG: This will cause a compile error.
foreach (var item in GetStreamingResponseAsync("topic")) 
{
    ...
}
You must use await foreach because the source of the data is asynchronous. The compiler cannot block synchronously to wait for the next item without causing deadlocks or performance issues.

2. Blocking the Producer Thread Inside the IAsyncEnumerable method (the producer), avoid long-running synchronous operations. 

// ❌ BAD PRACTICE:
public async IAsyncEnumerable<string> BadGenerator()
{
    // This blocks the thread doing nothing.
    // In a high-traffic server scenario, this wastes valuable thread pool threads.
    Thread.Sleep(1000); 
    yield return "Hello";
}

// ✅ GOOD PRACTICE:
public async IAsyncEnumerable<string> GoodGenerator()
{
    // This frees up the thread to do other work while waiting.
    await Task.Delay(1000); 
    yield return "Hello";
}

3. Treating it like a List IAsyncEnumerable<T> does not support indexing (e.g., stream[0]) or Count. It is a forward-only cursor. If you need to access items randomly or know the total length beforehand, you must buffer the stream into a List<T> or Array, but doing so defeats the purpose of streaming for memory efficiency.

4. Not Handling Cancellation In a real web server, a user might close their browser mid-stream. If your producer is calculating heavy tokens, you should respect cancellation. 

// Adding cancellation support
public async IAsyncEnumerable<string> GetStream(
    [EnumeratorCancellation] CancellationToken ct = default)
{
    while (!ct.IsCancellationRequested)
    {
        // Check cancellation before heavy work
        ct.ThrowIfCancellationRequested();

        await Task.Delay(100, ct);
        yield return "token";
    }
}

The chapter continues with advanced code, exercises and solutions with analysis, you can find them on the ebook on Leanpub.com or Amazon

Loading knowledge check...

Code License: All code examples are released under the MIT License. Github repo.

Content Copyright: Copyright © 2026 Edgar Milvus | Privacy & Cookie Policy. All rights reserved.

All textual explanations, original diagrams, and illustrations are the intellectual property of the author. To support the maintenance of this site via AdSense, please read this content exclusively online. Copying, redistribution, or reproduction is strictly prohibited.