Skip to content

Chapter 20: Capstone - Building a 'ChatGPT-Clone' Backend API

Theoretical Foundations

The theoretical foundation for building a production-ready AI backend API rests on the convergence of asynchronous programming paradigms, dependency injection (DI) architectures, and scalable data persistence strategies. Unlike traditional CRUD applications, AI-driven APIs—specifically those mimicking conversational agents like ChatGPT—introduce unique constraints regarding latency, statefulness, and resource management. We are not merely returning data; we are generating it, often in real-time streams, while maintaining the illusion of a continuous, stateful conversation over a stateless HTTP protocol.

The Statelessness of HTTP vs. The Statefulness of Conversation

At the heart of any chat API is a fundamental contradiction: the web is stateless, but a conversation is inherently stateful. When a user sends a message, the AI model does not inherently "remember" the previous turns. It requires the full context (history) to be provided in the prompt.

Analogy: The Amnesiac Librarian Imagine an amnesiac librarian sitting at a desk (the API endpoint). You walk up and ask a question. The librarian has no memory of your previous visits. To get a coherent answer, you must hand them a notebook containing the entire transcript of your previous conversations every single time you speak. This notebook is the Conversation History.

In our API, we cannot rely on server-side session memory for this "notebook" because:

  1. Scalability: If we scale our API to multiple instances (e.g., behind a load balancer), the user's next request might hit a different server that doesn't have the memory.
  2. Ephemerality: Modern cloud environments kill and restart containers frequently.

Therefore, we must externalize this state. We use a database (like PostgreSQL or Azure SQL) to store the "notebook." However, writing to a database for every single AI response can be slow. To solve this, we implement a Service Layer that abstracts the data access, allowing us to optimize how we read and write these "notebooks."

Asynchronous Streaming: The Illusion of Speed

LLMs (Large Language Models) are autoregressive; they generate text one token (roughly a word part) at a time. If we wait for the model to finish generating the entire response before sending it back to the user, we introduce significant latency. The user sees a loading spinner for 10 seconds.

Analogy: The Water Tap Imagine filling a glass of water. If you wait until the glass is completely full before turning on the tap, you have to wait for the entire volume to accumulate. This is Synchronous processing. Conversely, if you turn on the tap and let the water flow continuously into the glass, the user can start drinking immediately. This is Streaming.

In C#, we utilize IAsyncEnumerable<T> and System.Text.Json to stream JSON objects directly to the client. This requires a deep understanding of pipelining. We don't just open a connection and keep it open; we chunk the data. This is critical in AI because the "thinking time" of the model is the bottleneck, not the network bandwidth.

The Role of Interfaces and Dependency Injection

To build a robust system, we must adhere to the Dependency Inversion Principle (the 'D' in SOLID). We should never depend on a concrete implementation of an AI provider (e.g., OpenAI) directly in our controller.

Why is this crucial for AI?

  1. Vendor Lock-in: OpenAI might change their pricing or API structure.
  2. Hybrid Architectures: You might want to route simple queries to a cheaper, local model (like Llama 2) and complex queries to GPT-4.
  3. Testing: You cannot run integration tests against a real LLM every time; it's too slow and expensive.

Analogy: The Universal Remote Think of your TV (the Controller) and the DVD Player/Streaming Stick (the AI Service). The TV doesn't care if the signal comes from a DVD or a WiFi stream; it just needs a standard HDMI port (the Interface). If the TV was hardwired to a specific DVD player, you couldn't upgrade to a Blu-ray player without buying a new TV.

By defining an interface, we abstract the "brain" of our application.

using System.Collections.Generic;
using System.Threading.Tasks;

namespace ChatCloneApi.Services
{
    // The "Universal Remote" standard
    public interface IChatService
    {
        // Returns a stream of strings (tokens)
        IAsyncEnumerable<string> GenerateStreamedResponseAsync(string prompt, List<ChatMessage> history);
    }
}

Database Optimization for Chat History

Storing chat history is not as simple as a VARCHAR(MAX). As conversations grow, the context window (the amount of text the model can process) fills up. Sending 50,000 words of history for a simple "Hello" is wasteful and expensive.

Analogy: The Cluttered Desk If you keep every piece of paper you've ever written on your desk, finding the relevant document becomes impossible. You need a filing system. However, you also need a "Recency Bias." You usually care about the last 3 documents you touched, but you might need to reference a contract from last year.

In our theoretical architecture, we must implement Pagination and Context Truncation. We store messages in a relational database, but we retrieve them intelligently. We might use a "sliding window" approach: keep the most recent N messages (to maintain immediate context) and summarize or discard older ones.

Security: JWT and Scoped Access

AI APIs are expensive to run. We cannot allow unauthenticated access. We use JSON Web Tokens (JWT) not just for security, but for Tenancy.

Analogy: The Private Booth Imagine a restaurant (the API). The waiter (the Endpoint) needs to know which table (the User) is ordering which meal (the AI Request) so that the bill is charged to the right person. A JWT is like a numbered ticket you get when you sit down. The waiter doesn't need to know your life story; they just need to verify the ticket is valid and which table it belongs to.

In ASP.NET Core, we use Middleware to validate these tokens before the request even reaches our Controller logic. This ensures that User A cannot access User B's conversation history.

Architectural Visualization

The flow of data in our capstone project follows a strict pipeline. We can visualize this dependency graph using Graphviz. Notice how the Controller depends on Abstractions (Interfaces), not Concretions.

The diagram depicts a dependency graph where the Controller points to interfaces (Abstractions) rather than concrete classes, illustrating the Dependency Inversion Principle.
Hold "Ctrl" to enable pan & zoom

The diagram depicts a dependency graph where the Controller points to interfaces (Abstractions) rather than concrete classes, illustrating the Dependency Inversion Principle.

Deep Dive: The Mechanics of IAsyncEnumerable

In the context of AI, standard Task<IActionResult> is insufficient for streaming. If we return a Task<IActionResult>, the framework expects a single object to be serialized and sent at the end of the request.

To achieve the "Water Tap" effect described earlier, we leverage IAsyncEnumerable<T>. This is a C# feature that represents a stream of data that can be enumerated asynchronously.

How it works internally:

  1. The Producer (AI Service): As the LLM generates tokens, the AI Service yields them back to the caller using yield return.
  2. The Consumer (ASP.NET Core): The JSON serializer (System.Text.Json) detects that the return type is IAsyncEnumerable. It switches the HTTP response encoding to Transfer-Encoding: chunked.
  3. The Transport: Instead of sending one large JSON array, it sends a stream of JSON objects (or text) separated by newlines.

Edge Case Handling:

  • Cancellation: If the user closes the browser mid-stream, we must cancel the underlying AI request to save compute costs. This is handled via CancellationToken propagation.
  • Backpressure: If the client reads slower than the server generates, the buffer fills up. We must manage this to prevent OutOfMemory exceptions.

The "Why" of the Capstone Structure

Why build this specific architecture for a ChatGPT clone?

  1. Separation of Concerns: The Controller handles HTTP protocols. The Service handles AI logic. The Repository (implied in the Service) handles Database logic. This allows a frontend developer to work on the Controller without understanding the nuances of LLM tokenization.
  2. Testability: Because we use Interfaces (IChatService), we can unit test the Controller by mocking the service. We can verify that the Controller correctly handles JWT validation and passes the right parameters without actually calling an expensive AI model.
  3. Future-Proofing: By abstracting the AI provider, we can switch from OpenAI to a local open-source model (like Mistral) by simply writing a new class that implements IChatService and registering it in the Dependency Injection container. The rest of the application remains untouched.

Theoretical Foundations

To summarize this subsection, the theoretical foundation of our AI API is built on three pillars:

  1. State Management via Persistence: We treat conversation history as a first-class citizen in the database, using optimized queries to manage context windows.
  2. Asynchronous Streaming: We treat the AI response not as a data retrieval but as a continuous flow, utilizing IAsyncEnumerable to minimize perceived latency.
  3. Architectural Decoupling: We utilize Dependency Injection and Interfaces to isolate the AI provider logic, ensuring the system is modular, testable, and scalable.

This theoretical framework ensures that the application is not just a "proof of concept" but a production-ready system capable of handling real-world traffic and complexity.

Basic Code Example

Let's build a simple, self-contained "Hello World" example for an AI Chat API endpoint using ASP.NET Core. We will simulate an AI response using a simple text generation logic (like a "echo" or "reverse" function) to avoid external dependencies like OpenAI or local models, focusing purely on the API structure, dependency injection, and minimal streaming response.

Real-World Context

Imagine you are building the backend for a new AI assistant application. Before connecting to a massive Large Language Model (LLM) like GPT-4, you need to establish the foundational API structure. This example represents the very first step: creating an endpoint that accepts a user's message and returns a response, simulating the "AI" logic locally. This allows frontend developers to start building the UI immediately while the AI integration happens in parallel.

The Code Example

This code creates a minimal ASP.NET Core Web API with a single endpoint /chat/stream. It uses dependency injection to provide a mock AI service and implements a streaming response to simulate the real-time nature of AI chat.

using Microsoft.AspNetCore.Mvc;
using System.Text;
using System.Threading.Channels;

// 1. Define the request DTO (Data Transfer Object)
public record ChatRequest(string Message);

// 2. Define the response DTO
public record ChatResponse(string Content, bool IsComplete);

// 3. Define the AI Service Interface
public interface IChatService
{
    IAsyncEnumerable<string> GetResponseStreamAsync(string userMessage);
}

// 4. Implement the Mock AI Service
// This simulates a real AI model generating text token-by-token.
public class MockChatService : IChatService
{
    public async IAsyncEnumerable<string> GetResponseStreamAsync(string userMessage)
    {
        // Simulate processing delay (like network latency or model inference time)
        await Task.Delay(200);

        // Simulate a simple "echo" logic for the AI
        string responseText = $"Echoing your message: '{userMessage}'";

        // Break the response into "tokens" (words) to simulate streaming
        string[] tokens = responseText.Split(' ');

        foreach (var token in tokens)
        {
            yield return token + " "; // Yield each token individually

            // Simulate the time it takes to generate the next token
            await Task.Delay(100); 
        }
    }
}

// 5. The API Controller
[ApiController]
[Route("[controller]")]
public class ChatController : ControllerBase
{
    private readonly IChatService _chatService;

    public ChatController(IChatService chatService)
    {
        _chatService = chatService;
    }

    [HttpPost("stream")]
    public async Task StreamChat([FromBody] ChatRequest request)
    {
        // Set headers for streaming
        Response.ContentType = "text/plain";
        Response.Headers.Add("Cache-Control", "no-cache");
        Response.Headers.Add("Connection", "keep-alive");

        // Get the stream from the service
        await foreach (var token in _chatService.GetResponseStreamAsync(request.Message))
        {
            // Write token to the response body
            await Response.WriteAsync(token);

            // Flush the stream immediately to ensure the client receives data in real-time
            await Response.Body.FlushAsync();
        }
    }
}

// 6. Program.cs (Entry Point)
var builder = WebApplication.CreateBuilder(args);

// Register dependencies
builder.Services.AddControllers();
builder.Services.AddSingleton<IChatService, MockChatService>(); // Register the mock service

var app = builder.Build();

// Middleware pipeline
app.UseRouting();
app.MapControllers();

// Run the application
app.Run("http://localhost:5000");

Line-by-Line Explanation

1. Data Transfer Objects (DTOs)

public record ChatRequest(string Message);
public record ChatResponse(string Content, bool IsComplete);
  • public record ChatRequest(string Message);: We define a record to represent the incoming JSON payload. Using record provides immutability, value-based equality, and ToString() overrides automatically, which is standard in modern C# APIs.
  • public record ChatResponse(...): Defines the structure for a response. While our specific streaming example sends raw text strings, in a production app, you might wrap tokens in JSON objects (e.g., {"token": "hello"}).

2. The Service Layer (Abstraction)

public interface IChatService
{
    IAsyncEnumerable<string> GetResponseStreamAsync(string userMessage);
}
  • Dependency Inversion: We define an interface IChatService rather than hard-coding the logic in the controller. This allows us to swap the "Mock" service for a real OpenAI or Azure AI service later without changing the controller code.
  • IAsyncEnumerable<string>: This is a modern C# feature (introduced in C# 8.0) that allows us to return a sequence of values asynchronously. It is perfect for streaming data, as the consumer can iterate over the results as they become available, rather than waiting for the entire list to be generated.

3. The Mock AI Service Implementation

public class MockChatService : IChatService
{
    public async IAsyncEnumerable<string> GetResponseStreamAsync(string userMessage)
    {
        await Task.Delay(200); // Simulate initial processing
        string responseText = $"Echoing your message: '{userMessage}'";
        string[] tokens = responseText.Split(' ');

        foreach (var token in tokens)
        {
            yield return token + " "; 
            await Task.Delay(100); 
        }
    }
}
  • yield return: This keyword is the core of the streaming logic. When the controller requests the next item, the execution pauses at yield return and resumes from there when the next item is requested. This allows us to generate data on the fly without storing the entire response in memory.
  • await Task.Delay: We simulate the "thinking time" of an AI model. In a real scenario, this delay represents the time it takes for the API to receive the next token from the external AI provider (like OpenAI).

4. The API Controller

[ApiController]
[Route("[controller]")]
public class ChatController : ControllerBase
{
    private readonly IChatService _chatService;

    public ChatController(IChatService chatService)
    {
        _chatService = chatService;
    }

    [HttpPost("stream")]
    public async Task StreamChat([FromBody] ChatRequest request)
    {
        Response.ContentType = "text/plain";
        Response.Headers.Add("Cache-Control", "no-cache");
        Response.Headers.Add("Connection", "keep-alive");

        await foreach (var token in _chatService.GetResponseStreamAsync(request.Message))
        {
            await Response.WriteAsync(token);
            await Response.Body.FlushAsync();
        }
    }
}
  • Constructor Injection: The IChatService is injected via the constructor. This is the Inversion of Control (IoC) pattern. The ASP.NET Core DI container provides the instance (in this case, MockChatService).
  • [HttpPost("stream")]: Maps this method to POST /chat/stream.
  • Headers:
    • text/plain: We set the content type to plain text for simplicity. In production, this is often application/x-ndjson (Newline Delimited JSON) or text/event-stream (SSE).
    • no-cache: Tells the client (browser) not to buffer the response, ensuring real-time display.
  • await foreach: We iterate over the IAsyncEnumerable provided by the service. This loop runs as tokens arrive.
  • Response.WriteAsync(token): Writes the token (e.g., "Echoing") directly to the HTTP response stream.
  • Response.Body.FlushAsync(): Crucial Step. By default, ASP.NET Core buffers responses to optimize network usage. However, for chat, we need to send data immediately. Flushing forces the data to be sent to the client right away.

5. Program.cs (Composition Root)

var builder = WebApplication.CreateBuilder(args);
builder.Services.AddControllers();
builder.Services.AddSingleton<IChatService, MockChatService>();
var app = builder.Build();
app.UseRouting();
app.MapControllers();
app.Run("http://localhost:5000");
  • WebApplication.CreateBuilder: Initializes the minimal hosting model (modern .NET).
  • AddSingleton: Registers MockChatService as a singleton. This means one instance of the service handles all requests. This is safe for our mock service because it holds no state. For stateful services, Scoped lifetime would be used.
  • MapControllers: Tells the framework to look for classes inheriting from ControllerBase and map their routes.

Visualization of the Streaming Flow

This diagram illustrates the flow of data from the Client, through the API, to the Service, and back.

Diagram: G
Hold "Ctrl" to enable pan & zoom

Common Pitfalls

  1. Forgetting FlushAsync():

    • The Mistake: Omitting await Response.Body.FlushAsync() inside the streaming loop.
    • The Consequence: The server will buffer the entire response in memory until the loop finishes. The client will see a long delay (waiting for the whole response) and then receive all tokens at once, defeating the purpose of streaming.
    • The Fix: Always call FlushAsync() after writing to the response stream in a streaming endpoint.

  2. Buffering Middleware:

    • The Mistake: Placing middleware (like compression or caching middleware) that buffers the request/response before the controller in the pipeline.
    • The Consequence: The streaming effect is broken because the middleware holds onto the stream.
    • The Fix: Ensure no buffering middleware is active for streaming endpoints, or configure it to disable buffering for specific routes.

  3. Blocking the Thread:

    • The Mistake: Using Thread.Sleep() instead of await Task.Delay() inside the async method.
    • The Consequence: Thread.Sleep() blocks the thread completely. In ASP.NET Core, this ties up a thread from the thread pool, severely limiting the application's scalability (concurrent users).
    • The Fix: Always use asynchronous delays (Task.Delay) and await to free the thread while waiting for I/O or simulated processing.

  4. JSON vs. Plain Text:

    • The Mistake: Sending raw text tokens when the client expects JSON objects.
    • The Consequence: The frontend client (likely parsing JSON) will crash when it tries to parse a string like "Hello " as a JSON object.
    • The Fix: If the client expects JSON, serialize each token into a JSON structure (e.g., JsonSerializer.Serialize(new { token = "Hello" })) and ensure the content type is set appropriately.

The chapter continues with advanced code, exercises and solutions with analysis, you can find them on the ebook on Leanpub.com or Amazon

Loading knowledge check...

Code License: All code examples are released under the MIT License. Github repo.

Content Copyright: Copyright © 2026 Edgar Milvus | Privacy & Cookie Policy. All rights reserved.

All textual explanations, original diagrams, and illustrations are the intellectual property of the author. To support the maintenance of this site via AdSense, please read this content exclusively online. Copying, redistribution, or reproduction is strictly prohibited.