Skip to content

Chapter 17: Building a RAG Pipeline with Kernel Memory

Theoretical Foundations

At the heart of modern AI engineering lies a fundamental challenge: Large Language Models (LLMs) are brilliant orators with encyclopedic training, but they are static snapshots of the past. They hallucinate when asked about proprietary data, and they falter when the context of the world changes between their training cutoff and the present moment. Retrieval-Augmented Generation (RAG) is the architectural pattern designed to solve this, effectively giving the LLM a "search engine" for its own brain. However, building a production-grade RAG pipeline is deceptively complex. It is not merely about searching text; it is about the rigorous engineering of ingestion, partitioning, vectorization, and retrieval orchestration.

This is where Microsoft Kernel Memory (KM) enters the stage. It is not just a library; it is a multi-modal AI-native service designed specifically to bridge the gap between raw unstructured data and the semantic reasoning capabilities of an LLM.

Ingestion: Reading files (PDF, Word, PPT, Images). 2. Extraction: Parsing text, handling OCR for images, and reading tables. 3. Partitioning (Chunking): Breaking massive text into smaller, coherent semantic blocks. 4. Vectorization: Converting those text blocks into high-dimensional vectors (embeddings) that represent meaning. 5. Storage: Storing these vectors in a Vector Database (like Azure AI Search, Qdrant, or PostgreSQL with pgvector). 6. Retrieval: Accepting a user query, vectorizing it, and performing a similarity search against the database. 7. Synthesis: Injecting the retrieved context into a prompt and calling the LLM to generate an answer.

Doing this manually requires managing distinct SDKs for each step, handling asynchronous pipelines, and dealing with the fragility of file parsers. Kernel Memory abstracts this entire pipeline into a unified, plugin-based architecture.

The Supermarket Analogy: From Shelf Stocking to Semantic Shopping

Imagine a massive supermarket (your Data Source). The shelves are stocked with thousands of products (Information). You have a customer (the User) who wants to know, "What is the best gluten-free pasta?"

Without Kernel Memory (Manual RAG): The customer walks into the supermarket, but they are blindfolded. They have to feel every item on every shelf to find the pasta. It takes hours, and they often grab the wrong thing (Hallucination).

With Kernel Memory: Ingestion & Parsing: A truck arrives (Data Ingestion). KM automatically unpacks the crates. It doesn't just dump boxes on the floor; it opens them (PDF parsing), reads the labels (OCR), and understands the contents (Data Extraction). 2. Chunking & Indexing: KM realizes that a crate containing "Pasta, Sauce, and Cheese" is too broad. It separates them into specific aisles (Chunking) and assigns them a precise location on the shelf based on their characteristics (Vectorization). 3. Retrieval: When the customer asks for "Gluten-free pasta," KM doesn't just look for the word "pasta." It understands the concept of "pasta" and the constraint of "gluten-free." It instantly teleports the customer to the exact shelf location where that specific item sits.

In this analogy, Kernel Memory is the orchestrator that ensures the data is not just stored, but indexed semantically so that it can be retrieved with high fidelity.

The Architecture of Kernel Memory: The IMemoryStore and ITextEmbeddingGeneration

In the previous book, we discussed the Dependency Injection (DI) pattern in Semantic Kernel. We emphasized that the Kernel is a container for plugins and services. Kernel Memory extends this philosophy by treating memory as a distinct service with specific interfaces.

The core architectural pattern of Kernel Memory relies heavily on the Strategy Pattern, implemented through interfaces. This is critical for building AI applications because it allows you to decouple the logic of retrieval from the implementation of storage.

Consider the ITextEmbeddingGeneration interface. In an AI application, the ability to swap models is paramount. You might start using AzureOpenAIEmbeddings for development due to its reliability, but you might need to swap to OllamaEmbeddings or HuggingFace models for a local deployment or to reduce latency.

// Conceptual representation of the abstraction that enables flexibility
using System.Threading.Tasks;
using System.Collections.Generic;

namespace Microsoft.SemanticKernel.Memory
{
    // This interface is the contract that allows KM to talk to any embedding model
    public interface ITextEmbeddingGeneration
    {
        // Generates a vector (List<float>) representing the semantic meaning of text
        Task<IList<float>> GenerateEmbeddingsAsync(string text, CancellationToken cancellationToken = default);
    }
}

By coding against this interface, your RAG pipeline remains agnostic to the underlying AI model. If you switch from OpenAI to a local model, you only change the implementation injected into the Kernel; the retrieval logic remains untouched.

Similarly, the IMemoryStore interface abstracts the Vector Database.

// Conceptual representation of the storage abstraction
using System.Threading.Tasks;
using System.Collections.Generic;

namespace Microsoft.SemanticKernel.Memory
{
    public interface IMemoryStore
    {
        // Creates a collection (like a table) in the vector DB
        Task CreateCollectionAsync(string collectionName, CancellationToken cancellationToken = default);

        // Upserts a memory record (vector + metadata) into the store
        Task<string> UpsertAsync(string collectionName, MemoryRecord record, CancellationToken cancellationToken = default);

        // The core retrieval mechanism: Nearest Neighbor Search
        Task<MemoryRecord?> GetNearestMatchAsync(string collectionName, IList<float> embedding, double minRelevanceScore = 0.0, CancellationToken cancellationToken = default);
    }
}

This separation is vital. In a production environment, you might need to switch from a local VolatileMemoryStore (for testing) to AzureAISearchMemoryStore (for scale) without rewriting a single line of your retrieval logic.

The Kernel Memory Pipeline: Ingestion as a First-Class Citizen

Extraction: It identifies the file type. If it's a PDF, it uses a text extractor. If it's an image, it triggers OCR. 2. Partitioning (Chunking): This is the most nuanced step. LLMs have token limits. If you feed a 100-page document into an LLM in one go, it will fail or lose context. KM uses sophisticated algorithms to split text. It doesn't just cut by character count; it tries to respect sentence boundaries and semantic cohesion. 3. Embedding: It passes these chunks to the ITextEmbeddingGeneration service to create vectors. 4. Storage: It stores the vector along with the original text and metadata (like the file name, page number, and timestamps).

This pipeline is designed to be Asynchronous and Non-Blocking. In C#, this is achieved using async/await patterns extensively. This ensures that ingesting a massive dataset (like a corporate wiki) does not block the application, allowing for real-time updates to the knowledge base.

The Agentic Integration: Plugins as Retrievers

The true power of combining Semantic Kernel with Kernel Memory is revealed in the concept of Agentic Patterns. In the previous book, we defined a "Plugin" as a set of functions the AI can call. In the context of RAG, Kernel Memory allows us to treat the entire retrieval system as a Plugin.

When we build a Semantic Kernel Agent, we can register the Kernel Memory service as a native skill. This allows the LLM (the Agent) to decide when it needs to retrieve information.

User: "What is the current status of Project Phoenix?" 2. Agent (LLM): Analyzes the prompt. It realizes it lacks specific knowledge about "Project Phoenix." 3. Agent (LLM): Invokes the recall function (provided by KM integration). 4. Kernel Memory: Searches the vector store, finds relevant documents about Project Phoenix, and returns the context. 5. Agent (LLM): Receives the context and synthesizes a final answer.

This is a Multi-Step Agentic Workflow. The Agent isn't just generating text; it is actively querying a knowledge base to ground its generation.

Visualizing the Data Flow

To fully grasp the orchestration, let's visualize the flow of data from ingestion to retrieval.

The diagram illustrates the agent's iterative process of querying a structured knowledge base, retrieving relevant context, and synthesizing that information to generate a grounded response.
Hold "Ctrl" to enable pan & zoom

The diagram illustrates the agent's iterative process of querying a structured knowledge base, retrieving relevant context, and synthesizing that information to generate a grounded response.

Theoretical Foundations

In a theoretical discussion, we must address the edge cases that Kernel Memory handles to ensure robustness.

1. The "Lost in the Middle" Phenomenon: KM Solution: Kernel Memory allows for configurable MinRelevanceScore and ranking. It ensures that only the most relevant snippets are retrieved, preventing context flooding.

2. Multi-Modal Data: KM Solution: Kernel Memory is designed to be multi-modal. It can utilize vision models to describe images during ingestion. The text description of the image is then vectorized and stored. Later, a user can ask a text query like "Find the diagram showing the architecture," and KM will retrieve the image because the vector search matches the semantic description of the diagram.

3. Hybrid Search: KM Solution: Advanced KM configurations support Hybrid Search (Vector + Full Text). It combines the semantic understanding of embeddings with the precision of traditional keyword matching (BM25).

Theoretical Foundations

The theoretical foundation of "Building a RAG Pipeline with Kernel Memory" rests on the principle of Orchestration. It recognizes that RAG is not a single step but a complex lifecycle of data.

By leveraging C# features like Interfaces (IMemoryStore, ITextEmbeddingGeneration) and Async/Await, Kernel Memory provides a resilient, swappable, and scalable framework. It transforms the RAG pipeline from a fragile script into a robust service, allowing the Semantic Kernel Agent to act not just as a text generator, but as an informed entity capable of reasoning over private, dynamic, and unstructured data.

Basic Code Example

Here is a simple, self-contained "Hello World" example demonstrating how to ingest a document and query it using Microsoft Kernel Memory (KM) and the Semantic Kernel.

The Scenario

Imagine you are building an internal support bot for a company. A new employee handbook (a PDF) has just been released. The bot needs to be able to answer questions about this handbook immediately after the document is ingested, without requiring a manual retraining of a machine learning model. We will use Kernel Memory to ingest the text and perform a semantic search.

Code Example

using Microsoft.KernelMemory;
using Microsoft.SemanticKernel;
using System;
using System.IO;
using System.Threading.Tasks;

// The main entry point of our application
class Program
{
    static async Task Main(string[] args)
    {
        Console.WriteLine("=== Kernel Memory RAG Example ===\n");

        // 1. SETUP: Initialize the Kernel Memory builder with default settings.
        // By default, this uses volatile memory (RAM) for both text storage and vector embeddings.
        // This is perfect for testing, but in production, you would swap in Azure AI Search and Azure OpenAI.
        var memory = new KernelMemoryBuilder()
            .WithOpenAIDefaults(Environment.GetEnvironmentVariable("OPENAI_API_KEY"))
            .Build();

        // 2. INGESTION: Prepare a dummy document to simulate an employee handbook.
        // In a real scenario, this would be a PDF, Word doc, or Markdown file.
        string documentPath = "EmployeeHandbook.txt";
        await File.WriteAllTextAsync(documentPath, 
            "Welcome to the company! \n" +
            "Vacation Policy: Employees accrue 15 days per year. \n" +
            "IT Support: Contact helpdesk@company.com for issues. \n" +
            "Dress Code: Business casual is required in the office.");

        Console.WriteLine($"[1] Ingesting document: {documentPath}");

        // 3. PROCESSING: Import the document into Kernel Memory.
        // KM automatically chunks the text, generates embeddings, and stores them.
        // We assign a unique ID ("doc001") to reference it later.
        var ingestionResult = await memory.ImportDocumentAsync(
            new Document("doc001")
                .AddFile(documentPath)
        );

        // Wait for the background processing to complete (simulated here by checking status)
        // In a real async system, you might use events or polling.
        while (!await memory.IsDocumentReadyAsync(documentId: "doc001"))
        {
            await Task.Delay(100); // Wait 100ms
        }
        Console.WriteLine("[2] Ingestion complete. Document is indexed.\n");

        // 4. RETRIEVAL: Query the memory for specific information.
        // We ask a question that requires understanding context (semantic search).
        string question = "How many vacation days do I get?";
        Console.WriteLine($"[3] Querying: \"{question}\"");

        var answer = await memory.AskAsync(question);

        // 5. SYNTHESIS: Display the result.
        // The 'AskAsync' method retrieves relevant chunks and uses the LLM to synthesize an answer.
        Console.WriteLine($"\n[4] Result:\n{answer.Result}");

        // Cleanup
        if (File.Exists(documentPath)) File.Delete(documentPath);
    }
}

Detailed Explanation

Initialization (KernelMemoryBuilder): * We instantiate a KernelMemoryBuilder. This is the standard entry point for configuring KM. * .WithOpenAIDefaults(envVar) configures the pipeline to use OpenAI for both generating text embeddings (vectorization) and processing LLM queries. * Crucial Note: In a production environment, you would explicitly inject services like WithAzureOpenAITextGeneration, WithAzureOpenAITextEmbeddingGeneration, and WithAzureAISearch here to ensure data persistence and scalability.

  1. Document Preparation:

    • We create a local text file (EmployeeHandbook.txt). KM supports various formats (PDF, Word, Images, etc.), but a text file is the easiest to demonstrate without external dependencies.
    • This file acts as our "Knowledge Base."

  2. Ingestion (ImportDocumentAsync):

    • We create a Document object with a unique ID (doc001). This ID is vital for tracking the document lifecycle (e.g., deleting it later).
    • AddFile queues the file for processing.
    • The Pipeline: When ImportDocumentAsync is called, KM triggers a pipeline that:
      1. Extracts text from the file.
      2. Normalizes the text (fixing encoding, removing noise).
      3. Chunking: Splits the text into smaller, manageable segments (e.g., sentences or paragraphs).
      4. Vectorization: Converts each chunk into a numerical vector (embedding) using the configured AI service.
      5. Storage: Saves the text and the vector to the configured memory store.

  3. Polling (IsDocumentReadyAsync):

    • Ingestion is often an asynchronous background process (especially when using cloud queues). We poll IsDocumentReadyAsync to ensure the data is searchable before we ask a question. In a web app, this might be handled by a background worker service.

  4. Querying (AskAsync):

    • memory.AskAsync(question) performs a semantic search.
    • It converts the user's question ("How many vacation days...") into a vector.
    • It searches the vector store for the most similar chunks from the ingested handbook.
    • It passes the relevant chunks to the LLM to generate a natural language answer (RAG).

Visualizing the Kernel Memory Pipeline

The following diagram illustrates the data flow from ingestion to retrieval:

This diagram visualizes the Kernel Memory pipeline, showing how documents are first chunked and stored in a vector database during ingestion, and then how, during retrieval, a user query searches for similar chunks that are passed to an LLM to generate a final answer (RAG).
Hold "Ctrl" to enable pan & zoom

This diagram visualizes the Kernel Memory pipeline, showing how documents are first chunked and stored in a vector database during ingestion, and then how, during retrieval, a user query searches for similar chunks that are passed to an LLM to generate a final answer (RAG).

Missing API Keys: * The Issue: The code relies on Environment.GetEnvironmentVariable("OPENAI_API_KEY"). If this variable is null or empty, the initialization will fail, often with a generic "Unauthorized" or "Configuration error" deep in the pipeline. * The Fix: Always validate critical environment variables at startup. For local development, use a .env file or user secrets, and load them explicitly before building the memory instance.

  1. Assuming Synchronous Execution:

    • The Issue: A common mistake is calling ImportDocumentAsync and immediately querying the memory in the next line. Because KM often processes documents asynchronously (via queues), the vector might not be indexed yet, leading to "No memories found" responses.
    • The Fix: Always implement a check (like IsDocumentReadyAsync) or a retry mechanism. In production, decouple ingestion and querying using message queues (like Azure Service Bus) or event triggers.

  2. Ignoring Token Limits:

    • The Issue: While this example uses a small text file, real-world documents (like 100-page PDFs) can exceed the LLM's context window if not chunked correctly.
    • The Fix: Configure the chunking parameters in the builder. KM defaults are usually sensible, but for highly technical documents, you might need smaller chunks to preserve context, or overlapping chunks to prevent cutting sentences in half.

  3. Hardcoding Document IDs:

    • The Issue: Using a static ID like "doc001" is fine for a demo, but in a multi-user system, this causes collisions.
    • The Fix: Generate unique IDs (e.g., GUIDs) or use tenant-aware IDs (e.g., "user123_handbook_v1"). This is essential for security and data isolation when you implement WithSecurity plugins.

The chapter continues with advanced code, exercises and solutions with analysis, you can find them on the ebook on Leanpub.com or Amazon

Loading knowledge check...

Code License: All code examples are released under the MIT License. Github repo.

Content Copyright: Copyright © 2026 Edgar Milvus | Privacy & Cookie Policy. All rights reserved.

All textual explanations, original diagrams, and illustrations are the intellectual property of the author. To support the maintenance of this site via AdSense, please read this content exclusively online. Copying, redistribution, or reproduction is strictly prohibited.