Chapter 16: Voice-to-Voice Realtime WebSockets

Theoretical Foundations

In previous chapters, we explored how to build conversational agents using the Vercel AI SDK, specifically leveraging the useChat hook to manage streaming text responses. That hook abstracts away the complexity of managing WebSocket connections, handling message state, and streaming token-by-token updates from a server. However, that entire paradigm operates on the assumption that the input is text. When we move to a voice-to-voice system, we introduce a fundamental physical constraint: latency.

Latency in voice communication is not merely an inconvenience; it is a barrier to natural interaction. In a human conversation, the average latency between a statement and a response is approximately 200 to 500 milliseconds. If the total round-trip time (speaking → processing → synthesizing → listening) exceeds 1 second, the conversation feels disjointed and robotic. The challenge in building a real-time voice-to-voice system is managing the continuous, high-bandwidth flow of audio data while keeping the processing pipeline tight enough to maintain this illusion of immediacy.

To understand this, we must look at the system as a series of pipelines rather than discrete requests. Unlike the useChat hook, which handles distinct message exchanges, a voice-to-voice system treats the conversation as a continuous stream of audio chunks.

The Audio Pipeline: From Analog to Digital Packets

The journey begins in the browser, where we must capture analog sound waves and convert them into digital data that can be transmitted over a WebSocket. This is the domain of the Web Audio API.

Imagine the microphone as a painter continuously applying strokes to a canvas. If we try to send the entire painting (a complete recording) only after the painter finishes, we introduce massive latency. Instead, we need to capture the painting in small, manageable tiles. In web development terms, we are essentially creating a Node.js stream, but entirely in the browser.

We utilize an AudioContext to create an AudioWorklet. This is a specialized processor that runs on a separate audio thread, ensuring that the heavy lifting of audio processing doesn't block the main UI thread. The AudioWorklet slices the incoming audio stream into small buffers (e.g., 1024 or 2048 samples). These buffers are the "packets" of our real-time system.

Analogy: The Assembly Line Think of the WebSocket connection as a high-speed assembly line. 1. The Worker (AudioWorklet): Captures raw materials (audio samples) and packages them into boxes (buffers). 2. The Conveyor Belt (WebSocket): Moves these boxes to the processing plant (the local backend). 3. The Plant (Ollama/STT): Unpacks the boxes, processes the materials, and sends back a finished product (text). 4. The Return Belt (WebSocket): Sends the product to the packaging department (TTS).

If the conveyor belt is too slow or the boxes are too large, the line jams.

The WebSocket Bridge: Bridging Browser and Local Backend

While HTTP requests are suitable for the useChat hook's initial page loads or distinct queries, they are ill-suited for continuous audio streaming due to the overhead of connection handshakes and headers for every request. WebSockets provide a persistent, full-duplex communication channel between the client and the server.

In the context of local LLMs (running via Ollama), this is critical. We are not sending data to a cloud provider; we are sending it to a local process. The WebSocket acts as the bridge between the browser's sandboxed environment and the local machine's GPU/CPU resources.

The Protocol Design: Unlike a standard text chat where we send JSON objects containing { role: 'user', content: 'Hello' }, our voice stream must define a custom protocol. We need to distinguish between: * Audio Chunks: Raw binary data or Base64-encoded audio buffers. * Metadata: Sample rate, bit depth, and sequence numbers to handle packet loss or out-of-order delivery. * Control Signals: Messages indicating the start of a speech segment (Voice Activity Detection - VAD) or the end of a turn.

Web Development Analogy: The Event Emitter Think of the WebSocket connection as a global Event Emitter. * Client: socket.emit('audio_chunk', buffer) * Server: socket.on('audio_chunk', (buffer) => { processAudio(buffer); })

However, unlike a standard event emitter, we must handle backpressure. If the local STT model is slower than the microphone's input rate, the buffer fills up, leading to memory leaks or delayed processing. We need a flow-control mechanism where the server signals the client to pause transmission if the processing queue exceeds a certain threshold.

Speech-to-Text (STT): The Transcription Engine

Once the audio chunks arrive at the local backend, they must be transcribed into text. This is where the concept of Context Augmentation (introduced in previous chapters regarding RAG) takes on a new dimension. In a text-based RAG system, we retrieve text chunks to augment the query. In a voice system, the "retrieval" is actually the transcription step.

We are converting unstructured audio data into structured text. The local STT model (such as Whisper running via Ollama or a dedicated WASM module) processes these audio buffers.

The "Sliding Window" Analogy: Imagine reading a book through a keyhole. You can only see a few words at a time. To understand the sentence, you must slide the keyhole along the line. Similarly, the STT model doesn't wait for the user to stop speaking entirely. It uses a streaming inference approach. 1. It receives a chunk of audio. 2. It runs inference on that chunk, potentially updating previous transcriptions as context clarifies (e.g., realizing "to" was actually "two" based on the next word). 3. It emits partial transcripts in real-time.

This is where WebAssembly (WASM) becomes a hero. While Ollama runs on the host OS, running a lightweight STT model directly in the browser via WASM can reduce latency further by eliminating the network hop to the local backend entirely. WASM allows us to compile high-performance Rust or C++ audio processing libraries (like onnxruntime) to run at near-native speed inside the browser sandbox.

Text-to-Speech (TTS): The Voice Synthesis

Once the LLM generates a response text (using the same streaming principles as the useChat hook, but fed into a TTS model), we must synthesize audio. This is the inverse of the STT process.

The "Texture Synthesizer" Analogy: Think of TTS not as a voice recorder, but as a synthesizer generating a waveform from a frequency map. The text is the score, and the TTS engine is the orchestra. * Prosody: The system must predict intonation, pitch, and rhythm. * Streaming TTS: Unlike traditional TTS which generates a full file, streaming TTS (like VITS or FastSpeech) generates audio in small frames. As soon as the first few frames are generated, they are sent back to the client over the WebSocket.

Performance Optimization: The WebGPU Factor

The theoretical limit of this system is defined by the speed of inference. On a CPU, processing audio and running LLMs is slow. This is where WebGPU enters the equation.

WebGPU is a modern graphics and compute API for the web. While traditionally used for rendering 3D graphics, it allows general-purpose computation on the GPU (GPGPU). In our voice pipeline, WebGPU can accelerate: 1. Audio Feature Extraction: Converting raw audio waves into spectrograms (Mel-spectrograms) using FFT (Fast Fourier Transform) shaders. 2. Model Inference: Running the STT or TTS models (if converted to a format compatible with WebGPU backends like ONNX) directly on the GPU.

The Analogy: CPU vs. GPU Imagine the CPU as a single, highly skilled chef cooking a complex meal one dish at a time. The GPU is a massive kitchen staff capable of chopping thousands of vegetables simultaneously. * Without WebGPU: The chef (CPU) is overwhelmed by the continuous stream of audio chunks. * With WebGPU: The kitchen staff (GPU) processes the audio features in parallel, drastically reducing the time between receiving an audio chunk and emitting a text token.

Visualizing the Pipeline

The following diagram illustrates the flow of data through the system, highlighting the WebSocket bridge and the parallel processing paths.

Hold "Ctrl" to enable pan & zoom

This diagram illustrates the audio-to-text pipeline, where incoming audio chunks are routed through a WebSocket bridge to a WebGPU-accelerated processing unit that executes parallel computations to rapidly emit corresponding text tokens.

Under the Hood: Buffer Management and Jitter

To make this system robust, we must address two specific technical challenges:

1. Jitter Buffering: Network packets do not arrive at perfectly regular intervals. Some arrive early, some late. If we play audio immediately upon arrival, the output will sound glitchy (jitter). We implement a Jitter Buffer on the client side. This is a FIFO (First-In-First-Out) queue that holds audio chunks for a short duration (e.g., 50-100ms) before playing them. It smooths out the irregular arrival of packets, ensuring a steady stream of audio.

2. Voice Activity Detection (VAD): We cannot send audio data to the server continuously; it wastes bandwidth and processing power. We need to detect when the user is actually speaking. VAD algorithms analyze the audio buffer in the AudioWorklet to detect energy levels and frequency patterns characteristic of human speech. Only when speech is detected do we open the WebSocket stream and begin transmission. This is analogous to a "push-to-talk" button that activates automatically.

Summary

Building a voice-to-voice system requires shifting from a request-response mental model to a continuous stream processing model. It involves: 1. Browser Capture: Using the Web Audio API to slice audio into manageable chunks. 2. Transport: Utilizing WebSockets for low-latency, bidirectional communication. 3. Processing: Leveraging local LLMs and STT/TTS engines, potentially accelerated by WebGPU or compiled via WASM for browser-side execution. 4. Optimization: Implementing jitter buffers and VAD to maintain natural conversation flow.

This architecture transforms the web browser from a document viewer into a real-time communication device, capable of natural voice interaction with local AI models.

Basic Code Example

In this "Hello World" example, we build a minimal web application that captures microphone audio, streams it to a local backend server via WebSockets, simulates a text-to-speech response, and plays it back. This demonstrates the fundamental architecture of real-time voice communication.

The architecture consists of two parts: 1. Client (Browser): Captures audio chunks and plays audio buffers. 2. Server (Node.js): Receives audio chunks, processes them (simulated), and sends audio data back.

We will use the MediaRecorder API for capturing audio and a simple Node.js WebSocket server to handle the stream.

The Architecture Flow

The data flows in a continuous loop. The MediaRecorder produces Blob objects containing raw audio data. These are converted to ArrayBuffer and sent over the WebSocket. The server receives them, simulates an inference delay, and sends a synthetic audio buffer back.

Hold "Ctrl" to enable pan & zoom

This diagram illustrates a WebSocket communication flow where client-side audio data is converted to an `ArrayBuffer`, transmitted to a server for simulated inference, and returned as a synthesized audio buffer.

Implementation

1. Server Code (server.ts)

This Node.js server uses the ws library to handle WebSocket connections. It accepts audio chunks, simulates processing time, and sends back a generated sine wave (representing synthesized speech) as raw audio data.

// server.ts
import { WebSocketServer } from 'ws';
import * as http from 'http';

/**
 * Configuration for the audio stream.
 * 16kHz, 16-bit mono is standard for WebRTC/STT pipelines.
 */
const SAMPLE_RATE = 16000;
const CHANNELS = 1;

const server = http.createServer();
const wss = new WebSocketServer({ server });

console.log('Starting Voice-to-Voice WebSocket Server on port 8080...');

wss.on('connection', (ws) => {
    console.log('Client connected');

    ws.on('message', async (data: Buffer) => {
        // 1. RECEIVE AUDIO
        // In a real app, we would pipe this data to an STT model (e.g., Whisper).
        // Here, we simulate processing latency.
        console.log(`Received audio chunk: ${data.length} bytes`);

        // Simulate network jitter and model inference time (e.g., 150ms)
        await new Promise(resolve => setTimeout(resolve, 150));

        // 2. GENERATE RESPONSE AUDIO
        // Create a synthetic audio buffer (a simple sine wave for demonstration).
        // Duration: 1 second.
        const duration = 1; 
        const numSamples = SAMPLE_RATE * duration;
        const audioBuffer = new Float32Array(numSamples);

        // Generate a 440Hz tone (A4 note)
        const frequency = 440;
        for (let i = 0; i < numSamples; i++) {
            const t = i / SAMPLE_RATE;
            audioBuffer[i] = Math.sin(2 * Math.PI * frequency * t) * 0.5; // 50% volume
        }

        // Convert Float32Array to Buffer (Int16 PCM for standard compatibility)
        const int16Buffer = new Int16Array(audioBuffer.length);
        for (let i = 0; i < audioBuffer.length; i++) {
            int16Buffer[i] = Math.max(-1, Math.min(1, audioBuffer[i])) * 0x7FFF;
        }

        const responseBuffer = Buffer.from(int16Buffer.buffer);

        // 3. SEND AUDIO BACK
        ws.send(responseBuffer);
        console.log(`Sent audio response: ${responseBuffer.length} bytes`);
    });

    ws.on('close', () => {
        console.log('Client disconnected');
    });
});

server.listen(8080);

2. Client Code (client.ts)

This TypeScript code runs in the browser. It initializes the AudioContext, captures microphone input, streams it to the server, and plays the received audio chunks.

// client.ts

/**
 * Main application class handling the voice loop.
 */
class VoiceChatClient {
    private ws: WebSocket | null = null;
    private audioContext: AudioContext | null = null;
    private mediaRecorder: MediaRecorder | null = null;
    private audioQueue: Float32Array[] = [];
    private isPlaying: boolean = false;

    // Audio configuration
    private readonly WS_URL = 'ws://localhost:8080';
    private readonly CHUNK_SIZE_MS = 200; // Send audio every 200ms

    /**
     * Initializes the WebSocket connection and Audio Context.
     */
    public async start() {
        console.log('Initializing Voice Chat Client...');

        // 1. Setup WebSocket
        this.ws = new WebSocket(this.WS_URL);
        this.ws.binaryType = 'arraybuffer'; // Expect binary data

        this.ws.onopen = () => {
            console.log('WebSocket Connected');
            this.startMicrophone();
        };

        this.ws.onmessage = (event) => {
            // 2. Handle Incoming Audio
            this.handleIncomingAudio(event.data);
        };

        this.ws.onerror = (err) => console.error('WebSocket Error:', err);
    }

    /**
     * Captures audio from the user's microphone using MediaRecorder.
     */
    private async startMicrophone() {
        try {
            // Initialize AudioContext (requires user gesture in some browsers)
            this.audioContext = new (window.AudioContext || (window as any).webkitAudioContext)();

            const stream = await navigator.mediaDevices.getUserMedia({ audio: true });

            // 3. Setup MediaRecorder
            // Note: 'audio/webm' or 'audio/ogg' are common browser formats.
            // For raw PCM, we might need to use AudioWorklets, but MediaRecorder is simpler for "Hello World".
            this.mediaRecorder = new MediaRecorder(stream, { mimeType: 'audio/webm' });

            this.mediaRecorder.ondataavailable = (e) => {
                if (e.data.size > 0 && this.ws?.readyState === WebSocket.OPEN) {
                    // Convert Blob to ArrayBuffer to send over WebSocket
                    e.data.arrayBuffer().then((buffer) => {
                        this.ws!.send(buffer);
                    });
                }
            };

            // Trigger recording in chunks
            this.mediaRecorder.start(this.CHUNK_SIZE_MS);
            console.log('Microphone active. Streaming audio...');

        } catch (err) {
            console.error('Error accessing microphone:', err);
        }
    }

    /**
     * Handles the audio data received from the server.
     * @param data The raw ArrayBuffer received via WebSocket.
     */
    private async handleIncomingAudio(data: ArrayBuffer) {
        // 4. Decode Audio Data
        if (!this.audioContext) return;

        // Convert ArrayBuffer to Float32Array for Web Audio API processing
        // Assuming server sends 16-bit PCM (standard for raw audio)
        const int16Data = new Int16Array(data);
        const float32Data = new Float32Array(int16Data.length);

        for (let i = 0; i < int16Data.length; i++) {
            float32Data[i] = int16Data[i] / 32768.0; // Normalize 16-bit to float (-1 to 1)
        }

        // Add to queue to handle playback sequentially
        this.audioQueue.push(float32Data);

        if (!this.isPlaying) {
            this.playAudioQueue();
        }
    }

    /**
     * Plays the audio buffer queue sequentially to avoid glitches.
     */
    private async playAudioQueue() {
        if (!this.audioContext || this.audioQueue.length === 0) {
            this.isPlaying = false;
            return;
        }

        this.isPlaying = true;
        const audioData = this.audioQueue.shift()!;

        // Create an AudioBufferSourceNode to play the raw PCM data
        const source = this.audioContext.createBufferSource();
        const buffer = this.audioContext.createBuffer(
            1, // Mono
            audioData.length,
            this.audioContext.sampleRate
        );

        buffer.getChannelData(0).set(audioData);
        source.buffer = buffer;
        source.connect(this.audioContext.destination);

        source.onended = () => {
            // Recursive call to play next chunk in queue
            this.playAudioQueue();
        };

        source.start();
    }

    /**
     * Stops recording and closes connections.
     */
    public stop() {
        if (this.mediaRecorder) this.mediaRecorder.stop();
        if (this.ws) this.ws.close();
        if (this.audioContext) this.audioContext.close();
        console.log('Voice Chat Client stopped.');
    }
}

// Usage
// In a real app, you would attach this to a button click event.
const client = new VoiceChatClient();
// client.start(); // Uncomment to run

Detailed Line-by-Line Explanation

Server Code Analysis

1. Import & Setup: We import WebSocketServer from the ws library and create a standard HTTP server. The WebSocket server attaches to this HTTP server.
2. Audio Configuration: We define SAMPLE_RATE (16kHz) and CHANNELS. This is critical because the client and server must agree on the audio format to avoid distortion.
3. Connection Handling: wss.on('connection', ...) listens for new clients.
4. Receiving Data: ws.on('message', ...) triggers when binary data arrives.
   • data.length: We log the size to verify data is being transmitted.
   • Simulation: setTimeout simulates the latency of running a local LLM or STT model (like Ollama). Without this, the loop would be too fast to perceive in a "Hello World" demo.
5. Generating Audio:
   • We create a Float32Array representing raw PCM audio.
   • The loop calculates a sine wave (Math.sin) at 440Hz. This is a pure tone representing a synthesized voice response.
   • Conversion: Web Audio API typically uses Float32 (-1 to 1), but raw network transmission often uses Int16 (-32768 to 32767) for bandwidth efficiency. We convert the float values to Int16.
6. Sending Data: ws.send(responseBuffer) pushes the binary audio back to the client immediately.

Client Code Analysis

1. Class Structure: Encapsulating logic in VoiceChatClient keeps the global scope clean and manages state (like audioQueue and isPlaying).
2. WebSocket Setup:
   • binaryType = 'arraybuffer': This tells the browser to treat incoming messages as raw binary buffers, not text strings.
   • ws.onopen: We only start the microphone after the connection is established to ensure we don't capture audio with nowhere to send it.
3. Microphone Capture (startMicrophone):
   • AudioContext: The core of Web Audio. It handles creating nodes and processing audio.
   • MediaRecorder API: This is the simplest way to get audio chunks in the browser.
   • mimeType: We use 'audio/webm'. Browsers compress this automatically. Note: In a production STT pipeline, you might need to decode this webm data into raw PCM before sending, or use a library that handles it.
   • ondataavailable: Fires every CHUNK_SIZE_MS (200ms). We convert the Blob to an ArrayBuffer and send it via WebSocket.
4. Handling Incoming Audio (handleIncomingAudio):
   • Decoding: The server sends Int16 PCM. The Web Audio API works best with Float32. We normalize the Int16 values to the -1.0 to 1.0 range.
   • Queueing: Audio playback must be continuous. If we just played chunks as they arrived, network jitter would cause stuttering. We push chunks into audioQueue.
5. Playback (playAudioQueue):
   • AudioBuffer: We create a buffer specifically sized for the chunk we received.
   • AudioBufferSourceNode: This is a "one-shot" node. Once it plays the buffer, it is discarded. We create a new one for every chunk.
   • Recursion: source.onended triggers the function to play the next item in the queue, creating a seamless chain.

Common Pitfalls

1. CORS and Secure Contexts (HTTPS):

   • Issue: WebSockets and MediaDevices (Microphone) often fail in production if not served over HTTPS or localhost.
   • Fix: Ensure your server is accessible via wss:// (secure WebSocket) in production. Browsers will block microphone access on plain HTTP sites (except localhost).

2. Audio Sample Rate Mismatch:

   • Issue: If the browser captures audio at 48kHz and the server processes it as 16kHz (or vice versa), the playback speed will sound like chipmunks (too fast) or a slow-motion monster (too slow).
   • Fix: Explicitly resample audio on the server or configure the AudioContext on the client to match the expected input rate if possible.

3. MediaRecorder Blob vs. Raw PCM:

   • Issue: MediaRecorder outputs container formats (WebM/Opus), not raw PCM. Sending this directly to a raw PCM parser (like many simple STT engines) will cause errors.
   • Fix: For a "Hello World," we assume the backend can handle the container. For production, you must decode the WebM container into raw PCM (using libraries like ogg-opus-decoder) before sending to an STT model.

4. Memory Leaks in Audio Playback:

   • Issue: Creating AudioBufferSourceNode repeatedly without cleanup can cause memory spikes, though these nodes are garbage collected once they finish playing.
   • Fix: The code provided uses a queue system. Ensure you clear the queue (audioQueue = []) when the user stops the connection to prevent pending audio from playing after a disconnect.

5. Async/Await Race Conditions:

   • Issue: In the client, MediaRecorder might start before the WebSocket connection is fully open.
   • Fix: The code explicitly waits for ws.onopen before calling startMicrophone. Never assume network resources are immediately available.

6. Vercel/Serverless Timeouts:

   • Issue: If you deploy the backend to Vercel or AWS Lambda, standard WebSocket servers might timeout or be killed if the function execution duration is limited.
   • Fix: WebSockets require persistent connections. Use services specifically designed for long-lived connections (like AWS API Gateway WebSockets, Azure Web PubSub, or a dedicated VPS/EC2 instance running Node.js). Do not use standard Serverless HTTP functions for the WebSocket receiver.

The chapter continues with advanced code, exercises and solutions with analysis, you can find them on the ebook on Leanpub.com or Amazon

---

Loading knowledge check...

---

---

Code License: All code examples are released under the MIT License. Github repo.

Content Copyright: Copyright © 2026 Edgar Milvus | Privacy & Cookie Policy. All rights reserved.

All textual explanations, original diagrams, and illustrations are the intellectual property of the author. To support the maintenance of this site via AdSense, please read this content exclusively online. Copying, redistribution, or reproduction is strictly prohibited.