Reducing LLM RAG Latency from 8s to 900ms on Serverless Edge Runtimes

It is the most common issue in modern web applications using Large Language Models (LLMs): the interface feels slow. You click search, the loading spinner starts spinning, and you wait. Five seconds, eight seconds, sometimes ten seconds. In a web environment where a 100ms delay drops user satisfaction, multi-second loading loops are a critical bottleneck.

When you build a Retrieval-Augmented Generation (RAG) tool—which queries custom database context to inform LLM answers—the pipeline latency is compounded. You aren't just waiting for the model to generate text; you are waiting for embedding computation, vector search database lookups, context injection, and parsing. In this article, I will show you how we optimized our AI workflows to bring RAG latency down to 900ms using serverless edge caching, geographical lookups, and streamed JSON chunk parsing.

Deconstructing the 8-Second Nightmare

To fix the speed issue, we mapped out the latency budget of a typical RAG database search request:

1. Embedding Generation (1.2s): User prompt is sent to an embedding model (like OpenAI text-embedding-3-small) to build a mathematical vector.
2. Vector Database Lookup (1.8s): The vector is sent to pgvector or Pinecone to run a cosine similarity query, locating the most relevant context blocks.
3. Serverless Function Cold Start (1.5s): Node.js serverless functions spin up, resolve imports, and establish database client connections.
4. LLM Inference & Generation (3.5s): The LLM reads the context and prompt, processes tokens, compiles a structured response, and sends it back.

Total time: 8 seconds. To achieve sub-second speeds, we had to optimize or parallelize every single one of these steps.

"To build web software that feels alive, you cannot rely on sequential blocking network queries. You must cache embedding results, co-locate database lookups close to the user, and stream token responses chunk-by-chunk."

1. Co-locating Vector Lookups with Edge Caching

The first optimization targets the embedding and vector lookup stages. If a user asks a question that has been asked recently (or maps closely to common search keywords), generating fresh embeddings is a waste of processing cycles.

By routing incoming lookups through geographical Vercel Edge functions, we check an **in-memory Redis cache** (using Upstash, co-located in the user's nearest server region) before executing vector database queries. We use a lightweight semantic hashing algorithm to check key overlaps:

// Simple lookup cache validation on the Edge
async function checkSemanticCache(hashedQuery) {
    const cachedResponse = await redis.get(`cache:query:${hashedQuery}`);
    if (cachedResponse) {
        return JSON.parse(cachedResponse); // Sub-50ms cache hits!
    }
    return null;
}

If a cache miss occurs, the edge function executes vector similarity queries using an optimized pgvector query structure with custom indexing (`hnsw` index in Postgres), dropping search time from 1.8s down to **70ms**.

2. Bypassing Node.js Cold Starts with Vercel Edge

Establishing database connections inside standard Node.js serverless functions causes severe initialization delays. By moving API routers to the **Vercel Edge Runtime** (built on V8 isolates rather than standard Node VMs), cold start latency drops to **zero**.

To fetch data from PostgreSQL without dragging in heavy ORM layers or blocking connection pools, we utilize HTTP-based database tunnels (like Supabase's PostgREST or pg-connection pools with websockets). This keeps database lookup functions lightweight, containing only the code necessary to perform the raw SQL query.

3. Native Structured Output Streaming (AI Streams)

The biggest latency chunk is the LLM inference phase. Generating a long, structured JSON block can take over 3 seconds. If your backend waits for the model to finish generating the entire response before sending it back, the browser is left blocking.

The solution is **streaming token outputs**. By utilizing Gemini API's streaming response model, we stream raw text chunks immediately as they are generated by the model. Rather than writing complex JSON schemas that the LLM must complete before returning, we stream structured markdown blocks that a client-side parser renders live.

Here is an example setup for a streamed serverless RAG API endpoint:

import { GoogleGenAI } from '@google/generative-ai';

export const config = {
  runtime: 'edge',
};

export default async function handler(req) {
  const { prompt, context } = await req.json();
  const ai = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY });
  const model = ai.getGenerativeModel({ model: 'gemini-1.5-flash' });

  const aiPrompt = `Context:\n${context}\n\nQuestion:\n${prompt}\n\nProvide a precise response.`;

  // Start the stream
  const result = await model.generateContentStream({
    contents: [{ role: 'user', parts: [{ text: aiPrompt }] }],
  });

  // Transform output to web stream readable chunks
  const stream = new ReadableStream({
    async start(controller) {
      for await (const chunk of result.stream) {
        const text = chunk.text();
        controller.enqueue(new TextEncoder().encode(text));
      }
      controller.close();
    },
  });

  return new Response(stream, {
    headers: {
      'Content-Type': 'text/event-stream',
      'Cache-Control': 'no-cache',
      'Connection': 'keep-alive',
    },
  });
}

The Final Speed Sheet

By implementing semantic caching, shifting connection pipelines to the Edge, and streaming token chunks live, our RAG speed sheet changed drastically:

• Embedding validation (Redis match): 40ms
• pgvector similarity search (Cache Miss fallback): 80ms
• Function Cold Start / Connection Pool: 0ms (Edge isolates)
• First Contentful Token (Browser Stream render): 780ms

Total time to first interactive paint: 900ms. The interface responds instantly, and users see text blocks rendering live rather than staring at a loading indicator.

Key Takeaways for AI Architects

If you are building AI search bars, customer support bots, or custom business dashboards, keep these rules in mind:

1. Avoid sequential blocking: Fetch user details, system logs, and embeddings in parallel (`Promise.all`) rather than cascading awaits.
2. Cache semantically: Don't query pgvector for standard questions. Cache vector search hits behind a light hash system.
3. Leverage the Edge: Shift vector index lookup pipelines and token streaming to geography-aware edge servers closest to your user database nodes.