Engineering Blog

How We Build Production RAG Systems — From Prototype to Enterprise Scale

Published August 19, 2026 · Ondevtra Engineering · 12 min read

Every team that builds a RAG prototype gets the same first impression: this is straightforward. You embed some documents, store them in a vector database, retrieve the top-k results, and feed them into an LLM. The demo works. Stakeholders are impressed. Then you try to ship it to production and discover the gap between a working notebook and a reliable system is enormous.

We have deployed RAG systems for enterprise clients across legal document search, internal knowledge bases, customer support automation, and compliance workflows. This post captures the architecture patterns and engineering decisions that separate production RAG from prototype RAG — the lessons we learned by breaking things in real deployments.

Why Naive RAG Fails in Production

The basic RAG pattern — embed query, retrieve top-k chunks, generate answer — has failure modes that only reveal themselves at scale with real users:

These are not edge cases. They are the default experience when you move from a curated demo dataset to production corpora with inconsistent formatting, domain-specific terminology, and ambiguous user queries.

Our Architecture: The Production RAG Stack

After iterating across multiple deployments, we converged on an architecture that handles real-world complexity. Here is the high-level stack:

User Query
    |
    v
[FastAPI Gateway] --- auth, rate limiting, query preprocessing
    |
    v
[Hybrid Retrieval]
    |--- Dense: pgvector (OpenAI text-embedding-3-large)
    |--- Sparse: BM25 (Elasticsearch / custom index)
    |
    v
[Reciprocal Rank Fusion] --- merge dense + sparse results
    |
    v
[Cross-Encoder Reranker] --- ms-marco-MiniLM-L-12-v2
    |
    v
[Context Assembly] --- dedup, ordering, citation mapping
    |
    v
[LLM Generation] --- GPT-4o with structured output
    |
    v
[Response + Citations] --- traceable back to source chunks

Document Ingestion and Chunking

Chunking strategy is the single most impactful decision in a RAG system, yet it gets the least attention during prototyping. We have moved away from naive fixed-size chunking entirely.

Our approach uses hierarchical chunking with overlap:

from langchain.text_splitter import RecursiveCharacterTextSplitter

# Parent chunks for context preservation
parent_splitter = RecursiveCharacterTextSplitter(
    chunk_size=2000,
    chunk_overlap=200,
    separators=["\n\n", "\n", ". ", " "]
)

# Child chunks for precise retrieval
child_splitter = RecursiveCharacterTextSplitter(
    chunk_size=400,
    chunk_overlap=50,
    separators=["\n\n", "\n", ". ", " "]
)

# We retrieve on child chunks but expand to parent
# chunks before sending to the LLM
def retrieve_with_parent_context(query: str, top_k: int = 5):
    child_results = vector_store.similarity_search(query, k=top_k * 3)
    parent_ids = set(r.metadata["parent_id"] for r in child_results[:top_k])
    return [parent_store[pid] for pid in parent_ids]

The principle: retrieve on small, precise chunks (400 tokens) but expand to larger parent chunks (2000 tokens) before feeding into the LLM. This gives you retrieval precision without sacrificing the context the model needs to generate a coherent answer.

For structured documents (contracts, technical manuals, compliance docs), we also apply document-aware chunking that respects section boundaries, headers, and table structures. A chunk should never split a table row or break mid-paragraph unless the paragraph is unusually long.

Hybrid Retrieval: Dense + Sparse

Purely semantic search (dense vectors) fails on exact-match queries. A user searching for "SOC-2 Type II" needs lexical matching, not semantic similarity. Purely keyword search (BM25) fails on paraphrased or conceptual queries. You need both.

from fastapi import FastAPI, Depends
from pydantic import BaseModel
import numpy as np

app = FastAPI()

class RetrievalRequest(BaseModel):
    query: str
    top_k: int = 10
    alpha: float = 0.6  # weight for dense vs sparse

@app.post("/retrieve")
async def hybrid_retrieve(req: RetrievalRequest):
    # Dense retrieval via pgvector
    query_embedding = await embed_query(req.query)
    dense_results = await pgvector_search(
        query_embedding, k=req.top_k * 2
    )

    # Sparse retrieval via BM25
    sparse_results = await bm25_search(
        req.query, k=req.top_k * 2
    )

    # Reciprocal Rank Fusion
    fused = reciprocal_rank_fusion(
        [dense_results, sparse_results],
        weights=[req.alpha, 1 - req.alpha],
        k=60
    )

    return fused[:req.top_k]


def reciprocal_rank_fusion(result_lists, weights, k=60):
    """Merge multiple ranked lists using RRF."""
    scores = {}
    for results, weight in zip(result_lists, weights):
        for rank, doc in enumerate(results):
            doc_id = doc.metadata["id"]
            if doc_id not in scores:
                scores[doc_id] = {"doc": doc, "score": 0.0}
            scores[doc_id]["score"] += weight / (k + rank + 1)

    ranked = sorted(scores.values(), key=lambda x: x["score"], reverse=True)
    return [item["doc"] for item in ranked]

The alpha parameter controls the balance between dense and sparse. We default to 0.6 (favoring dense) but expose it as a tunable parameter per deployment. Some corpora — particularly those with heavy acronyms or product codes — need alpha closer to 0.4.

Reranking: The Critical Middle Layer

Initial retrieval is recall-optimized. You cast a wide net. The reranker is precision-optimized — it takes the top 20-30 candidates and re-scores them using a cross-encoder that sees query and document together.

This is the single highest-ROI addition to any RAG pipeline. In our measurements, adding a reranker consistently improves answer relevance by 15-25% (measured by RAGAS faithfulness) with only 200-400ms of added latency. For most enterprise use cases, that tradeoff is trivially worth it.

We use the cross-encoder/ms-marco-MiniLM-L-12-v2 model for reranking. It is small enough to run on CPU with acceptable latency, and the quality improvement over no-reranking is dramatic. For latency-sensitive deployments, we run it on a dedicated GPU instance and batch requests.

Multi-Scenario Retrieval Testing

In VLSI physical design, engineers run MCMM (Multi-Corner Multi-Mode) analysis to verify that a chip works across all operating conditions — fast/slow corners, different voltages, different temperature extremes. The equivalent in RAG is testing your retrieval across the full distribution of query types your system will encounter.

We define four retrieval scenarios that every deployment must pass:

We build golden datasets for each scenario — typically 50-100 query-answer pairs per category — and run automated evaluation on every pipeline change. If retrieval recall drops on any scenario after a configuration change, the change does not ship.

# retrieval_eval.py — scenario-based retrieval testing
import json
from ragas import evaluate
from ragas.metrics import context_recall, context_precision
from datasets import Dataset

def load_golden_dataset(scenario: str) -> Dataset:
    with open(f"eval/golden_{scenario}.json") as f:
        data = json.load(f)
    return Dataset.from_dict({
        "question": [d["query"] for d in data],
        "ground_truth": [d["expected_answer"] for d in data],
        "contexts": [retrieve(d["query"]) for d in data],
        "answer": [generate(d["query"]) for d in data],
    })

scenarios = ["short_specific", "long_contextual", "ambiguous", "out_of_domain"]

for scenario in scenarios:
    ds = load_golden_dataset(scenario)
    result = evaluate(ds, metrics=[context_recall, context_precision])
    print(f"{scenario}: recall={result['context_recall']:.3f}, "
          f"precision={result['context_precision']:.3f}")

    # Gate: fail if recall drops below threshold
    assert result["context_recall"] >= 0.85, (
        f"Retrieval recall for {scenario} below threshold"
    )

Evaluation: Measuring What Matters

You cannot improve what you cannot measure. We use RAGAS as the backbone of our evaluation framework, supplemented by human feedback loops for cases where automated metrics diverge from perceived quality.

The metrics we track in production:

We run RAGAS evaluation nightly against a held-out test set and alert when any metric drops more than 5% from the rolling baseline. This catches regressions from new document ingestion, embedding model changes, or infrastructure issues before they affect users.

Human Feedback Loops

Automated metrics get you 80% of the way. The remaining 20% requires human signal. We instrument every RAG response with thumbs up/down feedback and a free-text "what went wrong" field. This feeds into a weekly review cycle where we:

LangSmith provides the tracing infrastructure for this. Every query flows through LangSmith with full observability — we can see exactly which chunks were retrieved, what the reranker scored them, and how the LLM used them in generation. When a user reports a bad answer, we can reconstruct the entire pipeline execution in seconds.

Production Concerns

Latency Budget

Enterprise users expect sub-3-second responses for knowledge retrieval. Our latency budget breaks down as:

Total P95: 2.5-3.2 seconds. We stream the LLM response so users see tokens appearing within 500ms of their query, even though generation is not complete. This dramatically improves perceived latency.

Caching Strategy

We implement two-tier caching:

Cost Management

At enterprise scale, RAG costs compound quickly. A single query touches: embedding API (input tokens), vector DB compute, reranker inference, and LLM generation (input + output tokens). At 10,000 queries/day, this adds up.

Our cost control levers:

Guardrails and Citation Tracking

Production RAG systems need guardrails at multiple levels:

from pydantic import BaseModel
from typing import List

class Citation(BaseModel):
    text: str
    source_doc: str
    chunk_id: str
    page: int | None = None
    section: str | None = None

class RAGResponse(BaseModel):
    answer: str
    citations: List[Citation]
    confidence: float  # 0-1, based on retrieval scores
    retrieval_latency_ms: int
    total_latency_ms: int

# We use structured output to enforce citation format
# LLM must output in this schema — no ungrounded claims
response = await openai_client.beta.chat.completions.parse(
    model="gpt-4o",
    messages=[
        {"role": "system", "content": SYSTEM_PROMPT_WITH_CITATION_RULES},
        {"role": "user", "content": assembled_prompt},
    ],
    response_format=RAGResponse,
)

The Tech Stack

For reference, here is the complete stack we use across production RAG deployments:

Lessons Learned

After shipping multiple production RAG systems, these are the patterns we keep returning to:

RAG is not a solved problem — the field moves quickly, and production requirements vary significantly across domains. But the architecture patterns above give you a solid foundation that handles real-world complexity rather than breaking silently on the first query that deviates from your demo dataset.

Need a Production RAG System?

We design, build, and deploy RAG pipelines that work at enterprise scale — with evaluation frameworks, latency optimization, and production guardrails built in from day one.

Start an AI Project