RAG Architecture for Enterprise: What Actually Works in Production

RAG demos go well. The first week in production goes fine. Week two, someone asks a question that spans three documents from different quarters, and the system confidently returns the wrong answer. Cue the Slack message to the engineering team. This pattern shows up across every enterprise RAG deployment, regardless of the team or the industry.

The problem is almost never the model. The problem is the retrieval layer.

The Part Everyone Gets Wrong: Naive Chunking

Most RAG tutorials show you this:

# Split document into 500-token chunks
chunks = text_splitter.split_text(document, chunk_size=500)
# Embed and store
for chunk in chunks:
    vector_store.add(embed(chunk), chunk)

This works in a demo. It fails in production for one specific reason: sentences that contain the answer get split across chunk boundaries. The embedding captures the surrounding context but not the complete fact. Your retrieval pulls the wrong chunk.

The fix is not complicated. It requires caring about document structure.

The Architecture That Holds Up

Here is what our production RAG architecture looks like for an enterprise document corpus, typically 50,000 to 2 million documents:

Ingestion Pipeline
------------------
Raw Document (PDF, DOCX, HTML, email)
    → Document Parser (layout-aware, table extraction)
    → Semantic Chunker (paragraph/section boundaries)
    → Metadata Extractor (date, source, doc type, author)
    → Dual Encoder:
        ├── Dense embedding (OpenAI text-embedding-3-large or Cohere)
        └── Sparse embedding (BM25 keyword index)
    → Vector Store (Weaviate or pgvector depending on scale)

Query Pipeline
--------------
User Query
    → Query Rewriter (expand abbreviations, resolve pronouns)
    → Hybrid Retrieval:
        ├── Dense ANN search (top-k semantic matches)
        └── BM25 keyword search (exact term matches)
        → Reciprocal Rank Fusion (combine both result sets)
    → Reranker (cross-encoder, e.g. Cohere Rerank)
    → Context Assembly (with source metadata)
    → LLM Generation
    → Source Citation Injection

The part that most implementations skip is the reranker. Dense vector search retrieves semantically related chunks, which is great for conceptual queries. But when someone asks about a specific clause number or a date, pure semantic search often pulls the wrong chunk. The reranker runs a cross-encoder over the top 20 retrieved chunks and reorders them before they reach the LLM.

Adding a reranker improved answer accuracy by about 23% in our most recent financial services deployment. Not on a benchmark. On real user questions logged from the first two weeks of production.

Hybrid Retrieval Is Not Optional for Enterprise

Pure dense retrieval misses exact matches. A user searching for "Basel III Article 92(1)(a)" will not get reliable results from a semantic search alone. Pure BM25 keyword search misses synonyms and paraphrasing. Your documents say "capital adequacy ratio" and your user asks "how much capital do we need to hold."

Hybrid retrieval solves both problems. The implementation is straightforward with Weaviate, which supports hybrid search natively. With pgvector, you run both queries and merge results using Reciprocal Rank Fusion:

# Reciprocal Rank Fusion
def rrf_merge(dense_results, sparse_results, k=60):
    scores = {}
    for rank, doc_id in enumerate(dense_results):
        scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank + 1)
    for rank, doc_id in enumerate(sparse_results):
        scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank + 1)
    return sorted(scores.keys(), key=lambda x: scores[x], reverse=True)

Metadata Is Your Secret Weapon

Every chunk you store should carry metadata. At minimum:

• document_id: for source citation
• doc_type: contract, policy, report, email
• date: enables time-filtered queries
• department: enables role-based access at retrieval
• version: so you stop returning superseded policies

The version metadata deserves a separate callout. A common failure in enterprise RAG deployments is returning information from a policy document that had been superseded 18 months earlier. The old document was still in the corpus, still ranked highly, still cited. The fix is a status field on every document and a metadata filter at retrieval time.

Context Window Management

LLMs have finite context windows and you are paying per token. The naive approach passes all retrieved chunks to the LLM. At 20 chunks of 500 tokens each, you are using 10,000 tokens per query before the LLM writes a single word.

The production approach has three steps:

1. Retrieve broadly (top 20 chunks)
2. Rerank aggressively (keep top 5)
3. Compress if needed (extract the 2-3 sentences from each chunk that are most relevant to the query)

Step 3 is optional but valuable for very long documents. A contextual compression step where you run a lightweight model to extract relevant sentences before passing to the main LLM can cut context by 60-70% with minimal quality loss.

The Questions Legal Will Ask

Before you go live in a regulated environment, expect these questions:

"Where does the data go when a query is processed?" The answer needs to be specific: the query text is sent to [model provider], it is not retained for training under our enterprise agreement, and the retrieved chunks never leave your environment.

"Can you explain why the system gave a specific answer?" This is why source citations are not optional. Every generated answer must include the document name, section, and date of the source chunks.

"What happens if a document is updated?" Your ingestion pipeline needs a re-indexing workflow. When a policy document changes, the old chunks need to be retired and the new version indexed. Without this, the system will hallucinate from outdated information, which is worse than not having the system at all.

What We Actually Deploy

For teams under 50,000 documents: pgvector on Postgres. It is already in most enterprise data stacks, handles hybrid search adequately, and avoids introducing a new infrastructure component.

For teams over 50,000 documents or requiring sub-100ms query latency: Weaviate. Its HNSW indexing holds up at scale and its hybrid search is clean to implement.

For the embedding model: OpenAI text-embedding-3-large for most use cases. Note: text-embedding-3-large outputs 3,072 dimensions. pgvector's HNSW index has a 2,000-dimension limit (as of v0.8.x), so if you use pgvector you either need to reduce dimensions via the API's dimensions parameter to 1,536 or below, or switch to Weaviate which handles higher-dimensional vectors natively. Cohere's embedding model if you are deploying on AWS Bedrock and want everything under one enterprise agreement.

For the reranker: Cohere Rerank v3.5 or v4.0 remain strong choices for production use. Cohere Rerank v3.5 offers the best latency (around 600ms average); Rerank v4.0 Pro scores highest on quality benchmarks. Both support 100+ languages.

RAG is not hard to prototype. Getting it production-ready in a regulated environment, with versioned documents, role-based access, audit trails, and consistent accuracy, takes more care. The architecture above is where we start on every engagement. Adjust based on your document volume, compliance requirements, and existing infrastructure.

Sources

• pgvector GitHub — open-source vector similarity search for Postgres
• Weaviate Docs — Hybrid Search
• Cohere — Rerank Models (v3.5 / v4.0)
• OpenAI — Embeddings Guide (text-embedding-3-large dimensions)