How to Build a Production RAG System: Architecture, Patterns, and Best Practices

1. What Is RAG and Why It Matters

Retrieval-Augmented Generation (RAG) is an architecture pattern that grounds Large Language Model responses in external knowledge by retrieving relevant documents at inference time and injecting them into the generation context. Instead of relying solely on the parametric knowledge baked into model weights during training, RAG systems dynamically pull facts from a curated knowledge base, then pass those facts to the LLM alongside the user query.

RAG has become the dominant pattern for enterprise AI applications for several concrete reasons. First, it dramatically reduces hallucination by anchoring responses in verifiable source material. Second, it keeps proprietary data private: your documents never enter the training pipeline of a third-party model. Third, it enables real-time knowledge updates without retraining or fine-tuning. When your documentation changes, you re-index the updated content and retrieval quality improves immediately.

The alternative approaches each carry significant trade-offs. Fine-tuning embeds knowledge into model weights, but it is expensive, slow to update, and creates versioning complexity. Long-context models can accept large documents directly, but costs scale linearly with context length and attention quality degrades over very long inputs. Prompt stuffing hits token limits quickly and provides no mechanism for selective relevance filtering. RAG solves all of these by externalizing knowledge and making retrieval a first-class system component.

2. RAG Architecture Overview

A production RAG system operates as a three-stage pipeline: Ingest, Retrieve, and Generate. Each stage is independently optimizable, and the boundaries between them define the key architectural decisions you need to make early.

Stage 1: Ingest

Raw documents are parsed, cleaned, split into chunks, embedded into vector representations, and stored in a vector database alongside metadata. This is an offline or near-real-time batch process. Ingestion quality is the single largest determinant of downstream retrieval quality. Poor parsing, noisy text, or badly sized chunks propagate errors that no amount of retrieval sophistication can fix.

Stage 2: Retrieve

At query time, the user input is embedded using the same model that produced the document embeddings. The resulting query vector is compared against stored document vectors using similarity search (typically cosine similarity or dot product). Top-k results are returned, optionally reranked by a cross-encoder or additional scoring model. In advanced systems, this stage also includes query expansion, hypothetical document generation, or multi-query decomposition.

Stage 3: Generate

Retrieved chunks are assembled into a prompt alongside the original query, system instructions, and any conversation history. The LLM generates a response grounded in the retrieved context. The generation prompt must explicitly instruct the model to answer only from provided context, cite sources where possible, and acknowledge when the retrieved information is insufficient.

The distinction between naive RAG and advanced RAG lies primarily in retrieval sophistication. Naive RAG uses a single embedding lookup with top-k results passed directly to the LLM. Advanced RAG introduces reranking, hybrid search (combining dense vector and sparse keyword retrieval), query transformation, and iterative retrieval with feedback loops. Most production systems require at least some advanced retrieval techniques to achieve acceptable accuracy.

3. Document Ingestion Pipeline

The ingestion pipeline is where most RAG projects succeed or fail. A common mistake is treating ingestion as a solved problem and rushing to retrieval optimization. In practice, the quality ceiling of your RAG system is set during ingestion.

Document Parsing

Different source formats require different parsing strategies. PDFs with complex layouts, tables, and multi-column formatting are notoriously difficult. HTML documents carry structural information that should be preserved. Markdown and plain text are straightforward but may lack structural cues that help chunking.

• LlamaParse handles complex PDFs with layout awareness, preserving tables and figures as structured data rather than flattened text. It is the strongest option for documents where spatial layout carries semantic meaning.
• Unstructured.io provides a unified API across formats (PDF, DOCX, PPTX, HTML, images) with configurable extraction strategies. It supports both rule-based and model-based partitioning.
• Apache Tika is a mature, open-source option that handles 1,000+ file formats. It is reliable for basic extraction but lacks the layout awareness of newer tools.
• Docling (by IBM) provides high-quality PDF parsing with table extraction and OCR fallback. A strong open-source alternative for structured document processing.

Metadata Extraction

Every chunk should carry metadata: source document title, section heading, page number, author, creation date, document type, and any domain-specific tags. This metadata enables filtered retrieval (e.g., "only search documents from Q4 2025") and improves ranking. Metadata that seems redundant during ingestion becomes essential when debugging retrieval failures.

Cleaning and Normalization

Remove headers, footers, page numbers, watermarks, and repeated boilerplate. Normalize unicode, fix encoding issues, collapse excessive whitespace. For code-heavy documents, preserve code blocks as atomic units. For multilingual corpora, detect language per chunk and store it as metadata so you can match query language to document language during retrieval.

4. Chunking Strategies That Actually Work

Chunking determines the granularity of retrieval. Chunks that are too large dilute relevance with irrelevant content and waste context window tokens. Chunks that are too small lose the surrounding context needed for the LLM to generate a coherent answer. The optimal strategy depends on your document types, query patterns, and embedding model.

Overlap Strategies

Chunk overlap ensures that information spanning a chunk boundary is captured in at least one chunk. Without overlap, a fact split across two chunks may be partially retrieved and misinterpreted. A 10-20% overlap (e.g., 50-100 tokens for 512-token chunks) is standard. Higher overlap increases storage and indexing cost but reduces boundary-related retrieval failures.

Semantic Chunking in Detail

Semantic chunking uses embedding similarity between consecutive sentences to determine chunk boundaries. When the cosine similarity between adjacent sentence embeddings drops below a threshold, a new chunk starts. This produces chunks that align with topical shifts in the document rather than arbitrary character or token counts. Libraries like LlamaIndex and LangChain provide implementations, and Greg Kamradt's work on semantic chunking remains a practical reference.

5. Vector Database Selection Guide

Your vector database handles storage, indexing, and similarity search over embeddings. The right choice depends on your existing infrastructure, scale requirements, and whether you need hybrid search (combining vector and keyword retrieval).

Practical Recommendations

• You already use PostgreSQL: Start with pgvector. It eliminates an additional infrastructure dependency, supports ACID transactions for metadata, and handles millions of vectors with HNSW indexing. Use pgvector 0.7+ for HNSW support (significantly faster than IVFFlat for recall-sensitive workloads).
• You need the fastest path to production: Pinecone. Fully managed, no indexing configuration, built-in hybrid search with sparse-dense vectors. The trade-off is vendor lock-in and higher per-query costs at scale.
• Hybrid search is critical: Weaviate or Elasticsearch. Both provide native BM25 + vector search fusion with configurable weighting. Weaviate offers a cleaner developer experience; Elasticsearch offers broader ecosystem integration.
• Prototyping or small scale: Chroma runs embedded in your application process, requires zero infrastructure, and is sufficient for datasets under a million documents. Migrate to a production database when you outgrow it.

6. Retrieval Strategies Beyond Naive Search

Naive RAG retrieves the top-k most similar vectors and passes them directly to the LLM. This works for simple, single-topic queries against clean, well-chunked corpora. It fails for ambiguous queries, multi-faceted questions, out-of-vocabulary terms, and documents where keyword overlap matters as much as semantic similarity.

Semantic Search (Dense Retrieval)

The baseline: embed the query and retrieve by cosine similarity or dot product. Use models like text-embedding-3-large (OpenAI), voyage-3-large (Voyage AI), or bge-m3 (open source, multilingual). Dense retrieval excels at capturing meaning even when query and document use different terminology.

Keyword Search (Sparse Retrieval / BM25)

BM25 ranks documents by term frequency and inverse document frequency. It handles exact-match queries, proper nouns, technical identifiers, and code references better than dense retrieval. If a user searches for "error code NX-4012", BM25 will find it; embedding similarity may not.

Hybrid Search

Combine dense and sparse retrieval, then fuse the ranked results. The most common fusion method is Reciprocal Rank Fusion (RRF), which combines rankings without requiring score normalization. Hybrid search consistently outperforms either method alone in benchmarks and production systems. Weight the fusion toward dense retrieval for conceptual queries and toward sparse retrieval for keyword-heavy or technical queries.

Reranking

Retrieve a larger candidate set (e.g., top-50) with fast approximate search, then rerank with a more expensive cross-encoder model. Cross-encoders process the query and document together, enabling deeper interaction modeling than independent embedding comparison. Cohere Rerank, Jina Reranker, and open-source models like bge-reranker-v2-m3 are practical options. Reranking typically improves precision@5 by 10-25% over unreranked results.

Query Expansion

Use the LLM to generate multiple reformulations of the original query, retrieve for each, and merge results. This addresses the vocabulary mismatch problem: the user may phrase their question differently than the source document. Multi-query retrieval with deduplication is straightforward to implement and improves recall on ambiguous queries.

HyDE (Hypothetical Document Embeddings)

Ask the LLM to generate a hypothetical answer to the query, embed that answer, and use it as the retrieval vector. The hypothesis is more lexically and semantically similar to actual documents than the short query. HyDE adds one LLM call of latency but can significantly improve retrieval for complex or abstract questions.

Parent-Child Retrieval

Index small, precise chunks (child) for retrieval accuracy, but return the larger parent chunk (or full section) to the LLM for generation context. This gives you the best of both worlds: granular matching and comprehensive context. LlamaIndex and LangChain both support this pattern natively.

7. Generation: Prompt Design for RAG

The generation stage transforms retrieved chunks into a coherent, grounded response. Prompt design at this stage determines whether the LLM faithfully uses the retrieved context or hallucinates beyond it.

Context Window Management

Calculate your token budget: system prompt + retrieved context + user query + expected response length must fit within the model's context window. For a 128k-context model, this is rarely a constraint. For 8k or 16k models, aggressive chunking and top-k limits are essential. Always leave headroom: if you pack the context window to 95% capacity, the model has insufficient space for a complete response.

Prompt Structure

A production RAG prompt should contain four components in this order:

1. System instructions: Define the assistant's role, constraints, and behavior. Explicitly instruct the model to answer only from provided context, to cite source documents, and to say "I don't have enough information to answer this" when context is insufficient.
2. Retrieved context: Inject the top-k chunks, clearly delineated with source identifiers (e.g., [Source: document_title, page 14]). Order chunks by relevance score so the most relevant material appears first in the context.
3. Conversation history: For multi-turn interactions, include recent conversation turns so the LLM can resolve coreferences and maintain coherence.
4. User query: Place the current question last, immediately before the model generates its response. This recency positioning helps the model focus on the actual question.

Grounding and Attribution

Instruct the LLM to cite the specific source document(s) it uses for each claim. This enables users to verify answers and builds trust. Structured output (e.g., JSON with "answer" and "sources" fields) makes citation parsing programmatic. If your application serves end users, inline citations like [1] with a reference list at the end work well.

Handling Insufficient Context

The most dangerous failure mode in RAG is the LLM generating a confident-sounding answer when the retrieved context does not actually contain the answer. Your system prompt must address this explicitly: "If the provided context does not contain sufficient information to answer the question, state that clearly. Do not speculate or use information from outside the provided context." Test this behavior regularly with queries that intentionally have no matching documents.

Temperature and Sampling

For factual RAG responses, use low temperature (0.0-0.3). Higher temperatures increase creativity at the cost of factual grounding. For summarization or synthesis tasks where some stylistic variation is acceptable, 0.3-0.5 is reasonable. Never exceed 0.7 for RAG applications where accuracy matters.

8. Evaluation Framework

A RAG system without evaluation is guesswork. You need to measure both retrieval quality (are you finding the right chunks?) and generation quality (is the LLM producing faithful, relevant answers?). These are independent failure modes: perfect retrieval with poor prompting produces bad answers, and perfect prompting with bad retrieval produces hallucinations.

Retrieval Metrics

• Precision@k: Of the top-k retrieved chunks, what fraction are relevant to the query? Target: 0.7+ for top-5.
• Recall@k: Of all relevant chunks in the corpus, what fraction appear in the top-k results? Target: 0.8+ for top-10.
• Mean Reciprocal Rank (MRR): At what rank does the first relevant result appear? Higher MRR means the best result surfaces earlier. Target: 0.7+.
• Normalized Discounted Cumulative Gain (NDCG): Measures ranking quality with position-weighted relevance. Preferred over precision/recall when you care about the ordering of results, not just their presence.

Generation Metrics

• Faithfulness: Does the response only contain claims supported by the retrieved context? This is the hallucination detection metric. Target: 0.9+.
• Answer relevance: Does the response actually answer the question that was asked? A faithful but off-topic response fails this metric. Target: 0.85+.
• Context relevance: Are the retrieved chunks relevant to the query? Low context relevance indicates a retrieval problem, not a generation problem. Target: 0.75+.
• Completeness: Does the response cover all aspects of the question? Particularly important for multi-part queries.

Evaluation Frameworks

• RAGAS (Retrieval Augmented Generation Assessment) provides automated metrics for faithfulness, answer relevance, and context relevance using LLM-as-judge evaluation. It is the most widely adopted RAG evaluation framework.
• DeepEval extends RAGAS with additional metrics (hallucination, bias, toxicity) and integrates with CI/CD for regression testing.
• Custom pipelines: For domain-specific accuracy requirements (medical, legal, financial), build custom evaluation sets with expert-labeled ground truth. Automated metrics are useful for continuous monitoring, but human evaluation remains the gold standard for high-stakes domains.

9. Production Deployment Checklist

Deploying a RAG system to production requires more than a working retrieval pipeline. The following checklist covers the operational concerns that differentiate a demo from a reliable system.

1. Monitoring and observability. Log every query, retrieval result set, and generated response. Track retrieval latency (p50, p95, p99), generation latency, embedding model latency, and end-to-end response time. Set up alerts for latency spikes, retrieval quality degradation, and error rates. Tools like LangSmith, Langfuse, or Arize Phoenix provide purpose-built RAG observability.
2. Caching strategy. Cache embeddings for frequently repeated queries. Cache LLM responses for identical query + context combinations. Semantic caching (retrieving cached responses for semantically similar queries) reduces cost and latency for high-volume use cases. Redis or Momento are common choices for the caching layer.
3. Fallback behavior. Define what happens when retrieval returns no relevant results (low similarity scores). Define what happens when the LLM API is unavailable. Provide graceful degradation: a helpful "I cannot answer this" message is infinitely better than a hallucinated response or a 500 error.
4. Rate limiting and concurrency. Protect your embedding model and LLM API from traffic spikes. Implement request queuing with backpressure. Set per-user and per-tenant rate limits to prevent one consumer from starving others.
5. Cost controls. Track token usage per query. Set daily and monthly spend limits on LLM API calls. Monitor embedding model costs (they are often overlooked). Implement circuit breakers that switch to cheaper models or cached responses when spend thresholds are approached.
6. Latency optimization. Pre-compute and cache embeddings for known query patterns. Use approximate nearest neighbor (ANN) indexes (HNSW, IVF) instead of brute-force search. Stream LLM responses to reduce perceived latency. Run retrieval and any parallel operations concurrently.
7. Index refresh strategy. Define how and when new documents enter the index. Incremental indexing (adding new documents without re-indexing the entire corpus) is essential for large-scale systems. Set up a pipeline that detects document changes and triggers re-embedding. Handle document deletion and updates (not just additions).
8. Error handling and recovery. Retry transient failures with exponential backoff. Handle embedding model timeouts gracefully. Implement dead-letter queues for failed ingestion jobs. Log and alert on structured error categories, not just raw exceptions.
9. Security and access control. Implement document-level access control: users should only retrieve documents they are authorized to see. Filter retrieval results by user permissions before passing context to the LLM. Sanitize inputs to prevent prompt injection attacks that attempt to override system instructions.
10. User feedback loop. Capture explicit feedback (thumbs up/down, corrections) and implicit signals (query reformulation, follow-up questions) to identify retrieval and generation failures. Route feedback to your evaluation pipeline for continuous improvement.

10. Common Pitfalls and How to Avoid Them

Chunks too large

Large chunks (1000+ tokens) dilute the relevance signal. When a chunk contains five paragraphs, only one of which is relevant, the embedding represents an average of all five. Retrieval matches are weaker, and the LLM receives more noise than signal. Solution: Use smaller chunks (256-512 tokens) for retrieval, with parent-child strategies to expand context at generation time.

Chunks too small

Very small chunks (under 100 tokens) lack sufficient context for the embedding model to produce meaningful representations. They also force the LLM to synthesize across many fragments, increasing the risk of inconsistency. Solution: Ensure each chunk contains enough context to be interpretable in isolation. The chunk should answer the question: "If I read only this chunk, would I understand what it is about?"

No evaluation pipeline

Teams that optimize "by feel" (reading a few responses and deciding they look good) inevitably introduce regressions. A change that improves responses for one query type degrades another, and without systematic evaluation, you will not catch it. Solution: Build evaluation first, optimize second. Even 50 labeled examples are sufficient for a meaningful baseline.

Ignoring metadata

Metadata-free chunks force the retrieval system to rely entirely on semantic similarity, which fails for queries that require temporal, authorial, or categorical filtering. "What was our Q3 revenue policy?" requires metadata filtering by date and department, not just content similarity. Solution: Extract and index metadata during ingestion. Support filtered retrieval queries.

Skipping reranking

Embedding-based retrieval using approximate nearest neighbor search is optimized for speed, not precision. The top-10 results from ANN search frequently include 2-3 irrelevant chunks that a cross-encoder reranker would filter out. Solution: Add a reranking step for any use case where precision matters. The latency cost (50-200ms) is almost always acceptable.

Not handling no-result cases

When retrieval returns low-similarity results, the system should recognize that it does not have relevant information rather than passing garbage context to the LLM. Without a similarity threshold or confidence check, the LLM will attempt to answer from irrelevant context and confidently produce wrong answers. Solution: Set a minimum similarity threshold. When all results fall below it, respond with a clear "I don't have information about this" message.

Over-relying on embeddings alone

Dense retrieval captures semantic meaning but misses exact matches on technical terms, product codes, error messages, and proper nouns. A user searching for "CUDA error 719" needs exact keyword matching, not semantic similarity to the concept of GPU errors. Solution: Implement hybrid search from the start. The cost is minimal and the coverage improvement is significant.

No caching strategy

Every query triggers an embedding call, a vector search, and an LLM call. For high-volume systems, this adds up to substantial cost and latency. Many production workloads have significant query repetition (support systems, internal tools, FAQ-like patterns). Solution: Cache at every layer: embedding cache, retrieval result cache, and response cache. Semantic caching extends coverage to paraphrased queries.