Building Production RAG Systems: Architecture, Pitfalls, and Evaluation

Introduction

Retrieval-Augmented Generation (RAG) has become the dominant paradigm for building LLM applications that need to work with custom data. But the gap between a RAG demo and a production RAG system is enormous.

This article covers the architectural decisions, common pitfalls, and evaluation strategies for building RAG systems that actually work.

Why RAG?

Pure LLMs have fundamental limitations:

Knowledge cutoff: Training data has a fixed date
Hallucination: Models generate plausible but false information
No private data: Cannot access organization-specific information
No attribution: Cannot cite sources for claims

RAG addresses all of these by grounding LLM responses in retrieved evidence.

RAG Architecture

text

User Query
    ↓
Query Processing
    ↓
Embedding Model
    ↓
Vector Search
    ↓
Context Assembly
    ↓
LLM Generation
    ↓
Response + Citations

Let's examine each component.

Component 1: Document Ingestion

Before retrieval can happen, documents must be processed and indexed.

Chunking Strategies

Chunking is arguably the most impactful design decision in a RAG system.

python

def chunk_by_semantic_sections(text, max_tokens=512):
    """Chunk text by semantic sections rather than fixed size."""
    sections = []
    current_section = []
    current_tokens = 0
    
    for paragraph in text.split('\n\n'):
        paragraph_tokens = len(paragraph.split())
        
        if current_tokens + paragraph_tokens > max_tokens:
            sections.append('\n\n'.join(current_section))
            current_section = [paragraph]
            current_tokens = paragraph_tokens
        else:
            current_section.append(paragraph)
            current_tokens += paragraph_tokens
    
    if current_section:
        sections.append('\n\n'.join(current_section))
    
    return sections

Chunking Comparison

Strategy	Pros	Cons	Best For
Fixed-size	Simple, predictable	Breaks semantic units	Homogeneous docs
Sentence-based	Preserves meaning	Variable chunk sizes	Conversational text
Semantic	Context-aware	Slower processing	Technical docs
Document-aware	Respects structure	Requires format parsing	Structured content

Component 2: Embedding

Embeddings convert text into dense vector representations for similarity search.

Model Selection

python

from sentence_transformers import SentenceTransformer

# For general-purpose retrieval
model = SentenceTransformer('all-MiniLM-L6-v2')

# For higher quality at the cost of speed
model = SentenceTransformer('all-mpnet-base-v2')

def embed_chunks(chunks):
    """Generate embeddings for a list of text chunks."""
    embeddings = model.encode(
        chunks,
        batch_size=32,
        show_progress_bar=True,
        normalize_embeddings=True,
    )
    return embeddings

Component 3: Retrieval

Vector Search

python

import numpy as np

def retrieve(query, index, chunks, top_k=5):
    """Retrieve the most relevant chunks for a query."""
    query_embedding = model.encode([query], normalize_embeddings=True)
    
    # Cosine similarity (embeddings are normalized)
    similarities = np.dot(index, query_embedding.T).flatten()
    
    top_indices = np.argsort(similarities)[::-1][:top_k]
    
    results = []
    for idx in top_indices:
        results.append({
            'chunk': chunks[idx],
            'score': float(similarities[idx]),
        })
    
    return results

Hybrid Retrieval

Pure vector search misses exact keyword matches. Combine it with BM25:

text

Query
    ↓
┌───────────┬──────────────┐
│  Vector   │    BM25      │
│  Search   │   Search     │
└─────┬─────┴──────┬───────┘
      │             │
      └──────┬──────┘
             ↓
      Reciprocal Rank
         Fusion
             ↓
       Final Results

Component 4: Generation

Prompt Engineering for RAG

python

def build_rag_prompt(query, retrieved_chunks):
    """Build a grounded prompt with retrieved context."""
    context = "\n\n---\n\n".join([
        f"[Source {i+1}]: {chunk['chunk']}"
        for i, chunk in enumerate(retrieved_chunks)
    ])
    
    prompt = f"""Answer the following question using ONLY the provided context.
If the context doesn't contain enough information, say so.
Always cite your sources using [Source N] notation.

Context:
{context}

Question: {query}

Answer:"""
    
    return prompt

Evaluation

This is where most RAG implementations fall short. You need to evaluate both retrieval and generation.

Retrieval Metrics

Recall@K: What fraction of relevant documents are retrieved?
MRR: How high is the first relevant document ranked?
NDCG: Are relevant documents ranked in the right order?

Generation Metrics

Faithfulness: Does the answer stick to the retrieved context?
Answer Relevance: Does the answer address the question?
Context Precision: How much retrieved context is actually relevant?

RAG Evaluation Framework

text

Test Query
    ↓
┌─────────────────┐
│   Retrieval     │ → Recall, MRR, NDCG
│   Evaluation    │
└────────┬────────┘
         ↓
┌─────────────────┐
│   Generation    │ → Faithfulness, Relevance
│   Evaluation    │
└────────┬────────┘
         ↓
    Aggregate Score

Common Pitfalls

Chunk size too large: Dilutes relevant information with noise
Chunk size too small: Loses context needed for understanding
No overlap: Critical information at chunk boundaries is lost
Wrong embedding model: General models vs. domain-specific needs
Ignoring metadata: Dates, sources, and categories improve retrieval
No evaluation: "It looks good" is not a metric
Context window stuffing: More context ≠ better answers

Production Considerations

Caching: Cache embeddings and frequent queries
Monitoring: Track retrieval quality and user feedback
Versioning: Version your chunks, embeddings, and prompts
Fallback: Gracefully handle retrieval failures
Cost: Balance embedding model quality vs. API costs

Conclusion

Building a production RAG system requires careful attention to every component of the pipeline. The difference between a demo and a production system lies in the details — chunking strategy, retrieval quality, prompt design, and rigorous evaluation.

References

"Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" (Lewis et al., 2020)
LangChain RAG documentation
LlamaIndex query engine guide