Language models have a problem. They are trained on enormous corpora of text, acquiring broad knowledge and impressive generative ability. But that knowledge is frozen at the training cutoff. Ask a model about events after that date and it will either say it does not know or, more dangerously, hallucinate plausible-sounding answers. Ask it about your company's internal documents and it has never seen them. Ask it about your product's current pricing and it will either apologize or invent something.
The obvious engineering solution: give the model access to current, specific, or proprietary information at the time it answers. Not by retraining it — that is expensive and slow. By providing relevant documents as input, alongside the query, so the model can ground its response in verified content rather than generating from statistical inference alone.
This is Retrieval Augmented Generation — RAG. It was formalized in a 2020 paper from Facebook AI Research, but the underlying idea is straightforward: combine a retrieval system with a language model, so the model generates answers grounded in retrieved evidence rather than solely in learned parameters.
RAG has become the dominant pattern for deploying language models in production applications that require factual accuracy, current information, or access to specific knowledge bases — enterprise knowledge management, customer support, research tools, and document Q&A.
"Retrieval augmented generation addresses the fundamental challenge of parametric memory in language models: it's not efficient to encode every fact the world needs into model parameters. Retrieval provides access to a virtually unlimited, updatable knowledge store." — Patrick Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020)
Key Definitions
Retrieval Augmented Generation (RAG) — An AI system architecture that combines a language model with a retrieval system. A query triggers retrieval of relevant documents from a knowledge base; those documents are provided to the language model alongside the query; the model generates a response grounded in the retrieved content. RAG reduces hallucinations and enables models to access current or proprietary information without retraining.
Retriever — The component of a RAG system responsible for finding relevant documents in the knowledge base for a given query. Retrievers may use sparse methods (keyword-based search), dense methods (embedding-based semantic search), or hybrid approaches combining both.
Generator — The language model component of a RAG system that produces the final response, conditioned on both the query and the retrieved documents. The generator's quality determines how well it uses the provided context to generate accurate, grounded responses.
Vector embedding — A numerical vector representation of text that captures its semantic meaning. Similar texts have embeddings that are geometrically close (high cosine similarity); dissimilar texts have embeddings that are far apart. Embeddings are produced by embedding models trained specifically for this purpose. They are the foundation of semantic search in modern RAG systems.
Vector database — A database optimized for storing and querying high-dimensional vector embeddings. Enables similarity search: finding documents whose embeddings are closest to a query embedding. Used as the retrieval backend in most production RAG systems. Examples: Pinecone, Weaviate, Chroma, Milvus, pgvector.
Chunking — The process of dividing documents into smaller segments for indexing and retrieval. Because language models have limited context windows, documents too large to fit must be split. Chunking strategy — how large each chunk is, whether chunks overlap, whether chunks respect document structure — significantly affects retrieval quality.
Semantic search — Finding documents based on meaning rather than exact keyword match. A semantic search for "car maintenance" might retrieve documents about "vehicle upkeep" or "automobile service" even if those exact words do not appear in the query. Semantic search in RAG systems is enabled by embedding-based similarity search.
Sparse retrieval — Retrieval based on term frequency or keyword overlap, typically using algorithms like BM25 (Best Match 25). Sparse retrieval is fast and effective for queries where keywords matter — technical terms, proper nouns, specific phrases. It does not capture semantic similarity.
Dense retrieval — Retrieval based on embedding similarity. A query and all documents are converted to embeddings; retrieval finds the most semantically similar documents. Dense retrieval captures meaning better than sparse retrieval for natural language queries, but may miss exact keyword matches.
Hybrid retrieval — Combining sparse and dense retrieval, often by interleaving or reranking results from both methods. Hybrid retrieval generally outperforms either approach alone, capturing both keyword specificity and semantic similarity.
Reranking — A post-retrieval step that uses a separate, typically more powerful model to reorder retrieved documents by relevance to the query. Rerankers are typically cross-encoders (considering query and document jointly) and are more accurate than the initial retriever but too slow to apply to the full knowledge base.
Context window — The maximum amount of text a language model can process in a single forward pass. Context windows limit how many retrieved documents can be provided to the generator. Larger context windows reduce the need for aggressive chunking and allow more documents to be included.
The RAG Architecture: How It Works
A standard RAG system has two phases: indexing (offline) and querying (online).
Indexing Phase (Offline)
Before the system can answer queries, the knowledge base must be indexed:
Document ingestion: Source documents (PDFs, web pages, databases, internal documents) are collected and preprocessed — extracting text from various formats, cleaning formatting artifacts.
Chunking: Documents are split into segments sized to fit in the context window alongside the query and other chunks. Typical chunk sizes range from 256 to 1,024 tokens. Strategies include fixed-size chunking, sentence-aware chunking (splitting at sentence boundaries), paragraph-level chunking, and recursive chunking that tries to preserve document structure.
Embedding: Each chunk is converted to a vector embedding using an embedding model. This is the most computationally expensive part of indexing and is done once per document, then reused for all queries.
Storage: Chunk text and embeddings are stored in a vector database (or combined storage system) that supports efficient similarity search.
Query Phase (Online)
When a user submits a query:
Query embedding: The query is converted to an embedding using the same embedding model used during indexing.
Retrieval: The vector database returns the top-K chunks whose embeddings are most similar to the query embedding. K is typically 3–10, depending on context window size and chunk size.
Optional reranking: Retrieved chunks are reranked by a cross-encoder model to improve relevance ordering.
Context construction: Retrieved chunks are assembled into a context string, typically with their source information (document title, page number, URL) included.
Generation: The language model receives a prompt containing the query and the retrieved context, with instructions to answer based on the provided documents. The model generates a response grounded in the retrieved content.
Optional citation: The response may include references to the specific source documents from which information was drawn.
Chunking Strategies and Their Tradeoffs
Chunking is one of the most consequential design decisions in a RAG system. Poor chunking degrades retrieval quality even when the underlying documents are comprehensive.
| Strategy | Description | Strengths | Weaknesses |
|---|---|---|---|
| Fixed-size | Chunks of equal token count | Simple; predictable size | May split mid-sentence or mid-argument |
| Sentence-aware | Split at sentence boundaries | Preserves linguistic units | Variable chunk size |
| Paragraph-level | One chunk per paragraph | Preserves thematic coherence | Paragraphs vary widely in length |
| Hierarchical | Multiple granularities (paragraph → sentence) | Flexible retrieval at different granularities | More complex infrastructure |
| Semantic | Split where topic changes | Preserves semantic coherence | Requires semantic segmentation model |
Overlapping chunks — where adjacent chunks share some tokens — help preserve context that would otherwise be cut at chunk boundaries. A typical overlap is 10–20% of chunk size.
RAG vs. Fine-Tuning vs. Prompt Engineering
These three approaches are often considered as alternatives but are more complementary than competing:
Prompt engineering modifies the model's behavior through the instructions and context provided at inference time. It is the fastest to implement and requires no training, but it is limited to what can fit in the context window and cannot teach the model new information.
Fine-tuning modifies the model's weights through additional training on specific data. It is effective for teaching new reasoning patterns, adjusting response style, or improving performance in specific domains. It is not well-suited for providing access to large, changing knowledge bases: fine-tuning is expensive, cannot easily be updated as the knowledge base changes, and does not support attribution.
RAG provides specific documents at inference time. It is well-suited for large, changing, or proprietary knowledge bases; it supports attribution; it does not require modifying model weights; and it can be updated by adding new documents to the knowledge base without retraining.
"For applications where the core requirement is factual grounding in a specific knowledge base, RAG is usually the right starting point. For applications where the core requirement is behavioral modification — a different tone, domain-specific reasoning patterns, specialized task formats — fine-tuning is the right starting point." — Various practitioners (consensus view, 2023–2024)
Many production systems use both: RAG for knowledge grounding, fine-tuning for behavioral alignment.
Common Failure Modes
Retrieval Failures
The generator can only produce accurate responses if the retriever finds the right documents. Common retrieval failures:
- False negatives: Relevant documents are in the knowledge base but are not retrieved, either because the embedding model does not capture the semantic relationship or because the chunks are too small or too large to match the query.
- False positives: Irrelevant documents are retrieved and included in context, potentially misleading the generator or consuming context window capacity.
- Query-document mismatch: The way users phrase queries often differs substantially from how information is phrased in documents. Embedding models that handle this translation imperfectly produce poor retrieval.
Model Faithfulness Failures
Even with relevant documents retrieved, the generator may not use them faithfully:
- Hallucinating beyond the retrieved context: The model generates claims not supported by the provided documents, reverting to parametric memory.
- Ignoring contradictions: The model generates responses that ignore retrieved documents that contradict its prior knowledge.
- Lost-in-the-middle: Research by Liu et al. (2023) found that language models perform worse on information located in the middle of a long context compared to information at the beginning or end — important for RAG systems providing many retrieved documents.
Chunking and Context Problems
- Context fragmentation: A coherent argument or explanation spanning multiple paragraphs is split across chunks; no single chunk contains enough context for the retriever to match it to queries that require the full argument.
- Context window overflow: More retrieved documents than fit in the context window, requiring truncation and losing potentially relevant content.
Building Production RAG Systems
The gap between a prototype RAG system (query → retrieve → generate) and a production RAG system is substantial. Production systems require:
Evaluation pipelines: Systematic measurement of retrieval quality (precision, recall, mean reciprocal rank) and response quality (faithfulness to retrieved context, answer relevance, groundedness). Tools like RAGAS provide automated evaluation frameworks.
Monitoring: Tracking retrieval latency, response quality metrics, and failure modes in production traffic.
Document update handling: Strategies for re-embedding and re-indexing documents when they change, without requiring full re-indexing of the knowledge base.
Access control: Ensuring users can only retrieve documents they have permission to see — critical for enterprise deployments with multiple user roles and sensitive document categories.
For related concepts, see AI hallucinations explained, large language models explained, and AI prompt engineering guide.
References
- Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W., Rocktäschel, T., Riedel, S., & Kiela, D. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Advances in Neural Information Processing Systems, 33. https://arxiv.org/abs/2005.11401
- Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., & Wang, H. (2023). Retrieval-Augmented Generation for Large Language Models: A Survey. arXiv preprint arXiv:2312.10997. https://arxiv.org/abs/2312.10997
- Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2023). Lost in the Middle: How Language Models Use Long Contexts. arXiv preprint arXiv:2307.03172. https://arxiv.org/abs/2307.03172
- Karpukhin, V., Oğuz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., & Yih, W. (2020). Dense Passage Retrieval for Open-Domain Question Answering. Proceedings of EMNLP 2020. https://arxiv.org/abs/2004.04906
- Robertson, S., & Zaragoza, H. (2009). The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends in Information Retrieval, 3(4), 333–389. https://doi.org/10.1561/1500000019
- Es, S., James, J., Anke, L. E., & Schockaert, S. (2023). RAGAS: Automated Evaluation of Retrieval Augmented Generation. arXiv preprint arXiv:2309.15217. https://arxiv.org/abs/2309.15217
Frequently Asked Questions
What is Retrieval Augmented Generation (RAG)?
Retrieval Augmented Generation (RAG) is an AI architecture that combines a language model with a retrieval system. When a query is received, a retriever searches a knowledge base for relevant documents. Those documents are injected into the language model's context alongside the query, and the model generates a response grounded in the retrieved content. RAG allows language models to answer questions about specific, up-to-date, or proprietary information without retraining.
Why use RAG instead of just asking the language model directly?
Language models have fixed training data with a knowledge cutoff date, and they hallucinate when generating facts from statistical inference rather than retrieved sources. RAG addresses both problems: it provides the model with specific, current, or proprietary documents at query time, grounding responses in verifiable sources and significantly reducing hallucinations. RAG also allows attribution — responses can be traced back to specific documents.
How does RAG retrieve relevant documents?
The most common retrieval approach in modern RAG systems uses vector embeddings and semantic search. Documents are converted to numerical vector embeddings using an embedding model; the query is converted to an embedding in the same space. Relevant documents are retrieved by finding those whose embeddings are most similar to the query embedding — typically using cosine similarity or approximate nearest neighbor search in a vector database. This captures semantic similarity rather than just keyword matching.
What is a vector database?
A vector database is a database optimized for storing and searching high-dimensional numerical vectors (embeddings). Unlike traditional databases that search by exact or fuzzy string match, vector databases support similarity search: finding the vectors closest to a query vector in high-dimensional space. Popular vector databases include Pinecone, Weaviate, Chroma, Milvus, and pgvector (an extension for PostgreSQL). They are the retrieval backbone of most production RAG systems.
What is the difference between RAG and fine-tuning?
Fine-tuning updates the model's weights by training it on specific data. RAG leaves the model weights unchanged and instead provides relevant documents at inference time. Fine-tuning is better for teaching the model new reasoning patterns, styles, or domain-specific behaviors. RAG is better for providing access to specific factual information, keeping responses current, and enabling attribution. RAG is also more economical than fine-tuning for frequently-changing knowledge bases.
What are the main failure modes of RAG systems?
Common RAG failure modes include: retrieval failures (the retriever fails to find relevant documents, or retrieves irrelevant ones); context window overflow (retrieved documents exceed the model's context limit); model faithfulness failures (the model ignores or contradicts retrieved documents); chunking errors (documents are split in ways that lose important context); and embedding mismatch (the embedding model fails to capture the semantic similarity between query and relevant documents).
When should you use RAG versus other approaches?
Use RAG when you need the model to access specific, up-to-date, or proprietary factual information — company documents, current data, specialized knowledge bases. Use fine-tuning when you need to modify the model's reasoning style, domain-specific behavior, or output format. Use prompt engineering and system prompts when you need to control response tone, format, or general behavior without additional knowledge. Many production systems combine RAG with fine-tuning.