The Four Components of a Retrieval-Augmented Generation (RAG) System: Indexing, Retrieval, Generation, and Evaluation

Enterprise architects deciding how to deploy large language models face a fundamental constraint: they control the data the model can access, but they must define that access architecture before a user ever types a prompt. By implementing a Retrieval-Augmented Generation (RAG) system, developers are able to index their proprietary documents into a searchable database, determining exactly what facts the artificial intelligence will retrieve when it next generates an answer. This architectural choice shifts the burden of knowledge from the neural network's internal memory to an external, easily auditable storage system.

The approach solves the core limitation of generative artificial intelligence. Large language models are frozen in time at the exact moment their multi-million-dollar training runs complete. When asked about recent events, shifting market conditions, or private corporate data, they either fail to answer or hallucinate plausible-sounding falsehoods. Because retraining a foundation model from scratch is computationally prohibitive for most organizations, the industry required a mechanism to feed fresh, verifiable data to an existing model at inference time, bridging the gap between static training and dynamic real-world applications.

RAG bypasses the need for constant retraining by fundamentally altering how the model sources its facts. As Meta AI researchers Patrick Lewis and Douwe Kiela explained in their foundational 2020 paper, the architecture combines parametric memory—the knowledge stored in the model's weights—with nonparametric memory, an external corpus of documents that can be updated instantly. The entire system is built on four distinct operational components: indexing, retrieval, generation, and evaluation. Together, these phases ensure that the language model acts as a reasoning engine processing provided evidence, rather than a flawed database attempting to recall memorized trivia.[1][2]

The first component of the architecture is indexing. Before a RAG system can answer a single user question, it must build a comprehensive, searchable library. The indexing phase takes raw, unstructured data—PDFs, corporate wikis, research papers, and customer interaction logs—and breaks it down into manageable pieces called chunks. This preprocessing step is absolutely essential because language models have strict mathematical limits on how much text they can process at once, known as the context window. By dividing massive datasets into discrete blocks, developers ensure the system only retrieves the specific paragraphs relevant to a query.

According to IBM's architectural documentation, chunk size is a critical hyperparameter that dictates the system's overall performance and accuracy. If chunks are too large, they dilute the specific facts, retrieve irrelevant surrounding text, and quickly overwhelm the language model's context window. Conversely, if they are too small, they lose semantic coherency and strip away the necessary context required to understand the fragment. Typical chunk sizes range from 256 to 1024 tokens, requiring developers to carefully balance precision against comprehensiveness when designing the initial database structure.[4]

Once the documents are chunked, a specialized embedding model translates the raw text into high-dimensional vectors. "RAG systems use a process called embedding to transform data into numerical representations called vectors," IBM notes in its technical breakdown. These vectors—often containing 768 or 1536 distinct dimensions—are stored in a specialized vector database, mapping the entire knowledge base into a vast mathematical space where semantically similar concepts are clustered closely together. This spatial arrangement allows the system to understand the underlying meaning behind words, enabling it to match queries with relevant documents even if they do not share exact keywords.[4]

The second component of the pipeline is retrieval. When a user submits a prompt, the system does not immediately send that text to the generative language model. Instead, the retrieval component intercepts the query and translates it into a vector using the exact same embedding model that was used during the indexing phase. This critical step ensures the user's question exists in the exact same mathematical space as the stored documents, allowing the underlying database to calculate the geometric distance between the user's query and the available facts.

The system then performs a rapid similarity search—most often utilizing a metric known as cosine similarity—to find the vectors in the database that most closely match the query vector. The retriever fetches the "top-k" most relevant chunks, typically targeting the 5 to 10 best matches, depending on the specific system's configuration and the complexity of the domain. These retrieved documents form the factual foundation that will strictly constrain the artificial intelligence's eventual answer, making the precision and recall of this search algorithm the most critical factor in preventing downstream hallucinations.[3]

Meta's original research demonstrated this retrieval mechanism at scale, utilizing a massive index of 21 million Wikipedia documents to test the architecture's limits. "Given the prompt 'When did the first mammal appear on Earth?' for instance, RAG might surface documents for 'Mammal,' 'History of Earth,' and 'Evolution of Mammals,'" the Meta AI team wrote in their project breakdown. By surfacing these specific, highly relevant articles, the system guarantees that the subsequent generation phase has immediate access to the exact dates, scientific consensus, and evolutionary timelines required to formulate a factually correct response.[1][2]

Once the relevant documents are successfully retrieved, the third component—generation—takes over the pipeline. The system's integration layer concatenates the original user prompt with the raw text of the retrieved documents, engineering a massive, newly augmented prompt that contains both the user's question and the necessary factual grounding. This combined package is what finally reaches the large language model, fundamentally shifting its operational role from an all-knowing oracle expected to memorize everything to a highly capable synthesizer tasked strictly with summarizing and formatting the provided evidence.

This augmented prompt is then fed directly into the generative language model. Because the model is explicitly instructed via its system prompt to base its answer entirely on the provided context, the resulting output is grounded in the retrieved facts rather than the model's potentially outdated internal memory. Depending on the specific chunk size and top-k configuration, this retrieved grounding data consumes between 1,280 and 10,240 tokens of the model's context window before the user's actual question is even processed. This represents a significant, hard-coded computational overhead that enterprise developers must account for when budgeting inference costs.

This late-fusion approach allows organizations to completely swap out the underlying knowledge base without ever touching the language model's weights. "RAG allows NLP models to bypass the retraining step, accessing and drawing from up-to-date information," the Meta researchers noted, highlighting the architecture's primary economic advantage. If a company policy changes, a product is discontinued, or new scientific research is published, developers simply update the specific entries in the vector database. The generator instantly adapts to the new facts on the very next query, entirely avoiding the need for a multi-million-dollar fine-tuning run.[2]

The fourth and final component of the architecture is evaluation. Because RAG systems are highly dynamic and rely entirely on the quality of external data, traditional deterministic software testing is insufficient to guarantee performance. The evaluation phase continuously measures whether the system is actually retrieving the right documents and generating accurate, strictly grounded answers. Without continuous, automated monitoring, a failure in the retrieval step will inevitably cascade into the generation step, prompting the language model to produce a highly convincing but entirely false output based on irrelevant context.

A comprehensive July 2024 survey published on arXiv detailed the emerging, specialized frameworks currently used to score enterprise RAG pipelines. Metrics like "faithfulness" explicitly measure whether the generated answer is strictly derived from the retrieved context without adding outside hallucinations, while "answer relevance" scores how well the final output actually addresses the user's original query. These automated evaluation tools increasingly use a secondary, highly calibrated language model to grade the primary system's outputs at scale, creating a closed loop of continuous improvement that flags problematic queries for human review.[3]

Without this rigorous evaluation layer, a RAG system can easily fail silently in production environments. If the retrieval component fetches the wrong documents due to poor indexing or vague user queries, the generator will confidently synthesize an incorrect answer—a phenomenon the industry refers to as a grounded hallucination. By strictly isolating the architecture into these four distinct components, developers can pinpoint exactly where the pipeline broke down, allowing them to adjust their chunking strategies, swap their embedding models, or tune their retrieval limits to restore accuracy.