RAG Retrieval Strategies and Reranking

A high-performing RAG system isn't defined by its language model alone; its success hinges on the quality of the information it retrieves. You can have the most powerful generator, but if it's fed irrelevant or incomplete context, the output will be flawed. Advanced strategies and reranking techniques transform a basic document lookup into a precise, intelligent retrieval engine, forming the critical foundation for reliable, high-quality generation.

Foundational Retrieval: Search Strategies

The first step in any RAG pipeline is retrieval—the process of finding the most relevant document chunks or passages from a knowledge base in response to a user query. The strategy you choose here sets the ceiling for your system's potential performance.

Semantic Search is the modern, default approach for most RAG applications. It works by transforming both the query and all document chunks into high-dimensional numerical representations called embeddings. These vectors capture the semantic meaning of the text. Relevance is then calculated by measuring the vector similarity (e.g., cosine similarity) between the query embedding and all chunk embeddings, returning the chunks whose vectors are "closest" to the query's. This excels at understanding intent and conceptual matches, such as retrieving passages about "canine loyalty" when asked "What are the traits of a faithful dog?"

Keyword Search, often implemented via sparse retrieval algorithms like BM25, takes a traditional, lexical approach. It matches the exact terms in the query against terms in the documents, weighting them by frequency and inverse document frequency. It is highly effective for queries containing specific names, codes, or technical terms where precise word matching is crucial. For instance, searching for "React useEffect hook lifecycle" will reliably find documents containing those exact keywords.

Hybrid Search strategically combines the strengths of both semantic and keyword search to overcome their individual weaknesses. A common implementation involves performing both searches independently, normalizing their relevance scores, and then combining them with a weighted sum (e.g., 70% semantic, 30% keyword). This ensures you capture both the conceptual meaning and the precise lexical matches. If a query is "How do I implement a singleton pattern in Python?", hybrid search would find documents discussing design patterns (semantic) while prioritizing those that explicitly mention "singleton" and "Python" (keyword).

Precision Techniques: Reranking Retrieved Results

Initial retrieval often returns a broad set of potentially relevant documents (e.g., the top 100 chunks). Reranking is the computationally intensive but highly rewarding step of re-evaluating this smaller set to push the absolute best candidates to the top before they are passed to the generator.

Cross-Encoder Reranking is the gold standard for precision. Unlike the initial retrieval step which compares embeddings independently (a "bi-encoder" approach), a cross-encoder jointly processes the query and a single candidate document chunk together. This allows for deep, attention-based interaction between every word in the query and every word in the document. The model outputs a single, highly accurate relevance score. While too slow to run over an entire database, applying a cross-encoder to the top 50-100 initial results can dramatically improve the order of the top 5-10 passages fed to the LLM.

Maximum Marginal Relevance (MMR) is a reranking strategy designed to optimize for diversity and reduce redundancy in the retrieved context. The standard similarity-based approach can return several nearly identical passages. MMR balances relevance to the query with novelty relative to already-selected documents. It iteratively selects the next passage that maximizes a score like: $MMR=\lambda \ast Sim(Q,D)-(1-\lambda )\ast ma{x}_{D_i\in S}Sim(D,D_i)$. Here, $Sim(Q,D)$ is relevance to the query, $maxSim(D,D_i)$ is similarity to the most similar passage already in the selected set $S$, and $\lambda$ is a parameter controlling the trade-off. This is invaluable for queries like "Tell me the causes of World War I," ensuring you get context covering diplomatic, military, and economic causes rather than five passages explaining the same single cause.

Enhancing Recall: Improving Retrieval Coverage

Sometimes the correct information exists in your knowledge base but isn't retrieved because the query is poorly phrased, too brief, or phrased in a different "language" than the documents. These techniques aim to boost recall—the system's ability to find all relevant information.

Contextual Compression addresses the problem of long, noisy documents where only a small fraction is relevant. Instead of retrieving and passing an entire chunk, a compression step extracts or summarizes only the parts directly relevant to the query. A "compressor" LLM can take the query and a retrieved document as input and output a concise, focused context. This saves precious context window space and reduces distraction for the final generator.

Query Expansion enriches the original user query to make it more robust for retrieval. Simple techniques involve generating synonyms or related terms. More advanced methods use a language model to produce multiple, rephrased versions of the query. You then perform retrieval for each version and aggregate the results. For a query like "best way to learn guitar," an expanded set might include "effective guitar practice techniques," "beginner guitar tutorials," and "how to master guitar chords," casting a wider net over the document space.

Hypothetical Document Embeddings (HyDE) is a powerful, model-based technique to bridge the vocabulary gap. Given a query, you first instruct an LLM to generate a hypothetical ideal answer or document—even if it's factually fabricated. For "Explain quantum entanglement," the LLM might generate a plausible-sounding physics explanation. You then take the embedding of this hypothetical document and use it for semantic search against your real knowledge base. The hypothesis is written in the fluent, descriptive "language" of your documents, making its embedding a better semantic target than the embedding of the terse, question-formatted original query. This often retrieves more relevant factual passages than searching with the query alone.

Common Pitfalls

Over-Reliance on a Single Search Type. Using only semantic search can miss critical keyword matches, while only keyword search fails at conceptual understanding. Diagnose your query types and default to a well-tuned hybrid approach for robust performance across different questions.

Applying Reranking to Too Many Documents. Cross-encoder reranking is computationally expensive. Applying it to your initial top 500 results will be prohibitively slow. The standard pattern is to use fast, approximate retrieval (semantic/keyword/hybrid) to get a candidate pool of 50-200, then apply the powerful but slow reranker to this pool.

Ignoring the Context Window Budget. Retrieving twenty 500-word chunks will exhaust most LLM context windows, leaving no room for a thorough answer. You must implement a selection or compression strategy—whether via MMR, scoring thresholds, or a compressor—to ensure you send only the most salient, non-redundant information.

Using HyDE Without Validation. The hypothetical document is a guide, not an answer. Never present the HyDE-generated text as the final output. Its sole purpose is to create a better query vector for retrieval. Always retrieve real documents and have the generator ground its final answer solely in that retrieved factual context.

Summary

• Retrieval is multi-faceted: Effective RAG systems leverage semantic search for meaning, keyword search for precision, and hybrid search to combine their strengths for robust initial retrieval.
• Reranking is essential for precision: Cross-encoder models provide a significant final boost to ordering by deeply analyzing query-document pairs, while Maximum Marginal Relevance (MMR) ensures the final context set is diverse and non-redundant.
• Recall can be actively improved: Techniques like contextual compression focus on key information, query expansion broadens the search, and Hypothetical Document Embeddings (HyDE) uses an LLM-generated guide to find documents that better match the query's intent.
• Strategy is context-dependent: There is no single best pipeline. The choice and tuning of these strategies depend on your domain, query types, knowledge base structure, and available computational resources.