Hybrid Retrieval for RAG: Recall Without Losing Precision

VZ Lens

Through a VZ lens, this is not content for trend consumption - it is a decision signal. Vector search alone is rarely enough in production. Hybrid retrieval combines lexical and semantic signals to improve both coverage and answer reliability. The real leverage appears when the insight is translated into explicit operating choices.

TL;DR

Most RAG implementations rely exclusively on embedding-based (dense) search—leaving a well-established, 30-year-old tool on the shelf. BM25 (sparse search) consistently outperforms embeddings for short, precise keyword queries. Hybrid search—where the two signals are combined via RRF (Reciprocal Rank Fusion) re-ranking—approaches the theoretical maximum without requiring a choice. The question isn’t whether to use dense or sparse, but when the complexity of the hybrid pipeline is worth it.

---

The Librarian Who Never Forgot a Word

There’s an image I can’t get out of my head: an old librarian who knows by heart which shelf every book is on. Not because he understood all the content—but because he remembers the exact words, authors, and titles. If you say to him, “Akerlof, 1970, lemons,” he immediately shows you the way.

Embedding-based search, on the other hand, is more like an experienced reader: it understands the meaning of the text and senses the connections, but if you’re looking for an exact term—a part number, a specific law, a rare pronoun—it sometimes loses track. Semantic space doesn’t always reflect lexical precision.

This tension is the basis of what we call hybrid search.

What is the difference between dense and sparse search, and why does it matter?

Dense search embeds a text into a high-dimensional vector space where semantically similar texts are placed close to one another. If you ask, “How does inflation affect consumer confidence?”, the model understands the intent behind the question and retrieves documents that are contextually related, even if they don’t match word for word.

Sparse search (BM25 and its variants), on the other hand, relies on lexical overlap. It counts which words appear in the query and in the document, and weights them based on their frequency within the document and their rarity across the entire corpus. This is a refined descendant of the TF-IDF logic—simple, deterministic, fast.

The two approaches make different kinds of errors:

• Dense search performs poorly with exact keywords, rare concepts, product codes, and names, where embedding cannot establish a reliable connection.
• Sparse search performs poorly with synonyms, paraphrases, and contextual queries where the keywords do not match but the meaning does.

The hybrid strategy is therefore not simply better—it makes different mistakes, and the intersection of the two sets of errors is small.

How does RRF work, and why is it worth building on?

RRF (Reciprocal Rank Fusion) is an elegantly simple aggregation method. Both search systems return a ranked list. RRF calculates a score for each document: the reciprocal sum of its rankings across all lists. If a document achieves a high ranking in both systems, its combined score becomes dominant.

The formula: it adds a small constant (usually 60) to the ranks in each list, then takes the reciprocal and sums the values obtained across all lists. This simple mechanism is extremely robust against outliers—an extremely high dense score does not overwhelm the sparse score if it is weak.

The main advantage of this method is that it requires no training and is not sensitive to the differing score scales of the two systems. There is no need to calibrate how much a cosine similarity of 0.87 is worth compared to a BM25 score of 23—the RRF only looks at the relative order.

When should you build a hybrid model, and when is a pure dense model sufficient?

This is the question where most implementations fall short. The answer depends on the nature of the corpus.

A pure dense model is sufficient if:

• The corpus consists of well-structured texts with a consistent style (e.g., corporate knowledge bases, internal documentation).
• Queries are conceptual, not terminology-specific.
• The embedding model is well-calibrated to the domain (e.g., a finely tuned domain-specific model).

Hybrid is necessary if:

• The corpus is mixed: product descriptions, legal texts, technical glossaries, texts full of names and codes.
• Users search with exact keywords, not just natural language queries.
• The embedding model is general-purpose, and domain-specific terminology was not included in the training data.
• Queries include “needle in a haystack” type searches requiring exact matches.

In my experience, the vast majority of enterprise RAG systems fall into the second category. With documents from a financial services provider containing legal references, ISIN codes, and specific product names, pure dense search systematically misses the most important and precise queries.

The Cost of Complexity: When Is It Better to Keep It Simple?

Every hybrid pipeline increases infrastructure complexity. Two indexes must be maintained, two query paths must be synchronized, and RRF aggregation adds an extra step in latency. If the application is real-time, this matters.

In some cases, there is a simpler alternative: dense-only search with query expansion. The query is paraphrased using the LLM and expanded with multiple semantic variants, which are then searched in parallel. This helps a lot with the synonymy problem, but it doesn’t solve every lexical precision challenge.

The decision logic in brief: if the RAGAS evaluation (context recall, context precision) performed on your corpus shows that the matching errors are keyword-based—that is, the relevant document is in the corpus but does not make it into the top-k—then the hybrid approach brings immediate, measurable improvement. If the errors are conceptual—it is difficult to assess relevance, and the problem lies in the semantic distance between the query and the document—then you need to work on the quality of the embedding model and the chunking strategy, not the search method.

Hybrid search is not a magic solution. But a RAG system that never deals with sparse signals blindly ignores the most searchable dimension of a document collection.

---

Related Thoughts

• Context window vs. RAG — when to use which?
• RAG production pitfalls — what the tutorial doesn’t cover

---

Zoltán Varga - LinkedIn Neural • Knowledge Systems Architect | Enterprise RAG architect PKM • AI Ecosystems | Neural Awareness • Consciousness & Leadership Build for the query you actually get, not the one you wish for.

Strategic Synthesis

• Translate the core idea of “Hybrid Retrieval for RAG: Recall Without Losing Precision” into one concrete operating decision for the next 30 days.
• Define the trust and quality signals you will monitor weekly to validate progress.
• Run a short feedback loop: measure, refine, and re-prioritize based on real outcomes.