Dense vs. Sparse Retrieval

Two support engineers filed bug reports against the same search system on the same afternoon.

The first typed ERR_CONN_5521 — an exact error code that appears in precisely one runbook. The system, which used semantic vector search, returned five documents about network connectivity, TLS handshakes and firewall rules. All thematically adjacent. None of them the runbook. The code itself is a meaningless token to an embedding model; it had no learned meaning, so it contributed almost nothing to the query vector, and the model retrieved on the vague aura of "connection error" instead.

The second engineer typed "customer says the parcel never showed up". The team had, the week before, added a keyword index to fix exactly the first problem. The keyword index returned nothing useful, because the relevant article is titled "Handling non-delivery claims" and uses the words consignment, undelivered and claim. Not one word overlapped with the query.

These are not two bugs. They are the same bug seen from opposite sides: lexical matching and semantic matching fail on exactly the queries the other one handles. Understanding why, precisely, is what lets you build a retriever that survives both engineers.

What a retriever is actually doing

Strip away the implementation and every retriever does the same three things: turn the query into some representation, turn each document into the same kind of representation, and score the pairs. The families differ only in what that representation is.

The name is literal. A sparse vector for a 40-word document has maybe 30 non-zero entries out of 50,000 — 99.94% zeros. A dense vector has no zeros at all.

Sparse retrieval: matching the words themselves

Why raw word counting fails

Start with the naive version and watch it break. Score a document by how many query words it contains. Query: "the refund policy for enterprise contracts". A document containing "the" forty times scores 40 on that word alone and beats the actual refund policy. Two corrections are needed, and TF-IDF is exactly those two corrections.

TF-IDF: rare words carry the signal

Term frequency says a word appearing often in a document matters to that document. Inverse document frequency says a word appearing in almost every document tells you nothing:

With a corpus of N = 10,000 documents: "refund" appears in 500 of them, so its idf is ln(10000/500) = ln(20) = 3.00. "The" appears in 9,900, so its idf is ln(10000/9900) = ln(1.0101) = 0.0100. Three hundred times the weight for the word that actually discriminates. Documents and queries become tf-idf vectors and are compared by cosine similarity.

BM25: what everyone actually uses

TF-IDF has two remaining flaws. Term frequency grows linearly — a document mentioning "refund" thirty times scores thirty times a document mentioning it once, which is absurd; after the fourth or fifth mention you have learned everything you are going to learn. And long documents accumulate more matches simply by being long. BM25 fixes both:

with

Two knobs. k1 (typically 1.2–2.0) controls how fast term frequency saturates. b (typically 0.75) controls how hard long documents are penalised; b = 0 disables length normalisation entirely, b = 1 applies it fully.

Work it through. N = 10,000; "refund" appears in df = 500 documents; the candidate document is 200 words long against an average of 250; k1 = 1.5, b = 0.75.

The IDF: (10000 − 500 + 0.5) / (500 + 0.5) = 9500.5 / 500.5 = 18.982. Add 1 and take the log: ln(19.982) = 2.995.

The length normaliser: 1 − 0.75 + 0.75 × (200/250) = 0.25 + 0.60 = 0.85. So the denominator's constant term is k1 × 0.85 = 1.275.

Now watch saturation do its work as the word appears more often:

Thirty mentions of a word earn only 2.2 times the score of a single mention under BM25, against 30 times under raw TF-IDF. That saturation is why BM25 beat TF-IDF decisively and has stayed the lexical baseline for thirty years.

Strengths and limits

BM25 needs no training, no GPU, and no model to keep in sync. Indexing a million documents takes minutes and the index is a fraction of the size of a dense one. It is exact on identifiers — part numbers, error codes, SKUs, legal citations, surnames — because it matches the literal token. And every score decomposes: you can print which query terms contributed what.

Its limits are the vocabulary mismatch shown at the top. No shared word, no score, full stop. It cannot tell that "car" and "automobile" are the same thing, cannot resolve that "Apple revenue" is about a company rather than fruit, and generally collapses across languages.

1from rank_bm25 import BM25Okapi2import re34def tokenise(text):5    return re.findall(r"[a-z0-9_]+", text.lower())67corpus = [8    "Enterprise annual contracts carry a 14-day refund window.",9    "Error ERR_CONN_5521 indicates a TLS handshake timeout on port 8443.",10    "Non-delivery claims for undelivered consignments must be filed within 30 days.",11]12bm25 = BM25Okapi([tokenise(d) for d in corpus])1314for q in ["ERR_CONN_5521", "parcel never showed up"]:15    scores = bm25.get_scores(tokenise(q))16    best = max(range(len(corpus)), key=lambda i: scores[i])17    print(f"{q!r:30} -> doc {best}  score {scores[best]:.3f}")18# 'ERR_CONN_5521'                -> doc 1  score 0.51819# 'parcel never showed up'       -> doc 0  score 0.000   <-- vocabulary mismatch

The absolute scores are small because this corpus has three documents, so IDF has little to work with. What matters is the second line: every document scores zero, and "doc 0" is only the first of three ties.

Dense retrieval: matching what the words mean

The core idea

A neural network reads a piece of text and outputs a fixed-length vector, trained so that texts with similar meaning land close together. The training objective is contrastive: show the model a query with its correct passage (a positive) and a batch of wrong passages (negatives), and adjust the weights to pull the positive pair together and push the negatives apart. Repeat over millions of pairs and the geometry of the space starts to encode meaning.

The payoff is that "customer says the parcel never showed up" and "Handling non-delivery claims for undelivered consignments" end up with a cosine similarity around 0.78 despite sharing not one content word. The model learned during training that these phrasings co-occur with the same answers.

The architecture that makes it fast

Dense retrieval uses a bi-encoder: the query and the document are encoded separately, never together. This is the entire reason it scales. Because documents are encoded independently of any query, you embed the whole corpus once, offline, and store the vectors. At query time you embed one short string and do a nearest-neighbour search. Searching 10 million vectors with an HNSW index takes single-digit milliseconds.

The bi-encoder's separation of query and document is simultaneously its superpower and its ceiling: it is fast because the document vector cannot depend on the query, and it is imprecise for the same reason.

Choosing a model

Dimensionality is a storage decision as much as a quality one. One million chunks at 384 dimensions in float32 is 1,000,000 × 384 × 4 = 1.54 GB. At 1,536 dimensions it is 6.14 GB — four times the memory for a few points of benchmark accuracy. That trade is often not worth it.

1import numpy as np, faiss2from sentence_transformers import SentenceTransformer34model = SentenceTransformer("BAAI/bge-small-en-v1.5")5Q_PREFIX = "Represent this sentence for searching relevant passages: "67doc_vecs = model.encode(corpus, normalize_embeddings=True).astype("float32")8index = faiss.IndexFlatIP(doc_vecs.shape[1])   # normalised + inner product = cosine9index.add(doc_vecs)1011def dense_search(query, k=3):12    qv = model.encode([Q_PREFIX + query],13                      normalize_embeddings=True).astype("float32")14    scores, ids = index.search(qv, k)15    return [(int(i), float(s)) for i, s in zip(ids[0], scores[0])]1617print(dense_search("parcel never showed up"))   # finds the non-delivery doc18print(dense_search("ERR_CONN_5521"))            # often does NOT rank the runbook first

Where dense retrieval fails

Exact identifiers, as we have seen — rare tokens the model never learned. Numbers, which embeddings represent poorly: "contracts over 50,000 units" and "contracts over 500 units" sit almost on top of each other in vector space. Negation, which frequently barely moves the vector: "documents that are not confidential" can retrieve confidential documents. New jargon coined after the model was trained. And out-of-domain corpora, where a model trained on general web text has never seen your ontology.

Head to head

Hybrid retrieval: use both, properly

The failure profiles are close to complementary. Run both retrievers and merge. In practice hybrid beats either component on virtually every realistic query mix, and the gain is largest exactly where systems hurt most — the long tail of queries that mix a concept with an identifier, like "why does ERR_CONN_5521 happen on renewal".

Approach 1 — Weighted score fusion, and the trap in it

The obvious merge is a weighted sum: 0.5 × dense + 0.5 × bm25. This is broken as written, and the reason is worth internalising. Cosine similarity lives on a bounded scale of roughly 0 to 1. BM25 is unbounded and routinely reaches 15 or 30 depending on query length and corpus statistics. Add them directly and BM25 swamps the dense score entirely; your "hybrid" system is a keyword system with rounding noise.

Normalise each list first — min-max within the result set is the usual approach:

1def minmax(scores):2    lo, hi = min(scores), max(scores)3    if hi - lo < 1e-9:4        return [1.0] * len(scores)5    return [(s - lo) / (hi - lo) for s in scores]67def weighted_hybrid(dense_hits, sparse_hits, alpha=0.6):8    d = dict(zip([i for i, _ in dense_hits],9                 minmax([s for _, s in dense_hits])))10    s = dict(zip([i for i, _ in sparse_hits],11                 minmax([sc for _, sc in sparse_hits])))12    merged = {i: alpha * d.get(i, 0.0) + (1 - alpha) * s.get(i, 0.0)13              for i in set(d) | set(s)}14    return sorted(merged.items(), key=lambda kv: -kv[1])

Even normalised, this is fragile: min-max depends on which documents happened to be in each list, so the same document can get a different normalised score depending on its neighbours.

Approach 2 — Reciprocal Rank Fusion

RRF sidesteps the whole problem by throwing away the scores and keeping only the ranks:

Ranks are comparable across systems by construction. The constant k = 60 damps the influence of the very top positions so that one ranker cannot dominate on its own.

A worked fusion. Four documents, two rankers:

Read what happened. Document A was first on BM25 and still lost to B, because B was strong on both rankers while A was mediocre on one. Document C was second on BM25 but absent from the dense list entirely, and it collapsed to last. That is exactly the behaviour you want: agreement between independent rankers is evidence, and a single ranker's enthusiasm is not.

1from collections import defaultdict23def rrf(rankings, k=60, top_n=10):4    """rankings: list of lists of doc ids, each already sorted best-first."""5    scores = defaultdict(float)6    for ranking in rankings:7        for rank, doc_id in enumerate(ranking, start=1):8            scores[doc_id] += 1.0 / (k + rank)9    return sorted(scores.items(), key=lambda kv: -kv[1])[:top_n]1011fused = rrf([bm25_ranking, dense_ranking])

Approach 3 — Learned re-ranking on top

Both fusion methods are heuristics; neither reads the documents. The strongest hybrid uses fusion only to build a candidate pool, then scores that pool with a cross-encoder — a model that takes the query and one passage together and outputs a relevance score. Because it can attend across both texts jointly, it catches relationships neither BM25 nor a bi-encoder can see.

It is also roughly a thousand times slower per pair, which is precisely why it must run on 50 candidates and not on 61,000 documents.

The full hybrid

1from sentence_transformers import CrossEncoder23reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")45def hybrid_search(query, corpus, k_recall=50, k_final=5):6    dense_ids  = [i for i, _ in dense_search(query, k=k_recall)]7    sparse_ids = list(np.argsort(bm25.get_scores(tokenise(query)))[::-1][:k_recall])89    candidates = [doc_id for doc_id, _ in rrf([dense_ids, sparse_ids],10                                              top_n=k_recall)]11    pairs  = [(query, corpus[i]) for i in candidates]12    scores = reranker.predict(pairs)13    order  = np.argsort(scores)[::-1][:k_final]14    return [(candidates[j], float(scores[j])) for j in order]

Choosing, and staging, in production

The production shape almost everyone converges on is a funnel, and it is worth writing down with its budgets:

Stage 0   filter by metadata (tenant, status, date, access)   ~1 ms          10,000,000 -> 400,000 eligible documentsStage 1   BM25 top-100  +  dense top-100, fused by RRF        ~25 ms          400,000 -> 100 candidates          objective: RECALLStage 2   cross-encoder re-ranks the 100                      ~90 ms          100 -> 8 passages                  objective: PRECISIONStage 3   language model reads the 8                       ~1,300 ms          8 -> one grounded answer

Each stage is cheap per item and expensive per item in the opposite direction to the stage before it. Stage 1 touches hundreds of thousands of documents at nanoseconds each; stage 3 touches eight at hundreds of milliseconds each. Getting this ordering wrong — running the cross-encoder over the whole corpus, or filtering after retrieval instead of before — is how RAG systems end up costing a fortune and timing out.

What this means when you build one

Build BM25 first, even if you intend to be a dense shop. It takes twenty minutes, needs no GPU, and gives you the number that matters: the baseline your dense system must beat. Teams that skip this step routinely deploy an embedding pipeline that is worse than a keyword index on their domain and never find out, because they have nothing to compare against. On specialised corpora with heavy jargon, out-of-the-box dense retrieval losing to BM25 is common, not exotic.

Then look at your query log rather than guessing. Sample 200 real queries and mark each one: does it contain an identifier, a code, a proper noun, an exact number? Or is it a conceptual, colloquial phrasing? The ratio tells you where to invest. A log dominated by part numbers does not need a better embedding model; it needs BM25 weighted higher. A log full of "how do I..." questions does.

You may not have to build the fusion yourself. Several search engines and vector databases — Elasticsearch, OpenSearch, Weaviate and Qdrant among them — now run keyword and vector search together and fuse the lists, often with RRF, in a single query. Hybrid search is the normal production default rather than an advanced option.

When you do go hybrid, prefer RRF as the default fusion. It has one parameter, it needs no score calibration, it degrades gracefully when one retriever returns nothing, and it cannot be broken by a corpus change that shifts BM25's score distribution. Weighted fusion can beat it after careful tuning on your data — but that tuning has to be redone whenever the corpus changes materially, and in most teams it silently is not.

Finally, remember what the funnel is protecting. Every stage before the language model exists to make sure that the eight passages the model reads are the right eight. Effort spent there compounds; effort spent rewording the prompt while the wrong passages arrive does not.