Hybrid search and reranking

After re-chunking, recall@5 was 0.76. The team read the remaining misses and found a pattern. "How do I submit Form 12BB?" returned four chunks about tax declarations in general, but not the one chunk that actually says "Form 12BB". "What's the ESPP lock-in period?" returned chunks about bonuses and share options. "VPN error 809 on my laptop" returned general VPN setup steps, not the troubleshooting section that lists error 809.

Embedding models are trained to capture meaning, and they are very good at matching "carry forward leave" with "unused days roll over". The same training makes them blur rare, exact tokens: form numbers, product names, error codes, acronyms. To a semantic model, "Form 12BB" and "Form 16" are both "a tax form". To an employee, they are completely different things.

This lesson adds the older, simpler tool that does not blur them, combines the two rankings, and then adds a second-stage model that reads each candidate properly before choosing the final five.

Two kinds of similarity

BM25 is a classic ranking formula from search engines. It scores a chunk higher when it contains the query's words, especially rare words, and adjusts for chunk length. It has no idea what words mean, which is exactly why it complements embeddings.

The rank_bm25 library implements it in a few lines. The tokeniser matters more than the formula: PolicyPal lowercases text and keeps letters and digits together, so "12BB", "L4" and "809" survive as searchable tokens.

1# policypal/keyword.py2import re34import numpy as np5from rank_bm25 import BM25Okapi67TOKEN = re.compile(r"[a-z0-9]+")89def tokenize(text: str) -> list[str]:10    return TOKEN.findall(text.lower())1112class KeywordIndex:13    def __init__(self, chunks: list[dict]):14        self.bm25 = BM25Okapi([tokenize(c["text"]) for c in chunks])1516    def rank(self, question: str, allowed: np.ndarray, n: int = 40) -> list[int]:17        scores = self.bm25.get_scores(tokenize(question))18        scores = np.where(allowed, scores, -np.inf)19        top = np.argsort(-scores)[:n]20        return [int(i) for i in top if scores[i] > 0]

On its own, BM25 scored 0.69 recall@5 on the check set, lower than dense search's 0.76. But it found the Form 12BB, ESPP and error 809 chunks that dense search missed. The two methods fail on different questions, which is the ideal situation for combining them.

Fusing two rankings with reciprocal rank fusion

You cannot simply add a BM25 score to a cosine similarity. BM25 scores can be 0 to 30; cosine similarities sit between about 0.3 and 0.9. Their scales mean nothing to each other. Reciprocal rank fusion (RRF) sidesteps the problem by ignoring scores and using only positions.

1# policypal/fusion.py2from collections import defaultdict34def rrf(rankings: list[list[int]], k: int = 60) -> list[int]:5    """Each ranking is a list of chunk ids, best first."""6    score: dict[int, float] = defaultdict(float)7    for ranking in rankings:8        for position, chunk_id in enumerate(ranking, start=1):9            score[chunk_id] += 1.0 / (k + position)10    return sorted(score, key=score.get, reverse=True)

A chunk ranked first by one method gets 1/61; ranked tenth, it gets 1/70. A chunk that appears high in both lists collects two contributions and rises to the top. A chunk that only one method finds can still make the list if it ranks well there. The constant 60 comes from the original paper and works well in practice; it keeps the top few positions from dominating completely.

Hybrid search with RRF scored 0.84 recall@5. It is also cheap: BM25 over 6,200 chunks takes about 20 milliseconds, and the fusion takes microseconds.

Reranking with a cross-encoder

Both retrievers so far judge relevance quickly and roughly. The embedding model encodes the question and each chunk separately and compares the vectors; it never reads them together. A cross-encoder reads the question and one chunk together, as one input, and outputs a relevance score. That is much more accurate, and much slower, because it must run once per candidate. So you use it as a second stage: retrieve 40 candidates cheaply, then let the cross-encoder pick the best five.

1# policypal/retrieve.py  (version 2: hybrid + rerank)2import numpy as np3from sentence_transformers import CrossEncoder45from policypal.fusion import rrf67reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")89def retrieve(question: str, country: str, dense, keyword, chunks: list[dict],10             k: int = 5, pool: int = 40) -> list[dict]:11    allowed = np.isin(dense.country, [country, "Global"])12    candidates = rrf([dense.rank(question, allowed, n=pool),13                      keyword.rank(question, allowed, n=pool)])[:pool]14    scores = reranker.predict([(question, chunks[i]["text"]) for i in candidates])15    best = np.argsort(-scores)[:k]16    return [chunks[candidates[j]] | {"rerank": float(scores[j])} for j in best]

dense.rank is the dense search from the first lesson, changed to take the allowed mask and return chunk ids instead of chunk dictionaries. The reranker is a small model, about 22 million parameters, trained on search relevance data. Scoring 40 pairs takes about 150 milliseconds on a 4-core CPU, or around 15 on a GPU. With it, recall@5 rose to 0.91.

What each stage bought, and what it cost

One more number explains the ceiling. The hybrid top 40 contained a correct chunk for 97% of questions. The reranker can only reorder what it is given, so 0.97 is the best recall@5 this design could reach. To go higher, you would improve the first stage, not the reranker.

Is 150 milliseconds worth it? For PolicyPal, clearly: the model then takes 4 seconds to write the answer, and seven more questions out of every hundred now have the right passage. It would be a harder call for autocomplete-style search, where the whole budget might be 100 milliseconds. There are also cases where you can skip reranking: very small corpora, or when the first stage is already near its ceiling. Larger rerankers, such as multilingual ones that handle Hindi and Hinglish, are more accurate and several times slower, so measure them on your own traffic before switching.