Semantic Search & Retrieval Pipelines

An electronics retailer replaces its keyword search with a vector index. The demo is convincing: a shopper types "something to keep my laptop cool" and gets cooling pads and stands, none of which share a word with the query. The old search returned nothing at all. Ship it.

Three weeks in, the support queue fills with a single complaint. Customers typing the exact model number XR-4400 — printed on the box, on the invoice, on the warranty card — get five vaguely related monitors and not the XR-4400. Meanwhile the head of compliance discovers that a search for "internal pricing" from a customer account is returning staff-only documents, because relevance ranking pushed them into the top five and nothing checked whether the user was allowed to see them.

Neither failure is a bug in the index. The index did exactly what it was asked: return the five nearest vectors. The problem is that "return the nearest vectors" is one stage of a search system, and this one had no other stages. Embeddings blur precise tokens — an alphanumeric SKU carries almost no semantic signal, so XR-4400 lands somewhere in the neighbourhood of "product identifier"-shaped strings rather than on the exact item. And a similarity score has no concept of permission.

A vector index answers "what is most similar?". A search system has to answer "what is most useful, that this person is allowed to see, right now" — and those are different questions.

The stages between a keystroke and a result

  raw user query        |  [1] normalise        trim, collapse whitespace, enforce token limit        |  [2] encode           SAME model that built the index - no exceptions        |  [3] retrieve         vector search -> top 50-200 candidates (optimise RECALL)        |  [4] fuse             blend with keyword scores for exact-match queries        |  [5] filter           hard constraints: permissions, stock, date, tenant        |  [6] rerank           cross-encoder over survivors (optimise PRECISION)        |  [7] format           scores, snippets, highlights, empty-state handling        |  final results

The single most important structural idea here is that stages 3 and 6 optimise opposite things. Retrieval is cheap per document and slightly blunt, so you cast a wide net and care only about not missing anything. Reranking is expensive per document and very sharp, so you run it over a handful of survivors and care only about getting the order right. Every production search system worth the name has this shape.

Stage 1: normalisation, and the trap inside it

Normalisation sounds trivial and is where a surprising amount of quality leaks away. Two rules matter.

Do not lowercase, and do not strip punctuation. Most modern sentence encoders are trained on natural, cased, punctuated text, so lowercasing "Apple" to "apple" destroys a signal the model was trained to use. (A few, including all-MiniLM-L6-v2, lowercase inside their own tokenizer; for them your lowercasing simply does nothing.) Stripping punctuation turns "XR-4400" into "XR 4400" and "don't" into "dont", both of which tokenise differently and embed differently. The aggressive normalisation that helped keyword search actively harms embedding search.

Do enforce the token limit, loudly. This is the trap. all-MiniLM-L6-v2 has max_seq_length = 256 word-piece tokens, roughly 190 English words. Feed it a 900-word document and it encodes the first 256 tokens and silently discards the rest. No warning, no exception. You get a perfectly valid 384-dimensional vector representing the first quarter of the document, and every search against the other three quarters fails for reasons no log will ever show.

1import re23def normalise(text: str, tokenizer, max_tokens: int = 256) -> str:4    text = re.sub(r"\s+", " ", text).strip()      # collapse whitespace only5    n = len(tokenizer.tokenize(text)) + 2         # + [CLS] and [SEP]6    if n > max_tokens:7        raise ValueError(f"{n} tokens exceeds limit {max_tokens} - chunk this first")8    return text910print(model.max_seq_length)      # 256 - check this for YOUR model, it varies

Raising an error rather than truncating forces the real decision: split long documents into overlapping chunks and index each one separately. Truncation is not a smaller version of chunking. It is data loss with a plausible-looking vector on top.

Assembling the retrieval core

1import faiss2import numpy as np3from sentence_transformers import SentenceTransformer45class SemanticSearch:6    def __init__(self, model_name="all-MiniLM-L6-v2"):7        self.model_name = model_name              # store it - the index depends on it8        self.model = SentenceTransformer(model_name)9        self.documents, self.metadata, self.index = [], [], None1011    def index_documents(self, documents, metadata=None):12        self.documents = documents13        self.metadata = metadata or [{} for _ in documents]14        emb = self.model.encode(15            documents, batch_size=64, normalize_embeddings=True,16            show_progress_bar=True,17        ).astype("float32")18        self.index = faiss.IndexFlatIP(emb.shape[1])   # IP on unit vectors = cosine19        self.index.add(emb)2021    def retrieve(self, query, top_k=50):22        q = self.model.encode(query, normalize_embeddings=True)23        scores, ids = self.index.search(q.astype("float32").reshape(1, -1), top_k)24        return [25            {"doc": self.documents[i], "meta": self.metadata[i],26             "id": int(i), "score": float(s), "rank": r + 1}27            for r, (i, s) in enumerate(zip(ids[0], scores[0])) if i >= 028        ]

Three deliberate choices in that class. Normalising at encode time and using IndexFlatIP means the returned numbers are cosine similarities already, running from 0 to 1 with higher meaning better — no conversion, no chance of reading a distance backwards. The if i >= 0 guard matters because FAISS pads short result sets with -1, and self.documents[-1] quietly returns the last document instead of raising. And top_k defaults to 50, not 5, because this is the recall stage: it feeds later stages rather than the user.

Retrieve broadly, rerank narrowly

The model that built your index is a bi-encoder: it encodes the query and each document separately, then compares the two vectors. That separation is precisely what makes search fast — every document vector is computed once, offline, and reused forever. It is also the ceiling on precision, because the model never sees the query and the document at the same time. It cannot notice that the query asks about refunds after 30 days and the document is about refunds within 30 days.

A cross-encoder does the opposite. It concatenates query and document into one input, runs a full transformer over the pair, and outputs a single relevance score. It can attend to the word "after" against the word "within". It is also unusable at scale, for a simple reason: nothing can be precomputed. Every (query, document) pair needs its own forward pass.

That table contains the entire argument. Reranking one million documents costs twenty minutes; reranking the fifty candidates the bi-encoder already found costs sixty milliseconds. (The throughput figures are for a typical CPU and passages of about 100 words; measure your own.) For a sense of the quality gain, the Sentence Transformers model tables report MRR@10 on the MS MARCO dev set of 32.3 for a MiniLM-L6 bi-encoder (msmarco-MiniLM-L6-v3) and 39.0 for the reranker cross-encoder/ms-marco-MiniLM-L6-v2. The two were measured in different set-ups, so read the gap as a rough size, not a promise.

1from sentence_transformers import CrossEncoder23ranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L6-v2")    # load ONCE, at startup45def rerank(query, candidates, top_k=5):6    pairs = [[query, c["doc"]] for c in candidates]7    scores = ranker.predict(pairs, batch_size=32)8    for c, s in zip(candidates, scores):9        c["rerank_score"] = float(s)10    return sorted(candidates, key=lambda c: -c["rerank_score"])[:top_k]

Named failure mode: constructing the CrossEncoder inside the request handler. Loading the model takes 1–3 seconds; doing it per query turns a 60 ms rerank into a 2-second request and looks, from the outside, like "the reranker is slow". Load it once at process start.

Reranking cannot recover what retrieval missed

This is the constraint people most often miss. A reranker only reorders the candidate list it was handed. If the right document sat at rank 240 and you retrieved 50, no reranker will ever find it. Your recall@candidate-depth is a hard ceiling on final quality, so it is the number to measure first.

Those recall figures come from a specific corpus and you must measure your own, but the shape holds everywhere: recall climbs steeply then flattens, latency climbs linearly and never flattens. Somewhere around 50 to 100 candidates the two curves cross.

Hybrid search: giving exact matches a route back in

Return to XR-4400. An embedding model has seen that token roughly never during training, so it decomposes into subword pieces carrying almost no meaning, and the resulting vector points into a bland region of the space. Semantic search genuinely cannot solve this. Keyword search solves it trivially, because exactly three documents in the corpus contain that string.

BM25 is the standard keyword ranking function, and it is worth understanding rather than importing blindly:

where $f(t,D)$ counts occurrences of term $t$ in document $D$, $|D|$ is the document length, avgdl the average length, and $k_1$ and $b$ are tuning constants. $b = 0.75$ is standard everywhere; for $k_1$, Lucene and Elasticsearch default to 1.2 and the Python rank_bm25 package to 1.5, which is what this lesson uses. The inverse document frequency is

This is the Lucene form. rank_bm25's BM25Okapi leaves out the $+1$, so a word found in more than half the documents gets a negative IDF, which the package replaces with a small positive floor. The rare-word numbers below barely change; common words score slightly differently.

Work it on the retailer's catalogue: $N = 1000$ products, average description length 100 words.

Why rare tokens dominate

For XR-4400, appearing in $n_t = 3$ documents:

For the word "the", appearing in 950 documents:

The rare identifier is weighted 109 times more heavily than the common word. This is exactly the property embeddings lack, and exactly why blending the two recovers the failing case.

Frequency saturates, length penalises

Take a product page containing XR-4400 twice, 80 words long. The length term is $k_1(1 - b + b|D|/\text{avgdl}) = 1.5(0.25 + 0.75 \times 0.8) = 1.5 \times 0.85 = 1.275$, so

That ceiling is the anti-keyword-stuffing property: repeating a term 100 times instead of 10 buys almost nothing. And length normalisation bites too — the same two occurrences in a 400-word description give $1.5(0.25 + 3.0) = 4.875$, so the score falls to $5.656 \times 5/6.875 = 4.11$, less than half. A term is more meaningful in a short document.

Fusing the two score lists

Cosine similarities live in 0 to 1. BM25 scores are unbounded and, as just shown, routinely reach 14. Adding them directly means BM25 wins every time. The naive fix is min-max normalisation followed by a weighted blend:

1import numpy as np2from rank_bm25 import BM25Okapi34def minmax(x):5    x = np.asarray(x, dtype="float64")6    span = x.max() - x.min()7    return np.zeros_like(x) if span == 0 else (x - x.min()) / span89def blend(sem_scores, bm25_scores, alpha=0.6):10    return alpha * minmax(sem_scores) + (1 - alpha) * minmax(bm25_scores)

It works, and it has a real weakness: min-max normalisation is defined by the single highest score, so one outlier compresses everything else towards zero. With BM25 scores of [8.64, 0.05, 0.02, 0, 0], normalisation gives [1.0, 0.006, 0.002, 0, 0] — the second and third documents are now indistinguishable from irrelevant ones, even though they did match something.

Reciprocal Rank Fusion avoids the problem by discarding scores entirely and using only ranks:

1def rrf(rank_lists, k=60, top_n=10):2    """rank_lists: list of lists of doc ids, each already ordered best-first."""3    scores = {}4    for ranking in rank_lists:5        for position, doc_id in enumerate(ranking, start=1):6            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + position)7    return sorted(scores, key=scores.get, reverse=True)[:top_n]

Three documents, worked out:

Document X was the top semantic hit and still finishes third, because RRF rewards agreement between retrievers over a single strong opinion. The constant $k = 60$ flattens the difference between ranks 1 and 2 so that no single system can dominate on its own. RRF needs no tuning, no score calibration and no per-corpus alpha, which is why it has quietly become the default fusion method in production search.

Weighted blending needs a tuned alpha for every corpus and query mix; rank fusion needs nothing. Start with RRF, and only reach for a weighted blend when you have measurements showing one retriever really should count more than the other.

Filtering: correctness, not relevance

The compliance failure at the start has a precise cause: the filter was applied in application code, after retrieval.

1# WRONG - post-filter2hits = collection.query(query_texts=[q], n_results=10)3visible = [d for d, m in zip(hits["documents"][0], hits["metadatas"][0])4           if m["visibility"] == "public"]        # often ends up with 2 of 1056# RIGHT - pre-filter, evaluated inside the query7hits = collection.query(8    query_texts=[q],9    n_results=10,10    where={"$and": [{"visibility": {"$eq": "public"}},11                   {"region": {"$in": ["EU", "UK"]}}]},12)

Post-filtering produces two distinct failures. The visible one is result starvation: ask for 10, discard 8 as ineligible, show 2, while thousands of eligible documents sit unexamined further down the ranking. The user sees a nearly empty page and concludes search is broken. The dangerous one is that the retrieval, the logs and often the response payload all touched documents this user should never have been near. A permission check that runs after the data has been fetched is not a permission boundary.

Anything that determines whether a user is allowed to see a document belongs in the query, not in a list comprehension after it. Relevance ranking is not an access control mechanism.

One honest caveat: pre-filtering over a graph index is not free. If a filter matches only 40 documents out of two million, the index has very few eligible neighbours to traverse and recall degrades, or the engine falls back to scanning. When filters are that selective, resolve the eligible set through a conventional database index first and run vector search only over what survives.

Where this leads: grounding a language model

question -> [retrieve top-k passages] -> [insert into prompt] -> [LLM] -> answer

Retrieval-augmented generation is this pipeline with a generation step bolted on the end: fetch relevant passages, paste them into a prompt as context, and ask a language model to answer using only that context. The appeal is that answers become grounded in your documents rather than in whatever the model absorbed during training.

The number that governs whether it works is retrieval recall. If recall@5 is 0.70, then in 30% of requests the passage containing the answer was never retrieved — and a language model handed five irrelevant passages does not say "I could not find this". It writes a fluent, confident, wrong answer from whatever it was given. Retrieval quality is a hard ceiling on answer quality, and no amount of prompt engineering raises it.

Which means every technique above is a RAG improvement. Reranking puts the passage that actually contains the answer at position 1 rather than 7, where a context window might cut it off. Hybrid retrieval rescues questions containing an error code or a version number. Pre-filtering stops the model from citing a document the asker was never permitted to see. Fix retrieval first; measure it in isolation, with recall@k on questions whose answers you already know, before adding a generation step that will disguise the failures.

Making it survive production

Caching, and the bug that lives inside it

Real query traffic is heavily repetitive — a modest LRU cache over a support search typically absorbs 55–70% of requests. The arithmetic is worth doing: at a 60% hit rate with a 0.4 ms cached response and a 65 ms uncached one, mean latency becomes $0.6 \times 0.4 + 0.4 \times 65 = 26.2$ ms, a 2.5× improvement, and the reranker load drops by 60%.

1from functools import lru_cache2import hashlib, json34def cache_key(query, top_k, filters, user_scopes):5    payload = json.dumps(6        {"q": query.strip(), "k": top_k, "f": filters,7         "s": sorted(user_scopes)},         # NEVER omit this8        sort_keys=True,9    )10    return hashlib.sha256(payload.encode()).hexdigest()

Named failure mode, and it is a serious one: keying the cache on the query text alone. Two users type "quarterly revenue". The first is a finance administrator whose filtered search legitimately returns internal figures; the result is cached under the key "quarterly revenue". The second is a contractor, hits the same key, and is served the administrator's results — filters bypassed entirely, because the cache answered before any filter ran. Every input that changes what a user may see must be part of the key. Add a TTL as well, or edited documents keep serving their old text indefinitely.

What to log, so that failures are diagnosable

The most useful single metric a search system can emit is its zero-result rate: the share of queries returning nothing. It is a direct measure of users hitting a dead end, and it moves before anyone files a complaint. But the aggregate only tells you that something is wrong; the per-stage counts tell you what.

1import time, logging2log = logging.getLogger("search")34def search(query, top_k=5, filters=None, scopes=()):5    t0 = time.perf_counter()6    candidates = retriever.retrieve(normalise(query), top_k=50)7    t1 = time.perf_counter()8    eligible = apply_filters(candidates, filters, scopes)9    t2 = time.perf_counter()10    final = rerank(query, eligible, top_k=top_k) if eligible else []11    t3 = time.perf_counter()1213    log.info(14        "search",15        extra={"query": query, "n_retrieved": len(candidates),16               "n_eligible": len(eligible), "n_returned": len(final),17               "top_score": final[0]["rerank_score"] if final else None,18               "ms_retrieve": round((t1 - t0) * 1000, 1),19               "ms_filter": round((t2 - t1) * 1000, 1),20               "ms_rerank": round((t3 - t2) * 1000, 1),21               "zero_result": not final},22    )23    return final

Those three counts diagnose an empty result set on their own. n_retrieved = 0 means the index is empty or the query failed to encode. n_retrieved = 50, n_eligible = 0 means the filter is too tight — a permission scope, a date range, a category that no longer exists. n_eligible = 50, n_returned = 0 means your relevance threshold is rejecting everything. Without the intermediate counts, all three look identical in the logs, and you will spend an afternoon guessing.

What this means when you build something

Build the pipeline in the order that exposes problems earliest. Retrieval first, and measure recall@50 against fifty real queries with hand-labelled answers before writing anything else. That single number caps everything downstream, and if it is 0.6 no reranker, no fusion strategy and no prompt will save you — the fix is in chunking, or the model, or the data. Add reranking second, because it is the largest precision gain for the least code. Add hybrid retrieval when your logs show queries containing identifiers, and not before.

Draw a hard line between relevance and eligibility, and put it in the schema. Relevance is a score, it is fuzzy, and it belongs to the ranking stages. Eligibility is a boolean, it is not negotiable, and it belongs in the query and in the cache key. Teams that blur the two end up relying on the ranker to hide documents, which works right up until the day a query happens to rank one highly.

Instrument the stage boundaries from day one, not after the first incident. Counts in and out of each stage, timing per stage, and a zero-result flag cost about ten lines and turn every future "search is broken for this query" report into a two-minute investigation instead of a reproduction attempt against a corpus that has since changed.

And keep a small evaluation set — fifty to two hundred queries with known correct answers — in version control next to the code. Every parameter in this pipeline (candidate depth, fusion constant, rerank cut-off, similarity threshold) is a number someone will eventually want to change. With an evaluation set, changing one is an experiment with a result. Without it, it is a matter of opinion, and search quality drifts in whatever direction the last opinion pointed.