Semantic search: what embeddings find, and what they miss

Somebody types laptop won't charge into your support search. The document that answers them is titled Notebook battery not powering after sleep. A keyword index compares those two strings and finds nothing in common — not a single shared term. The answer is sitting in the corpus, correctly written and correctly tagged, and the search returns nothing useful. The user concludes you have no documentation about charging.

That gap between the words a person uses and the words an author used is the entire reason semantic search exists. It is also where teams overcorrect: they rip out lexical search, drop in a vector index, and discover that laptop won't charge now works beautifully while error TS2345 returns twelve articles about TypeScript in general and nothing about that error. Good retrieval is not a choice between the two approaches. It is a stack — a lexical scorer, a vector scorer, a fusion step, an approximate index to keep it fast, a reranker to clean up the top of the list, and a small set of judged queries that tells you whether any of it is actually working. This post walks through each layer and what it costs you.

The vocabulary gap, and the tricks that don't close it

Classic search builds an inverted index. Documents are tokenised into terms, and for each term the engine stores a posting list of the documents containing it. At query time it tokenises your query, pulls the posting lists for those terms, and scores only the documents that appear in them. The whole design is fast because it never looks at documents that share no words with the query — which is also precisely why it cannot find the notebook-battery article.

The standard patches help a little. Stemming collapses charge, charging, and charged to a common root, so morphology stops being a problem. A synonym dictionary can map laptop to notebook — as long as somebody maintains that dictionary for every product line, every language, and every new term the market invents. Query expansion goes further and adds related terms automatically, which raises recall and drags in noise: expand charge generously and you will start matching billing documents about charges and fees.

The limitation is structural, not a tuning problem. The keys in an inverted index are strings, and strings have no notion of being near each other. Any nearness has to be supplied by hand, term by term.

BM25 is a stronger baseline than its reputation

Before replacing lexical search it is worth knowing what you are giving up, because BM25 is not a naive word counter. It scores a document by summing a contribution per query term, and each contribution combines three ideas: how often the term appears in this document (term frequency), how rare the term is across the corpus (inverse document frequency), and how long the document is compared to average, so a sprawling page doesn't win just by containing more words.

The part people underrate is saturation. Term frequency does not scale linearly — the fifth occurrence of a word adds far less than the second. This snippet computes a single term's contribution as its frequency rises, holding everything else fixed.

import mathdef bm25_term(tf, df, N, dl, avgdl, k1=1.2, b=0.75):    idf  = math.log(1 + (N - df + 0.5) / (df + 0.5))    norm = k1 * (1 - b + b * dl / avgdl)    return idf * (tf * (k1 + 1)) / (tf + norm)N, avgdl = 100_000, 250          # corpus size, mean document lengthfor tf in (1, 2, 5, 20):    score = bm25_term(tf, df=500, N=N, dl=250, avgdl=avgdl)    print(tf, round(score, 2))

The scores come out at 5.3, 7.28, 9.4 and 10.99. Twenty occurrences are worth about twice one occurrence, not twenty times, and the contribution can never exceed idf * (k1 + 1) — about 11.65 here. That ceiling is what stops keyword stuffing from dominating a ranking. Note the other half too: idf rewards rarity, so a token appearing in 3 documents out of 100,000 carries enormous weight. That is exactly the behaviour you want for a part number or an error code, and it is the behaviour embeddings are worst at.

BM25 is the default scorer in Lucene, Elasticsearch and OpenSearch. It needs no GPU, no model version to track, and no reindexing when a vendor deprecates something. Skipping it and going straight to vectors usually means reinventing it six months later under the name "hybrid search".

Text as coordinates

An embedding model takes a piece of text and returns a fixed-length list of floating point numbers — commonly 384, 768, or 1536 of them. The model is trained on pairs of texts that people treat as related (a question and its answer, a title and its body, a query and the document someone clicked) with an objective that pulls related pairs together and pushes unrelated ones apart. After training, the position of a text in that space encodes what it is about.

Individual dimensions mean nothing you can name. The geometry is what carries information: laptop won't charge and notebook battery not powering land close together because the model saw thousands of examples where those two phrasings served the same purpose, while carburettor rebuild kit lands far away. None of that requires shared words.

Closeness is almost always measured with cosine similarity: the dot product of two vectors divided by the product of their lengths. The dot product measures how much two vectors point the same way; dividing by the lengths removes magnitude, so a long verbose document isn't automatically scored higher than a short precise one. Here it is on three tiny vectors, so the arithmetic is visible.

import numpy as np# Real embeddings have hundreds of dimensions. The arithmetic is identical.q  = np.array([4.0, 3.0, 0.0])   # "laptop won't charge"d1 = np.array([8.0, 6.0, 1.0])   # "notebook battery not powering"d2 = np.array([1.0, 0.0, 5.0])   # "how to change your laptop wallpaper"def cosine(a, b):    return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))print(round(cosine(q, d1), 4))   # 0.995print(round(cosine(q, d2), 4))   # 0.1569

Work the first one by hand: the dot product is 4*8 + 3*6 + 0*1 = 50, the length of q is 5, the length of d1 is the square root of 101, about 10.05, and 50 / (5 * 10.05) = 0.995. Two things are worth staring at. First, d1 is roughly twice as long as q and points in nearly the same direction, and cosine reports the direction, not the size. Second, d2 is the only document that literally contains the word laptop, and it scores 0.1569 — lexical overlap and semantic relatedness are genuinely different measurements.

In production you normalise every document vector once at index time, which turns the whole search into one matrix multiply.

docs   = np.vstack([d1, d2])                                   # (n_docs, dims)docs_n = docs / np.linalg.norm(docs, axis=1, keepdims=True)    # do this at index timeq_n    = q / np.linalg.norm(q)scores = docs_n @ q_n              # cosine for every document, one matmulorder  = np.argsort(-scores)       # best firstprint(order, np.round(scores[order], 4))

That prints the order [0 1] with scores 0.995 and 0.1569. Once vectors are unit length, cosine similarity is the dot product, which is why vector databases store normalised vectors and talk about inner product search.

What embeddings genuinely cannot do

Vector search fails in specific, predictable ways, and knowing them is the difference between a system you can debug and one you keep re-tuning by feel.

• Exact identifiers. A SKU like MX-4471-B, an order number, a stack-trace symbol, an error code. The tokeniser shreds these into subword fragments, and the resulting vector lands in a vague neighbourhood of other codes that look similar. Nearest-neighbour search then returns a confidently wrong identifier, which is worse than returning nothing.
• Negation. "Battery is charging" and "battery is not charging" produce nearly identical vectors. The training objective rewards topical similarity, not truth conditions, and one negation word barely moves the point.
• Rare proper nouns. Your internal service names, a new drug name, a customer's company. If the model never saw the token during training it falls back to subword pieces and guesses from shape.
• Numbers and ranges. A query about chargers under 40 watts happily matches a 65 watt charger. Embeddings do not do arithmetic.
• Domain drift. A general-purpose model dropped onto legal contracts, clinical notes, or your own internal jargon has no idea which distinctions your users consider meaningful.

There is also a capacity limit that gets ignored. One vector holds one summary. Squeeze a 2,000-word page into a single 768-dimension point and you get the average of five topics, which is close to nothing in particular. This is why chunking matters: your chunk boundaries decide what is findable at all. Check your model's token limit too — most embedding APIs silently truncate long inputs, so the tail of a long document may simply not be in the index.

Every one of these failures is a case where BM25 is strong. That is the argument for hybrid search, and it is a mechanical argument, not a hedge.

Fusing two rankings instead of two scores

The obvious way to combine lexical and vector results is to add the scores. It does not work well. BM25 scores are unbounded and depend on corpus statistics; cosine scores sit in a narrow band, and for many models every plausible result looks like 0.7 to 0.9. Min-max normalising per query helps until a query returns three results instead of fifty and the scale shifts under you.

Reciprocal rank fusion sidesteps the problem by throwing the scores away and using only positions. Each list contributes 1 / (k + rank) for each document it returns, and the contributions are summed.

from collections import defaultdictdef rrf(rankings, k=60):    """rankings: lists of doc ids, best first, one list per retriever."""    scores = defaultdict(float)    for ranking in rankings:        for rank, doc_id in enumerate(ranking, start=1):            scores[doc_id] += 1.0 / (k + rank)    return sorted(scores.items(), key=lambda kv: -kv[1])bm25   = ["d9", "d1", "d3", "d5"]vector = ["d1", "d7", "d9", "d3"]for doc_id, score in rrf([bm25, vector]):    print(doc_id, round(score, 4))

The output ranks d1 first at 0.0325, then d9 at 0.0323, d3 at 0.0315, d7 at 0.0161 and d5 at 0.0156. Look at what happened: every document both retrievers found floats above every document only one retriever found, even though d7 was ranked second by the vector index while d3 was ranked third by both. Agreement between independent retrievers is the signal RRF is built to exploit.

The constant k controls how much a top position is worth relative to a middling one. Small k makes rank 1 dominant; large k flattens the list toward pure vote counting. The conventional 60 works acceptably almost everywhere, and that is RRF's real virtue — a decent hybrid ranking with no per-corpus tuning and no score calibration. If you need to favour one retriever, multiply its contribution by a weight, but earn that weight on measured results rather than intuition.

Why exact nearest-neighbour search stops scaling

The matmul above is exact: it compares the query against every document. At 50,000 documents that is trivial. At 10 million documents with 768 dimensions it is about 7.7 billion multiply-adds per query, every query, forever. Add a hundred queries per second and you are buying hardware to brute-force a problem that has a better shape.

Approximate nearest neighbour indexes trade a small amount of correctness for a large amount of speed. They will occasionally miss a document that genuinely belonged in the true top ten, and in exchange they inspect a few hundred candidates instead of ten million.

HNSW, the most common choice, builds a proximity graph. Every vector becomes a node linked to a handful of its near neighbours, and the graph is stacked in layers: the top layer is sparse with long-range links, and each layer down is denser and more local. A search enters at the top layer, greedily hops to whichever neighbour is closer to the query, and when it cannot improve, drops a layer and continues. It is a skip list over geometry — coarse jumps across the space first, fine local refinement at the bottom. Each hop is a few distance computations, so total work grows roughly logarithmically with corpus size rather than linearly.

The knobs are worth knowing by name, because vector database defaults are chosen for demos, not for your corpus.

Three honest warnings. Recall is measurable, not a vibe: take a sample of queries, run exact search to get the true top-k, and compute what fraction of it your ANN index actually returned. Repeat as the index grows, because recall drifts. Second, filters and graphs interact badly — filtering after the search can empty your result list, and filtering during traversal can disconnect the graph, so read your engine's documentation on which one it does. Third, deletions in HNSW are usually tombstones rather than real removals, so a corpus with heavy churn needs periodic rebuilds, and the graph generally has to live in RAM.

Reranking: the accurate model you cannot run everywhere

Everything so far shares one property. The query and the document are encoded independently, which is what lets you embed the corpus once and reuse it forever. That independence is also the ceiling: the model never sees the query and the document at the same time, so it cannot judge which part of a passage answers which part of the question. It can only ask whether two summaries point the same way.

A cross-encoder removes the independence. It concatenates the query and one document into a single input, runs a transformer over the pair, and emits a relevance score. Attention now runs across both texts, so the model can notice that a passage describes the opposite case, or that it mentions the exact error code the user typed, or that a paragraph name-drops the topic without answering anything. This is why rerankers repair precisely the failures listed earlier.

The cost is brutal and unavoidable. There is nothing to precompute — every query-document pair needs its own forward pass, over an input longer than either text alone. Scoring a million documents per query is not a tuning problem, it is arithmetic. So you split the work: hybrid retrieval returns 50 to 200 candidates cheaply, the cross-encoder reorders just those, and you show the top handful. Latency is roughly the candidate count times the per-pair cost, batched, which is how a reranker fits inside a normal request budget.

One consequence deserves to be stated flatly: a reranker cannot recover a document the retriever never returned. Final quality is capped by retrieval recall at the candidate depth. If recall at 100 is 0.7, no reranker on earth takes you past 0.7. That is why the retrieval stage and the ranking stage need separate metrics.

Measuring retrieval instead of eyeballing it

Most teams tune retrieval by typing a few favourite queries and squinting at the results. That finds catastrophic bugs and nothing else — it cannot tell you whether switching embedding models helped, and it is heavily biased toward the queries you already know work. You need a labelled set and three metrics, each answering a different question.

Recall@k asks whether the relevant document appeared anywhere in the top k. It is the retrieval stage's metric, and you should use a generous k such as 50 or 100, because ordering within the candidate set is the reranker's job. MRR is one divided by the rank of the first relevant result, which is the right measure when there is essentially one correct answer — navigational queries like "reset password page". nDCG@10 handles graded relevance with a position discount, which is what you want when several documents are partly useful and being second matters less than being first.

import mathdef recall_at_k(retrieved, relevant, k=100):    return len(set(retrieved[:k]) & relevant) / len(relevant)def reciprocal_rank(retrieved, relevant):    for i, doc in enumerate(retrieved, start=1):        if doc in relevant:            return 1.0 / i    return 0.0def dcg(gains):    return sum(g / math.log2(i + 1) for i, g in enumerate(gains, start=1))def ndcg_at_k(retrieved, grades, k=10):    actual = dcg([grades.get(d, 0) for d in retrieved[:k]])    ideal  = dcg(sorted(grades.values(), reverse=True)[:k])    return actual / ideal if ideal else 0.0

Here relevant is a set of document ids and grades maps a document id to a judgement from 0 to 3. Average each metric across your query set and you have numbers you can compare between configurations instead of anecdotes.

Building the labelled set is a day of work, not a project. Pull 100 to 200 real queries from your logs, stratified so the head and the long tail are both represented — tail queries are where semantic search earns its keep, and a head-only sample will lie to you. For each query, run every configuration you are considering and pool the union of their top 10, so judges see a merged list rather than one system's worldview. Have a person grade each result 0 to 3 against a written rubric, blind to which system produced it, and get a second person to grade an overlapping slice so you know how much your judges disagree with each other. Then freeze the set, version it alongside the index, and hold back a slice you never tune against.

A 150-query set will not detect a one-percent improvement, and you should not pretend otherwise. It will tell you unambiguously whether adding a reranker helped, whether a new embedding model is better on your data, and whether the hybrid stack beats plain BM25 — which are the decisions actually in front of you.

Where these systems quietly go wrong

The failures that survive a good demo are rarely about the model. They are plumbing, and they share a habit of degrading silently instead of erroring.

• Chunk sizes picked before looking at the documents. Chunks that are too large average several topics into one vector; chunks that are too small lose the referents that made them meaningful. Read a dozen real documents before choosing, and confirm nothing is being cut off at the model's token limit.
• Swapping the embedding model without reindexing. Vectors from two different models are not comparable, and the index will cheerfully return neighbours anyway — just wrong ones. Store the model id and dimension count with the index and refuse to serve on a mismatch.
• Permission filters applied after ranking. A top ten that becomes a top three for half your users looks like a broken feature. Filter inside retrieval, and measure recall separately per permission scope.
• No lexical baseline on record. If you never measured BM25 on your labelled set, you cannot claim the vector stack is an improvement. Run it once and keep the number.
• Duplicates crowding the top of the list. Near-identical documents will fill every slot with the same answer. Deduplicate by content hash or near-duplicate clustering before blaming the ranker.

If you are starting fresh, build it in the order that makes each layer prove itself. Ship BM25 and assemble the judged query set in the same week, so your first number exists before you have anything to defend. Add vector retrieval, fuse it with RRF, and check that recall@100 moved. Add the cross-encoder reranker and check that nDCG@10 moved. Only then spend time on ANN parameters, quantisation, or a fine-tuned embedding model — those are optimisations, and optimising a stack you cannot measure is just rearranging it.