Vector Databases (FAISS & Chroma)

A team ships semantic search over a help centre. 8,000 articles, a NumPy array of embeddings, one matrix multiply per query. Search returns in 4 milliseconds and everyone is delighted.

Eighteen months later the same code serves a merged product catalogue: 2 million items. Search takes 1.9 seconds, the pod holds 3.2 GB resident, and it falls over when three users type at once. Nothing broke. The algorithm is exactly as correct as on day one. It was simply asked to do 250 times more work per query, and the work scales linearly.

Do the arithmetic the team should have done in advance. Two million vectors at 384 dimensions in float32 occupy $2{,}000{,}000 \times 384 \times 4 = 3{,}072{,}000{,}000$ bytes — 3.07 GB. Every query multiplies the query vector against all of it: 768 million multiply-adds, about 1.5 GFLOP. That sounds trivial for a modern CPU, and this is where most people mis-diagnose the problem.

The bottleneck is not arithmetic. It is memory bandwidth. To compute those 768 million products the CPU must physically read 3.07 GB out of RAM, and a typical server sustains perhaps 20 GB/s to main memory. 3.07 GB ÷ 20 GB/s ≈ 154 ms of pure data movement per query, before any computation, before any Python overhead. You cannot optimise your way out of that with a faster loop. The only fix is to stop reading most of the data.

The approximate nearest neighbour bargain

An index is a structure built ahead of time that lets you skip most of the collection at query time; approximate nearest neighbour (ANN) search is the family of such structures for vectors. The word "approximate" is the price: these indexes occasionally miss a true nearest neighbour.

The metric for how often they miss is recall@k: of the k results an exact search would return, what fraction did the approximate search actually return? If exact search gives {3, 17, 41, 55, 89} as the top 5 and the index returns {3, 17, 41, 55, 203}, that is recall@5 = 4/5 = 0.80.

ANN search does not make similarity search faster. It makes it cheaper by looking at less, and charges you in recall. Every knob you will ever tune on a vector index is a point on that one trade-off curve.

Whether the trade is worth it depends on what happens downstream. For a "related articles" sidebar, missing one item in ten is invisible. For a compliance search that must not miss a document, or a retrieval step feeding a language model that will confidently answer from whatever it is handed, 0.85 recall means 15% of answers rest on an incomplete evidence set. Decide which case you are in before you pick an index.

FAISS: the low-level, no-magic option

FAISS (Facebook AI Similarity Search) is a C++ library with Python bindings, built by Meta for exactly this problem. It is not a database — no server, no query language, no metadata, no persistence beyond "write these bytes to a file". It is a very fast array of vectors with a search method, and that narrowness is the point: you control precisely what is stored and how it is searched.

1pip install faiss-cpu          # CPU build - what almost everyone needs2# pip install faiss-gpu        # only with a working CUDA toolchain34python -c "import faiss; print(faiss.__version__)"

Exact search first

1import faiss2import numpy as np3from sentence_transformers import SentenceTransformer45model = SentenceTransformer("all-MiniLM-L6-v2")6documents = [7    "Python is a programming language",8    "Java is also a programming language",9    "The weather is sunny today",10    "I like coding in Python",11    "Dogs are loyal pets",12]1314# FAISS works in float32. encode() already returns float32; the cast is a15# cheap guard for vectors that come from elsewhere (older FAISS rejects float64).16embeddings = model.encode(documents).astype("float32")17dimension = embeddings.shape[1]            # 384 for MiniLM-L61819index = faiss.IndexFlatL2(dimension)20index.add(embeddings)21print(index.ntotal)                        # 52223query = model.encode("Python programming").astype("float32").reshape(1, -1)24distances, ids = index.search(query, k=3)2526print(distances)   # [[0.395 0.415 1.266]]  <- SQUARED L2 distances27print(ids)         # [[3 0 1]]

IndexFlatL2 is exhaustive: it compares the query against every stored vector, so it is not an ANN index at all. That is deliberate — it is the correct default until measurement proves otherwise. On a few hundred thousand vectors it answers in single-digit milliseconds, it is exact, it needs no training, and it has no parameters to get wrong.

Both return values have shape (n_queries, k). If the index holds fewer than k vectors, FAISS pads the id array with -1 and the distance array with a huge value. Indexing a Python list with -1 silently returns the last document, a wonderfully confusing bug. Filter for id >= 0 first.

The distance trap that catches nearly everyone

FAISS reports distance, where smaller is better. Similarity scores run the other way. Print one and read it as the other and your ranking is inverted — and because rank 1 is still rank 1, the bug hides until someone notices the worst results at the top.

There is a second, subtler trap layered on the first. IndexFlatL2 returns squared L2 distances, not distances. FAISS skips the square root because it does not change the ordering and costs time. So the standard conversion from L2 to cosine, which for unit-length vectors is

must be applied to what FAISS actually gave you, which is already $d^2$. Work an example. Two normalised vectors with true cosine similarity 0.82 sit at $d^2 = 2 - 2(0.82) = 0.36$, and 0.36 is the number FAISS prints.

The wrong version is still monotonic in the right direction, so the ranking looks fine and nobody notices. It bites only when someone sets a relevance threshold on those numbers — a 0.9 cutoff that should reject a match now admits it.

def cosine_from_flat_l2(squared_distances):    """IndexFlatL2 returns squared L2. Valid only for unit-length vectors."""    return 1.0 - squared_distances / 2.0

IndexFlatIP: skip the conversion entirely

If every vector has unit length, the inner product of two vectors is their cosine similarity, and IndexFlatIP gives you readable scores with no post-processing:

1import faiss, numpy as np23embeddings = model.encode(documents, normalize_embeddings=True).astype("float32")4# or, if you already have raw vectors:5#   faiss.normalize_L2(embeddings)   # normalises IN PLACE, returns None67index = faiss.IndexFlatIP(dimension)8index.add(embeddings)910q = model.encode("Python programming", normalize_embeddings=True)11q = q.astype("float32").reshape(1, -1)12scores, ids = index.search(q, k=3)13print(scores)     # [[0.803 0.793 0.367]]  <- cosine similarities, higher is better

faiss.normalize_L2() modifies the array in place and returns None, so writing emb = faiss.normalize_L2(emb) sets your embeddings to None and the next line raises something unhelpful.

Using IndexFlatIP without normalising is the single most common FAISS bug. Concretely: query [1, 0], document A at [0.9, 0.436] with length 1.0, document B at [1.5, 2.598] with length 3.0. Cosine says A wins, 0.9 against 0.5. Raw inner product says B wins, 1.5 against 0.9 — a worse match promoted purely for being longer. Nothing raises. Long documents simply drift to the top of every result list, and it reads as "the model prefers verbose text".

When exact search really does run out

Say measurement shows IndexFlatIP at 2 million vectors costs 150 ms and you need 20. Now the approximate index types earn their complexity.

IVF: cluster, then search a few clusters

IndexIVFFlat runs k-means over your vectors once, splitting them into nlist cells, each with a centroid. At query time it compares the query against the nlist centroids, picks the nprobe closest cells, and searches only inside those.

Put numbers on it. With 2,000,000 vectors and nlist = 4096, each cell holds about $2{,}000{,}000 / 4096 = 488$ vectors. At nprobe = 16 a query touches $16 \times 488 = 7{,}813$ vectors, plus the 4,096 centroid comparisons — call it 12,000 comparisons instead of 2,000,000. That is a 167× reduction in work, and it moves 150 ms to roughly 1 ms.

What it costs you is the vectors that were genuinely near the query but happened to fall in the 4,080 cells you did not open. That is exactly what nprobe buys back, and it is a query-time parameter — you can change it per request without rebuilding anything:

Those recall figures are indicative, not promises — they depend on how clustered your data is, and you must measure them on your own corpus. A sensible starting point for nlist is $4\sqrt{n}$ to $16\sqrt{n}$; with $\sqrt{2{,}000{,}000} = 1414$, 4096 sits comfortably in range.

1quantizer = faiss.IndexFlatIP(dimension)      # finds the nearest centroids2index = faiss.IndexIVFFlat(quantizer, dimension, 4096, faiss.METRIC_INNER_PRODUCT)34print(index.is_trained)      # False - IVF must learn its centroids first5index.train(embeddings)      # k-means; FAISS wants >= 39 * nlist vectors6index.add(embeddings)78index.nprobe = 16            # tune per query, no rebuild needed9scores, ids = index.search(q, k=10)

Named failure mode: calling add() before train() throws RuntimeError: Error: 'is_trained' failed. Flat indexes never need training, so people who learned on IndexFlatL2 hit this the first time they try IVF and assume the library is broken. A related and quieter one: training on far too few vectors. FAISS wants 39 to 256 examples per centroid; train 4,096 centroids on 5,000 vectors and it will warn, then build cells so lopsided that recall collapses.

HNSW: walk a graph towards the query

IndexHNSWFlat builds a multi-layer graph where each vector links to M neighbours. Search enters at a sparse top layer, greedily hops towards the query, drops a layer, and repeats. It is usually the best speed-versus-recall curve available and it needs no training pass.

The costs are memory and mutability. Each vector stores its links as 4-byte integers — with M = 32 the bottom layer alone holds up to 64 links, so budget roughly 300 to 400 extra bytes per vector on top of the 1,536 the vector itself occupies at 384 dimensions, around 20–25% overhead. Deletion is worse: HNSW cannot truly remove a node without damaging the graph. FAISS's HNSW index does not support removal at all (remove_ids raises an error), and databases built on HNSW mark deleted nodes as tombstones and rebuild periodically.

1index = faiss.IndexHNSWFlat(dimension, 32, faiss.METRIC_INNER_PRODUCT)  # M = 322index.hnsw.efConstruction = 200     # build-time breadth: slower build, better graph3index.add(embeddings)               # no train() step45index.hnsw.efSearch = 64            # query-time breadth: the recall/latency dial6scores, ids = index.search(q, k=10)

efSearch is HNSW's nprobe. Raising it from 16 to 64 typically lifts recall@10 from around 0.90 to around 0.98 and roughly triples query time; pushing to 256 buys another point of recall for another 4× the latency. Note the asymmetry: efConstruction is fixed when you build, efSearch is free to change per query.

PQ: compress the vectors themselves

Product Quantisation attacks memory rather than comparison count. It splits each vector into m sub-vectors and replaces each with an 8-bit code pointing into a learned 256-entry codebook. At 384 dimensions with m = 48, each vector goes from 1,536 bytes to 48 bytes — a 32× reduction, turning 3.07 GB into 96 MB.

The catch is that distances are computed against approximations of your vectors, so accuracy drops noticeably. PQ is rarely used alone; the standard production combination is IndexIVFPQ — cluster to cut comparisons, quantise to cut memory — with a reranking pass that re-scores the top 100 candidates against full-precision vectors.

IndexIDMap: your own identifiers

FAISS numbers vectors by insertion order: 0, 1, 2. Delete one and every later id shifts, which quietly corrupts any mapping you kept on the side. IndexIDMap wraps any index so you can supply your own 64-bit ids and delete by them.

1base = faiss.IndexFlatIP(dimension)2index = faiss.IndexIDMap(base)3index.add_with_ids(embeddings, np.array([1001, 1002, 1003, 1004, 1005], dtype="int64"))45index.remove_ids(np.array([1003], dtype="int64"))6print(index.ntotal)   # 4 - and 1001, 1002, 1004, 1005 keep their ids78faiss.write_index(index, "catalogue.faiss")     # persistence is manual9index = faiss.read_index("catalogue.faiss")

Chroma: a database rather than an array

FAISS stores vectors. Chroma stores documents — text, vector, metadata and id together — and gives you a Python API that handles embedding, persistence and filtering without you wiring anything up. It uses HNSW underneath, so you are getting the same class of index with the plumbing supplied.

pip install chromadb

1import chromadb23client = chromadb.Client()                      # in-memory; gone at process exit4collection = client.get_or_create_collection(5    name="articles",6    configuration={"hnsw": {"space": "cosine"}},   # default is "l2" - set this explicitly7)89collection.add(10    ids=["a1", "a2", "a3"],11    documents=[12        "Machine learning advances in 2024",13        "Deep learning trends and architectures",14        "A recipe for slow-cooked beef stew",15    ],16    metadatas=[17        {"category": "AI", "year": 2024, "words": 1200},18        {"category": "AI", "year": 2023, "words": 800},19        {"category": "Food", "year": 2024, "words": 400},20    ],21)2223res = collection.query(query_texts=["AI trends"], n_results=2)24for doc, dist in zip(res["documents"][0], res["distances"][0]):25    print(f"{1 - dist:.3f}  {doc}")             # cosine DISTANCE -> similarity

This is the Chroma 1.x form. Older code sets the space with metadata={"hnsw:space": "cosine"}, which current versions still accept.

Two things there deserve attention. First, you never called model.encode(). Chroma downloads a default embedding model (an ONNX build of all-MiniLM-L6-v2) and uses it silently — convenient for a demo, and a liability the moment you also embed text elsewhere in your stack, because those two models produce incompatible vectors. Pass embeddings=[...] explicitly, or supply an embedding_function, so the model is a decision you made rather than a default you inherited.

Second, distances is a distance again. With hnsw:space set to cosine, Chroma returns cosine distance, which is $1 - \cos\theta$, so 0.18 means a similarity of 0.82. With the default l2 space it returns squared L2, a different scale entirely. Setting the space explicitly decides how you read every number that comes back.

Metadata filtering, and the pre-filter versus post-filter problem

Filtering is the feature that most often decides Chroma over raw FAISS. You attach arbitrary metadata and constrain queries with where:

1res = collection.query(2    query_texts=["AI trends"],3    n_results=5,4    where={"$and": [{"category": {"$eq": "AI"}}, {"year": {"$gte": 2024}}]},5    where_document={"$contains": "learning"},    # substring match on the text6)

The available operators are $eq, $ne, $gt, $gte, $lt, $lte, $in and $nin, combined with $and / $or. Two details catch people. A where with two keys side by side, such as {"category": "AI", "year": {"$gte": 2024}}, is rejected with "Expected where to have exactly one operator"; wrap the conditions in $and as above. And a metadata value may be a list of scalars of one type, such as "tags": ["ml", "research"], which you match with {"tags": {"$contains": "ml"}} (or $not_contains). Nested dictionaries are not allowed.

The important idea is when the filter is applied. Doing it yourself in Python after the search — a post-filter — produces a specific and very common bug. Suppose 50,000 documents match category = "AI" and you run:

1# WRONG: retrieve 10, then filter2res = collection.query(query_texts=["AI trends"], n_results=10)3hits = [d for d, m in zip(res["documents"][0], res["metadatas"][0])4        if m["category"] == "AI"]5print(len(hits))     # 2

Eight of the top ten happened to be Food articles, so you display two results — while 50,000 eligible AI articles sit in the index unexamined. The user sees a nearly empty page and concludes your search is broken. Push the filter into the query, as a pre-filter, and the engine searches only eligible documents, returning a full ten.

Filter inside the query, never after it. A post-filter cannot return anything the unfiltered search did not already surface, so its result count silently depends on how the unrelated documents happened to rank.

The honest caveat: pre-filtering over a graph index is not free. When a filter is extremely selective — 40 matching documents out of 2 million — the HNSW graph has few eligible neighbours to hop between, and recall degrades or the engine falls back to scanning. If your filters are that narrow, narrow the candidate set by conventional means first and run vector search over what remains.

Persistence

1client = chromadb.PersistentClient(path="./chroma_data")2collection = client.get_or_create_collection(name="articles")3collection.add(ids=["1"], documents=["Persistent document"])45# ... a new process, later ...6client = chromadb.PersistentClient(path="./chroma_data")7print(client.get_collection("articles").count())    # 1

With FAISS this is your job: faiss.write_index() saves vectors and structure, and nothing else. The text, the metadata and the id mapping must be stored separately — usually in SQLite or Postgres — and kept in step with the index. Every rebuild is an opportunity for the two to drift apart.

Choosing between them

The path most projects should take: start with Chroma, because it gets you end-to-end search in an afternoon, and move to FAISS only when a measurement — not an intuition — says you need control Chroma does not expose. The migration is small, because the embeddings are identical either way; only the storage and search layer changes.

The five mistakes that account for most broken indexes

The fourth deserves emphasis because it produces no error at all. Two models can both emit 384 dimensions and still occupy unrelated spaces — each learned its own axes from scratch, so dimension 42 means something different in each. Build an index with one model, query with another, and FAISS happily returns ten results with plausible-looking scores, all of them meaningless. This happens most often during a model upgrade, when new documents get the new encoder and the existing index is never rebuilt.

An index is only valid for the exact model that produced its vectors. Change the model and every vector in the index becomes noise, silently.

The fifth mistake has a twin: reaching for an approximate index before you need one. Below roughly a million vectors, IndexFlatIP is simpler, exact, parameter-free and usually fast enough. Adding IVF at 50,000 documents buys a training step, two tuning parameters, an unmeasured recall regression, and no perceptible speed-up.

What this means when you build something

Measure before you choose. Load your real vectors into IndexFlatIP, time a hundred real queries, and look at the P95 latency. If it clears your budget — and at anything under a million vectors it usually will — you are finished, with an exact index and no parameters to regret. The number that should drive the decision is your measured latency against your actual corpus, not the size of the dataset in someone's benchmark.

If you do move to an approximate index, build a recall harness in the same hour. Take 200 real queries, run each against an exact index to get ground-truth top-10 sets, then run the same queries against the approximate index and compute the mean overlap. Without that number you are tuning nprobe or efSearch by feel, and the feedback loop for "search got slightly worse" is measured in weeks of user complaints. With it, the trade-off becomes a table you can read: at nprobe = 8 you get 4 ms and 0.92 recall, at 32 you get 12 ms and 0.98, and you pick.

Store provenance next to the index. Write a small JSON file beside every index file recording the model name, dimensionality, whether vectors were normalised, the metric and the row count. Five lines, and it is the difference between a routine model upgrade and an afternoon working out why search quality dropped 8% last Tuesday.

Finally, plan for the index to be rebuilt, not merely appended to. Documents get edited and deleted, HNSW graphs accumulate tombstones, and IVF centroids trained on last year's corpus stop matching this year's distribution. Make a full rebuild from source documents a scheduled job you can run any time, rather than a recovery procedure someone invents under pressure.