Capstone Mini-Project: FAISS vs. Qdrant Benchmarking Study

Here is a benchmark result you can find variations of all over the internet: FAISS 0.4 ms, Qdrant 11 ms — FAISS is 27× faster. Both numbers are real measurements. The comparison is worthless.

FAISS was measured inside the Python process, with 1,000 queries submitted as one batch, using all sixteen cores. Qdrant was measured one query at a time over HTTP, including JSON serialisation, a network round trip and a single server thread. Of that 11 ms, roughly 1.5 ms was search and the rest was transport. And nobody reported recall, so we do not know whether either system was returning the right answers — an index tuned to nprobe=1 is extremely fast and extremely wrong.

What you are going to build here is a benchmark that would survive someone hostile reading it. The engineering is straightforward; the discipline is the hard part. By the end you will have a recall-versus-latency curve for both systems on identical data, a memory measurement, a cost model, and a written recommendation you could defend in a design review.

What the study has to answer

A benchmark exists to answer a decision. Write the question down before writing any code, because it determines what you measure:

At recall@10 of at least 0.95, which system gives lower p95 single-query latency, and what does each cost per million queries at that operating point?

Notice what that question forces. You cannot report latency without recall, because latency alone is meaningless. You cannot report an average, because users experience tails. And you must report cost, because a system that is 2 ms faster and four times the price is not a win.

The five phases below produce exactly the artefacts that question needs.

Phase 1 — Environment, data, and ground truth

1python -m venv .venv && source .venv/bin/activate2pip install faiss-cpu numpy pandas matplotlib qdrant-client psutil34docker run -d --name qdrant-bench -p 6333:6333 \5  -e QDRANT__SERVICE__GRPC_PORT=6334 -p 6334:6334 \6  qdrant/qdrant

Generating a dataset that behaves like real embeddings

The single most common way to ruin a vector benchmark is to test on uniformly random vectors. Random points in 768 dimensions are all roughly equidistant from one another — there is no cluster structure for an index to exploit, so IVF partitions are meaningless, HNSW graphs degenerate, and every index looks equally mediocre. Real embeddings sit on a low-dimensional manifold with dense clusters, and that structure is exactly what ANN algorithms exploit.

So generate clustered data, or better, embed a real corpus. Clustered synthetic data at least reproduces the property that matters:

1import numpy as np23N, D, N_CLUSTERS = 1_000_000, 768, 2_0004rng = np.random.default_rng(42)56centroids = rng.normal(0, 1.0, size=(N_CLUSTERS, D)).astype("float32")78def make_chunk(n):9    """Points scattered tightly around random centroids, then normalised."""10    idx = rng.integers(0, N_CLUSTERS, size=n)11    pts = centroids[idx] + rng.normal(0, 0.35, size=(n, D)).astype("float32")12    pts /= np.linalg.norm(pts, axis=1, keepdims=True)13    return pts1415# 1M x 768 float32 is 3.07 GB - build it on disk in chunks, not in one array16vectors = np.lib.format.open_memmap("vectors.npy", mode="w+",17                                    dtype="float32", shape=(N, D))18CHUNK = 50_00019for start in range(0, N, CHUNK):20    vectors[start:start + CHUNK] = make_chunk(min(CHUNK, N - start))21vectors.flush()

Two deliberate choices there. The vectors are normalised, so inner product equals cosine similarity and every system can be configured consistently. And the array is a memmap on disk, because 1,000,000 × 768 × 4 bytes = 3.07 GB and materialising two copies of that during generation is how laptops die.

The query set and the ground truth

Ground truth means the true top-10 neighbours, computed by exhaustive search. Everything you report later is relative to this, so it must be exact.

1N_QUERIES = 5_0002queries = make_chunk(N_QUERIES)          # drawn from the same distribution3np.save("queries.npy", queries)45K = 106truth = np.zeros((N_QUERIES, K), dtype="int64")78# Brute force in blocks: full 5000 x 1000000 score matrix would be 20 GB9BLOCK = 50_00010best_scores = np.full((N_QUERIES, K), -np.inf, dtype="float32")11for start in range(0, N, BLOCK):12    block = np.asarray(vectors[start:start + BLOCK])13    scores = queries @ block.T                       # (5000, BLOCK)14    part = np.argpartition(-scores, K, axis=1)[:, :K]15    cand_scores = np.take_along_axis(scores, part, axis=1)16    cand_ids = part + start17    merged_s = np.concatenate([best_scores, cand_scores], axis=1)18    merged_i = np.concatenate([truth, cand_ids], axis=1)19    keep = np.argpartition(-merged_s, K, axis=1)[:, :K]20    best_scores = np.take_along_axis(merged_s, keep, axis=1)21    truth = np.take_along_axis(merged_i, keep, axis=1)2223order = np.argsort(-best_scores, axis=1)24truth = np.take_along_axis(truth, order, axis=1)25np.save("ground_truth.npy", truth)

Two things worth noticing. The block loop exists because the full score matrix would be 5,000 × 1,000,000 × 4 bytes = 20 GB. And the arithmetic is $2 \times 5{,}000 \times 10^{6} \times 768 \approx 7.7$ TFLOP, which on a 16-core machine sustaining ~500 GFLOP/s takes around fifteen seconds. Exact search is not slow because it is complicated; it is slow because there is a lot of it.

Use 5,000 queries, not 100. If you want to report a p99, the 99th percentile of 100 samples is the single worst measurement — pure noise. With 5,000 queries the p99 is the average of a stable tail.

Phase 2 — The FAISS benchmark

The bug that produces silently wrong numbers

An IndexIVFPQ takes a coarse quantiser: the index used to assign vectors to their nearest of nlist cells. That quantiser must do exact search over the centroids. People sometimes pass a compressed index there, reasoning that compression is good everywhere:

# WRONG - a PQ index as the coarse quantiserquantizer = faiss.IndexPQ(D, 96, 8)index = faiss.IndexIVFPQ(quantizer, D, 4096, 96, 8)

Nothing raises an exception. The index trains, accepts vectors, and answers queries. But every assignment — at build time and at query time — is now made from an approximate distance, so vectors land in the wrong cells and queries probe the wrong cells. Recall collapses to something like 0.3 while latency looks great, and you conclude that IVF-PQ is a bad algorithm rather than that you wired it up wrong.

1# RIGHT - the coarse quantiser is exact; only the residuals are compressed2import faiss34D, NLIST, M_PQ, NBITS = 768, 4096, 96, 85quantizer = faiss.IndexFlatIP(D)                       # exact, over 4096 centroids6index = faiss.IndexIVFPQ(quantizer, D, NLIST, M_PQ, NBITS,7                         faiss.METRIC_INNER_PRODUCT)

The economics explain why this is the right shape. The coarse quantiser searches 4,096 centroids per query — trivial, so exactness is free there. The PQ compression applies to the million stored vectors, where 96 bytes instead of 3,072 saves 2.9 GB. Compress where the data is; stay exact where the decisions are.

The harness

1import time, faiss, numpy as np23vectors = np.load("vectors.npy", mmap_mode="r")4queries = np.load("queries.npy")5truth   = np.load("ground_truth.npy")6K = 1078def recall_at_k(pred, truth, k=K):9    hits = sum(len(set(p[:k]).intersection(t[:k])) for p, t in zip(pred, truth))10    return hits / (len(truth) * k)1112def measure(search_fn, queries, warmup=200):13    for q in queries[:warmup]:                 # warm caches; discard these14        search_fn(q)15    lat, preds = [], []16    for q in queries:17        t0 = time.perf_counter()18        ids = search_fn(q)19        lat.append((time.perf_counter() - t0) * 1000.0)20        preds.append(ids)21    lat = np.array(lat)22    return {23        "p50": float(np.percentile(lat, 50)),24        "p95": float(np.percentile(lat, 95)),25        "p99": float(np.percentile(lat, 99)),26        "recall": recall_at_k(preds, truth),27    }2829faiss.omp_set_num_threads(1)      # one query at a time, one thread - honest units3031results = []32index = faiss.read_index("ivfpq.faiss")33for nprobe in [1, 4, 8, 16, 32, 64, 128]:34    index.nprobe = nprobe35    fn = lambda q: index.search(q.reshape(1, -1), K)[1][0]36    r = measure(fn, queries)37    r.update(system="faiss-ivfpq", param=f"nprobe={nprobe}")38    results.append(r)

Four decisions in that harness are what separate a benchmark from a number.

• Warm-up runs are discarded. The first queries pay page faults on the memmap and cold caches, and are routinely 10× slower. Including them corrupts the p50 as well as the tail.
• faiss.omp_set_num_threads(1). Without it FAISS parallelises a single query across every core and you are comparing sixteen cores against a Qdrant server thread. Measure single-threaded, then discuss throughput separately.
• Queries are issued one at a time. Batching is a legitimate mode — report it as its own row if your workload batches — but it is not what a request-response API does.
• Every latency row carries a recall. A row with latency and no recall is not a data point.

Sweeping nprobe is what produces a curve rather than a point. On a run like this you might see:

Read the shape, not the numbers: latency roughly doubles each time you double nprobe, while recall gains shrink from +31 points to +1.7. That is the universal shape of an ANN trade-off curve, and it is why picking an operating point is a business decision, not a technical one.

Phase 3 — The Qdrant benchmark

Two API traps

First, recreate_collection is deprecated and destructive — it drops an existing collection and its data. It appears in a great deal of older sample code. Use the explicit form:

1from qdrant_client import QdrantClient, models23client = QdrantClient(url="http://localhost:6333", timeout=120)45if not client.collection_exists("bench"):6    client.create_collection(7        collection_name="bench",8        vectors_config=models.VectorParams(size=768,9                                           distance=models.Distance.COSINE),10        hnsw_config=models.HnswConfigDiff(m=16, ef_construct=200),11    )

Second — and this one invalidates more benchmarks than any other single mistake — Qdrant indexes in the background. Immediately after upserting a million points, the HNSW graph does not exist yet, and queries fall back to exact scan. Measure then and you record slow latency and perfect recall, and conclude that Qdrant's HNSW is terrible. Wait for the collection to report green:

1import time2from qdrant_client.models import PointStruct34BATCH = 10005for start in range(0, N, BATCH):6    chunk = np.asarray(vectors[start:start + BATCH])7    client.upsert(8        "bench",9        points=[PointStruct(id=start + i, vector=chunk[i].tolist())10                for i in range(len(chunk))],11        wait=False,           # fire-and-forget is far faster for bulk load12    )1314# Block until indexing has actually finished15while True:16    info = client.get_collection("bench")17    if info.status == models.CollectionStatus.GREEN:18        break19    print("indexing:", info.status, "indexed:", info.indexed_vectors_count)20    time.sleep(5)

Note the asymmetry with the earlier advice: wait=False is correct for bulk loading because it keeps the pipeline full, but you must then explicitly wait for green before measuring. In a correctness test that writes one point and reads it back, use wait=True instead.

Measuring, and what you are actually measuring

1def qdrant_search(q, hnsw_ef):2    return [p.id for p in client.query_points(3        collection_name="bench",4        query=q.tolist(),5        limit=K,6        search_params=models.SearchParams(hnsw_ef=hnsw_ef),7        with_payload=False,8    ).points]910for ef in [16, 32, 64, 128, 256]:11    r = measure(lambda q: qdrant_search(q, ef), queries)12    r.update(system="qdrant-hnsw", param=f"hnsw_ef={ef}")13    results.append(r)

A Qdrant timing includes serialising 768 floats to JSON, an HTTP round trip, server-side deserialisation, the search, and the response. FAISS timing includes only the search. Comparing them directly is the mistake from the opening paragraph, so measure the overhead and report it separately: time a retrieve-by-id call, which does no vector search at all, and treat that as your transport floor. Typically it is 0.4–1.0 ms over local HTTP, and considerably less over gRPC — switching the client to prefer_grpc=True often removes half the gap and is worth reporting as its own row.

State the comparison honestly in your report: FAISS numbers are library latency, Qdrant numbers are service latency, and the transport component is X ms. That single sentence is what makes the study credible.

Phase 4 — Analysis

The primary artefact is the recall-versus-latency Pareto curve: p95 latency on the x-axis (log scale), recall@10 on the y-axis, one line per system with each point labelled by its parameter. Then draw a vertical line at your latency objective and a horizontal one at your recall objective. The systems whose curves pass through the upper-left quadrant are your candidates; everything else is eliminated visually.

1import pandas as pd, matplotlib.pyplot as plt23df = pd.DataFrame(results)4fig, ax = plt.subplots(figsize=(7, 5))5for name, grp in df.groupby("system"):6    grp = grp.sort_values("p95")7    ax.plot(grp["p95"], grp["recall"], marker="o", label=name)8    for _, row in grp.iterrows():9        ax.annotate(row["param"], (row["p95"], row["recall"]), fontsize=7)10ax.axhline(0.95, ls="--", c="grey"); ax.axvline(10, ls="--", c="grey")11ax.set_xscale("log"); ax.set_xlabel("p95 latency (ms)"); ax.set_ylabel("recall@10")12ax.legend(); fig.tight_layout(); fig.savefig("pareto.png", dpi=150)

Alongside the chart, one table with a row per operating point:

Measure the RAM column properly. For FAISS, sample the process resident set size with psutil.Process().memory_info().rss before and after loading. For Qdrant, read the container's memory usage from docker stats after indexing has gone green and settled — not during the build, which peaks much higher.

Then write the two or three sentences that constitute the actual finding. Something of the form: at the 0.95 recall threshold, FAISS HNSW serves p95 in 1.24 ms in-process while Qdrant serves 2.10 ms including 0.71 ms of transport, so the search engines are within 15% of each other and the remaining difference is the API boundary — which buys filtering, persistence and concurrent writes. That sentence is the deliverable. Everything else is evidence for it.

Phase 5 — Cost analysis

Cloud prices change every few months, so a price table dates your report immediately. A formula does not.

with $b$ bytes per component (4 for float32, 1 for int8), $g \approx 138$ bytes for an HNSW graph at M = 16, $p$ the payload bytes, and $h \approx 1.5$ for headroom. For this study, at 1M vectors and 768 dimensions with no payload:

Then convert to the only figure that compares across vendors — cost per million queries:

capacity   = cores * target_utilisation / p95_seconds           = 4 * 0.6 / 0.0021                      = 1,143 QPSper month  = 1,143 * 2,628,000 s                   = 3.00e9 queriesnode cost  = 0.10 USD/hr * 730 hr                  = 73 USD/monthcost/1M    = 73 / 3,000                            = 0.024 USD per million

Do the same arithmetic for the int8 configuration (smaller node, similar latency) and for a managed service using its own metering, and put all three in one table with the same units. Reviewers can then substitute today's prices into your formula without re-running anything, which is what makes a cost section age well.

Report the formula and the inputs, not the total. A number goes stale in a quarter; the model stays correct as long as the architecture does.

What to hand in

A report of roughly five to eight pages, in this order:

1. The question and the answer. Two paragraphs. State the recommendation and the one number that drives it, before any methodology.
2. Setup. Hardware, versions of every library and the Qdrant image tag, dataset construction, query count, threading configuration. Enough for someone to reproduce you.
3. Results. The Pareto chart and the operating-point table.
4. Cost model. The formula, your inputs, cost per million queries per configuration.
5. Threats to validity. Written by you, before a reviewer writes it for you: synthetic data, no filters, single node, transport asymmetry, whatever else applies.
6. Recommendation, with the conditions that would change it.

Failure modes to avoid, and how to prove you avoided them

Reporting latency without recall. Any row of your table without a recall figure should be deleted. Set nprobe=1 and you can claim 0.18 ms; you would also be returning the wrong documents two thirds of the time.

Measuring before the index is built. Qdrant's background optimiser is the classic case, but FAISS has its own version: querying an IndexIVFPQ before train() raises an error, while querying one whose training sample was too small silently gives poor recall. Train on at least 40 × nlist vectors — with nlist = 4,096 that is 163,840 samples minimum.

Comparing across dimensions or datasets. A result on 128-dimension SIFT vectors says nothing about 768-dimension text embeddings. If you change the data, you have started a new study.

Ignoring build time. A 12-minute HNSW build is irrelevant for a static corpus and disqualifying for one rebuilt hourly. Report it; it is a column in the table for a reason.

Reporting means. A mean latency hides the tail entirely, and the tail is what your users complain about. p50, p95, p99, always.

Running the benchmark once. Run each configuration three times and report the median of the three p95 values. Machines have noisy neighbours, thermal throttling and background processes; a single run silently includes all of them.

Stretching the study further

Each of these adds a genuinely new dimension rather than more of the same:

• Add filters. Attach a payload field with known selectivity — 0.1%, 1%, 10%, 50% — and re-run. This is where FAISS has no answer at all and Qdrant's cardinality-aware strategy switching shows up as a discontinuity in the curve. It is usually the most decision-relevant chart in the whole report.
• Benchmark under concurrent writes. Run a background upsert stream at 500 points per second and re-measure query latency. Steady-state numbers on a frozen index are the friendliest possible conditions.
• Vary concurrency. Sweep 1, 8, 32, 128 concurrent clients and plot p95 against offered load. The knee in that curve is your real capacity, and it always arrives earlier than the throughput number suggests.
• Use real embeddings. Embed a public text corpus and repeat. Compare the recall curves against the synthetic run — if they differ noticeably, you have just demonstrated why synthetic benchmarks mislead, which is a finding in itself.
• Add a third system. pgvector with an HNSW index is the interesting comparison, because for many teams the honest answer is that their existing PostgreSQL is sufficient. Build it with CREATE INDEX ON items USING hnsw (embedding vector_cosine_ops) WITH (m = 16, ef_construction = 200); (the defaults are 16 and 64), sweep SET hnsw.ef_search = ... (default 40) as you swept hnsw_ef, and query with the cosine-distance operator <=> so the index is actually used. For the filter experiment, turn on SET hnsw.iterative_scan = relaxed_order; (pgvector 0.8 or later) so a selective WHERE clause does not starve the results.

When you are done, the test of whether the study is any good is simple: hand it to someone who disagrees with your conclusion and see whether they argue with your numbers or with your method. If they argue with the numbers, you have written a benchmark. If they argue with the method, you have written an advertisement.