Build a multi-query RAG system that merges results.

What you need to know

Multi-query retrieval attacks the same problem as query expansion — the user's wording is only one way to ask — but instead of making one longer query, it makes several separate ones and combines the results.

Three building blocks:

• Variant generation — the llm_expand function from the query-expansion question: the original plus n cleaned paraphrases.
• Concurrent retrieval — retrieval is I/O-bound (a network call to the vector store), so a thread pool runs all variants at the same time. Wall-clock time is the slowest retrieval, not the sum.
• Rank fusion — scores from different queries are not comparable (a vague paraphrase gets lower scores everywhere), so fuse by rank with RRF: 1 / (60 + rank) summed across lists.

The function below takes the expansion step as a parameter, so it can be tested without a model.

1from collections.abc import Callable2from concurrent.futures import ThreadPoolExecutor34Hit = tuple[str, str]                     # (chunk_id, text)56def reciprocal_rank_fusion(rankings: list[list[str]], k: int = 60) -> list[str]:7    fused: dict[str, float] = {}8    for ranking in rankings:9        for rank, doc_id in enumerate(ranking, start=1):10            fused[doc_id] = fused.get(doc_id, 0.0) + 1.0 / (k + rank)11    return sorted(fused, key=fused.__getitem__, reverse=True)1213def multi_query_retrieve(14    query: str,15    expand: Callable[[str], list[str]],          # e.g. lambda q: llm_expand(q, llm_fn)16    retrieve: Callable[[str, int], list[Hit]],   # (query, n) -> hits in rank order17    top_k: int = 5, pool: int = 10,18) -> list[Hit]:19    """Retrieve for every variant in parallel and fuse the lists with RRF."""20    queries = expand(query) or [query]2122    def safe(q: str) -> list[Hit]:23        try:24            return retrieve(q, pool)25        except Exception:26            return []                            # one failed variant must not fail the request2728    with ThreadPoolExecutor(max_workers=min(8, len(queries))) as ex:29        results = list(ex.map(safe, queries))3031    texts: dict[str, str] = {}32    for hits in results:33        texts.update(hits)34    fused = reciprocal_rank_fusion([[cid for cid, _ in hits] for hits in results])35    return [(cid, texts[cid]) for cid in fused[:top_k]]

The tricky parts:

• safe turns a failing retrieval into an empty list. With four variants, losing one still leaves a good answer; crashing the whole request does not.
• texts.update(hits) builds an id-to-text map across all lists, so a chunk found by three variants appears once in the output.
• expand(query) or [query] degrades to plain single-query RAG if the model returns nothing usable.

Complexity: q retrievals in parallel, then O(H) to fuse H total hits and O(U log U) to sort U unique ids. Space O(H). Model calls: one for expansion, then the usual one for the answer.

A real-life example

A stub retriever returns fixed rankings for three variants of "is my order refundable":

1RANKS = {2    "is my order refundable":       ["c7", "c2", "c9"],3    "return policy for my order":   ["c2", "c4", "c7"],4    "can I get my money back":      ["c2", "c7", "c1"],5}6retrieve = lambda q, n: [(cid, f"text of {cid}") for cid in RANKS[q][:n]]7expand = lambda q: list(RANKS)8print(multi_query_retrieve("is my order refundable", expand, retrieve, top_k=3))9# [('c2', 'text of c2'), ('c7', 'text of c7'), ('c4', 'text of c4')]

The original query ranked c7 first, but c2 was near the top for every phrasing, so fusion puts it first. c4 beats c9 and c1 because rank 2 in one list is worth more than rank 3 in one list.

An insurance chatbot uses this pattern because customers ask the same thing as "claim rejected", "why was my claim denied" and "policy did not pay".

Follow-up questions to expect

• "Isn't this 4x the retrieval cost?" — Yes, in load on the vector store. In latency it is close to one retrieval because they run in parallel. If load matters, cache variant results and use fewer variants.
• "How is this different from query expansion?" — Expansion makes one richer query; multi-query keeps separate queries and fuses their results, so one bad variant cannot drag the whole search off topic.
• "How would you test the fusion?" — Exactly as above: a stub retriever with fixed lists and an assertion against hand-computed RRF order.