Implement context compression before passing data to the LLM.

What you need to know

Retrieval returns whole chunks, but usually only one or two sentences in each chunk answer the question. Context compression removes the rest before the prompt is built. It pays three ways: fewer input tokens (cost), a shorter prompt (latency), and less distracting text (accuracy).

Three tiers, from cheap to expensive:

Greedy selection under a budget is the knapsack idea simplified: walk sentences from most to least relevant and take each one that still fits. It is not optimal, but it is O(s) after sorting and close enough.

Token estimate. Roughly 4 characters per token for English prose. Good enough for budgeting; use the real tokenizer for hard limits.

1import re2import numpy as np34def split_sentences(text: str) -> list[str]:5    return [s.strip() for s in re.split(r"(?<=[.!?])\s+", text) if s.strip()]67def compress(query: str, chunks: list[str], embed_fn, token_budget: int = 800) -> str:8    """Keep the query-relevant sentences that fit the budget, in reading order."""9    sentences = list(dict.fromkeys(s for c in chunks for s in split_sentences(c)))10    if not sentences:11        return ""12    v = np.asarray(embed_fn(sentences), dtype=np.float32)13    v /= np.linalg.norm(v, axis=1, keepdims=True) + 1e-1014    q = np.asarray(embed_fn([query])[0], dtype=np.float32)15    scores = v @ (q / (np.linalg.norm(q) + 1e-10))1617    chosen, used = [], 018    for i in np.argsort(-scores, kind="stable"):          # most relevant first19        cost = max(1, len(sentences[i]) // 4)             # about 4 chars per token20        if used + cost <= token_budget:21            chosen.append(int(i))22            used += cost23    chosen.sort()                                         # back to reading order24    return " ".join(sentences[i] for i in chosen)

The tricky parts:

• chosen.sort() — select by relevance, emit by position. Without it the model gets sentences shuffled out of order, and "It takes 5 days" can appear before the sentence saying what "it" is.
• continue, not break, when a sentence does not fit (the if just skips it). A long sentence that does not fit should not stop a shorter, still-relevant sentence from being added.
• dict.fromkeys removes duplicate sentences that appear in overlapping chunks, keeping the first.

Complexity: one embedding call for s sentences, O(s·d) for the scores, O(s log s) for the sort, O(s) for the greedy pass. Space O(s·d).

A real-life example

With the keyword toy embedder (refund, delivery, password) and a budget of 20 tokens:

1VOCAB = ["refund", "delivery", "password"]2toy_embed = lambda ts: [[float(t.lower().count(w)) for w in VOCAB] for t in ts]3chunks = ["Our app has many features. A refund reaches your UPI account in 5 days.",4          "Delivery is free above 199 rupees. Refund requests need the order id."]5print(compress("how long does a refund take", chunks, toy_embed, token_budget=20))6# A refund reaches your UPI account in 5 days. Refund requests need the order id.

The query embeds to [1, 0, 0]. The four sentences are visited in score order (ties keep reading order):

The kept indices are sorted back to reading order and joined: 19 tokens instead of 32. With token_budget=15 only the first refund sentence fits; with token_budget=5 nothing fits and the function returns "" — the edge case in the first follow-up below.

In a legal-document assistant, compression is what lets ten retrieved contract clauses fit in a prompt with room left for the answer.

Follow-up questions to expect

• "A single sentence is bigger than the budget — what happens?" — It is skipped and you may return nothing. Truncate that sentence to the budget, or always keep the top sentence even if it has to be cut.
• "Why not ask the LLM to compress?" — You can (LangChain's LLMChainExtractor does this), and it is more accurate, but it costs a full model call on every query. Start with the embedding filter and measure.
• "How do you know compression did not hurt?" — Compare answer accuracy on a held-out set with and without it, and plant the answer sentence inside filler to check it survives small budgets.