Implement sliding window memory for conversations.

What you need to know

The simplest conversation memory: keep the most recent messages that fit, drop the oldest. Three design choices matter:

• Budget in tokens, not turns. "Keep 10 turns" breaks the first time someone pastes a 3,000-line stack trace. A token budget bounds the prompt size regardless of message length.
• The system prompt is not part of the window. It is pinned. If it were evicted, the assistant would forget its role and rules halfway through a long chat.
• A valid start. Chat APIs require the conversation to start with a user message, and a reply without its question confuses the model. After evicting, drop leading assistant messages.

A running total is the trick that makes it cheap: add a message's cost when it enters, subtract it when it leaves, and never re-measure the whole history.

1from collections import deque2from collections.abc import Callable34def approx_tokens(text: str) -> int:5    return max(1, len(text) // 4)67class SlidingWindowMemory:8    """Most recent messages within max_tokens; the system prompt is pinned outside."""910    def __init__(self, system: str, max_tokens: int = 4000,11                 measure: Callable[[str], int] = approx_tokens) -> None:12        self.system, self.max_tokens, self.measure = system, max_tokens, measure13        self.turns: deque[dict] = deque()14        self.used = 01516    def add(self, role: str, content: str) -> None:17        self.turns.append({"role": role, "content": content})18        self.used += self.measure(content)19        self._trim()2021    def _pop_left(self) -> None:22        self.used -= self.measure(self.turns.popleft()["content"])2324    def _trim(self) -> None:25        while self.used > self.max_tokens and len(self.turns) > 1:26            self._pop_left()27        while self.turns and self.turns[0]["role"] != "user":28            self._pop_left()                     # never start on an orphaned reply2930    def messages(self) -> list[dict]:31        return list(self.turns)                  # send with system=self.system

The tricky parts:

• len(self.turns) > 1 stops the first loop from evicting the newest message even if it alone exceeds the budget; that message should be truncated instead (see the follow-ups).
• The second loop fixes the start of the window after the first loop has cut it at an arbitrary point. It has no > 1 guard on purpose: a window holding only an assistant reply is an orphan, and an empty window is the better state — the next user message starts it again.
• measure is injectable, so production can pass the real tokenizer and tests can pass a word counter.
• messages() excludes the system prompt. The Anthropic API takes it as a separate system parameter; for APIs that use a system message, prepend it here instead.

Complexity: each message is appended once and popped at most once, so add is amortised O(1) (plus one measure call per message in and out). Memory is O(max_tokens) worth of text.

A real-life example

A budget of 12 "tokens", measured as words so the arithmetic is easy:

1mem = SlidingWindowMemory("You are a train-booking assistant.", max_tokens=12,2                          measure=lambda s: len(s.split()))3mem.add("user", "Book Mumbai to Pune")                 # 44mem.add("assistant", "Which date?")                    # 2   total 65mem.add("user", "Tomorrow morning")                    # 2   total 86mem.add("assistant", "Deccan Queen at 7:10, 2S or CC?")   # 7   total 15 -> trim7print([m["content"] for m in mem.messages()], mem.used)8# ['Tomorrow morning', 'Deccan Queen at 7:10, 2S or CC?'] 9

The window is valid but has lost the destination. That is the sliding window's real weakness, and why it is usually combined with a summary of older turns.

Customer-support chat widgets on e-commerce sites commonly run exactly this, with a budget of a few thousand tokens.

Follow-up questions to expect

• "One pasted message is bigger than the whole budget — what happens?" — It is kept alone (the > 1 guard). Truncate it to the budget, or summarise it, before it is stored.
• "What about tool calls?" — Treat a tool_use message and its tool_result as one unit and evict them together; the API rejects a result without its call.
• "How do you keep important early facts?" — Add a rolling summary of evicted messages, or extract durable facts into long-term memory before eviction.