Live Coding Interview Prep

Course Content

Live Coding Interview Prep

7 sections · 50 lessons

Implement semantic caching for similar queries.

Three lookups against a cache at threshold 0.93resetpassword0.991hit, correctunder Rs 5000.998hit, WRONGunder Rs 5000.426missnearest storedscoredecisionforgot passwordover Rs 500store hours
Embeddings barely notice 'under' versus 'over', so the most similar pair is the one that gives a wrong answer.

What you need to know

An exact cache misses "How do I reset my password?" when the stored question was "I forgot my password, what now?". A semantic cache matches by meaning: two queries whose embeddings are close enough share an answer.

Everything depends on the similarity threshold:

  • Too low, and different questions share answers (false hits) — a wrong answer delivered confidently, the worst outcome.
  • Too high, and paraphrases miss, so the cache saves little.

Embeddings are weak at exactly the words that flip meaning: numbers, negation and comparisons ("under" vs "over", "can" vs "cannot"). Two sentences differing in one of those words often score above 0.95. So a semantic cache needs:

  • a threshold tuned on labelled "should hit" and "must not hit" pairs,
  • rules that exclude personal or time-sensitive queries ("my balance", "today"),
  • a separate index per tenant or user group.
Python
import timefrom collections.abc import Callableimport numpy as npclass SemanticCache:    """Return a stored answer when a new query is similar enough to an old one."""    def __init__(self, embed_fn: Callable, threshold: float = 0.93, ttl: float = 3600,                 maxsize: int = 5000, clock: Callable[[], float] = time.monotonic) -> None:        self.embed_fn, self.threshold, self.ttl = embed_fn, threshold, ttl        self.maxsize, self.clock = maxsize, clock        self.entries: list[dict] = []              # {"query", "value", "expires"}        self.matrix: np.ndarray | None = None      # (n, d), unit rows    def _embed(self, text: str) -> np.ndarray:        v = np.asarray(self.embed_fn([text])[0], dtype=np.float32)        return v / (np.linalg.norm(v) + 1e-10)    def _drop_expired(self) -> None:        keep = [i for i, e in enumerate(self.entries) if e["expires"] > self.clock()]        if len(keep) != len(self.entries):            self.entries = [self.entries[i] for i in keep]            self.matrix = self.matrix[keep] if keep else None    def get(self, query: str) -> tuple[object | None, float]:        """Return (value or None, best score)."""        self._drop_expired()        if self.matrix is None:            return None, 0.0        scores = self.matrix @ self._embed(query)        i = int(np.argmax(scores))        return (self.entries[i]["value"] if scores[i] >= self.threshold else None), float(scores[i])    def set(self, query: str, value) -> None:        v = self._embed(query)[None, :]        self.entries.append({"query": query, "value": value, "expires": self.clock() + self.ttl})        self.matrix = v if self.matrix is None else np.vstack([self.matrix, v])        if len(self.entries) > self.maxsize:       # evict the oldest (FIFO)            self.entries.pop(0)            self.matrix = self.matrix[1:]

The tricky parts:

  • get returns the score as well as the value. Logging the best score of every lookup is how you tune the threshold later.
  • Expiry drops rows from both the list and the matrix with the same index list, so they never get out of step.
  • argmax is enough: only the single nearest stored query matters.

Complexity: get is one embedding call, an O(n) expiry pass and an O(n·d) scan. set is one embedding call plus an O(n·d) copy from vstack. Space O(n·d). Past tens of thousands of entries, put the vectors in an ANN index.

A real-life example

Hand-made 3-dimensional vectors stand in for real embeddings so the scores are easy to follow:

Python
VECS = {"How do I reset my password?":           [0.9, 0.1, 0.0],        "I forgot my password, what now?":       [0.85, 0.2, 0.05],        "Free delivery on orders under Rs 500?": [0.1, 0.9, 0.3],        "Free delivery on orders over Rs 500?":  [0.1, 0.88, 0.35],        "What are your store hours?":            [0.1, 0.1, 0.9]}cache = SemanticCache(lambda ts: [VECS[t] for t in ts], threshold=0.93)cache.set("How do I reset my password?", "Use 'Forgot password' on the login page.")cache.set("Free delivery on orders under Rs 500?", "No, delivery costs Rs 40 under Rs 500.")for q in ["I forgot my password, what now?", "Free delivery on orders over Rs 500?",          "What are your store hours?"]:    value, score = cache.get(q)    print(f"{score:.3f}", "HIT " if value else "MISS", q)# 0.991 HIT  I forgot my password, what now?# 0.998 HIT  Free delivery on orders over Rs 500?# 0.426 MISS What are your store hours?
querynearest stored queryscoredecision
I forgot my password…reset my password0.991hit — correct
… orders over Rs 500?… orders under Rs 500?0.998hit — wrong answer
store hoursunder Rs 5000.426miss — correct

The second row is the whole lesson: the vectors (like real embeddings) barely notice "under" vs "over", so the customer is told delivery costs Rs 40 on a Rs 800 order. The fix is a rule that queries containing amounts or comparisons skip the semantic cache, or a second check that the numbers match.

Customer-support deployments at telecom and e-commerce companies use semantic caches for FAQ-style traffic, with exactly these exclusions.

Follow-up questions to expect

  • "How do you choose the threshold?" — Collect a few hundred labelled pairs, plot hit rate and false-hit rate against the threshold, and pick the point where false hits are acceptable — usually very close to zero.
  • "What metric do you watch in production?" — False-hit rate (sampled and judged), not only hit rate. Hit rate can rise while quality falls.
  • "Can you make a near-hit safer?" — Yes: on a score between, say, 0.90 and 0.95, ask a cheap model "do these two questions have the same answer?" before reusing it.