Surviving outages: timeouts, fallbacks, degraded modes

On the Friday before Diwali, at 8:15 pm, the model provider's error rate climbed. About 30% of calls returned "overloaded" errors, and many of the rest took 20 seconds or more. TiffinGo's ticket volume was already double the normal rate. Drafts stopped appearing, and 45 agents stared at spinning placeholders while the queue passed 2,000 tickets.

The assistant had been built on the assumption that the model is always there. It is not. Every hosted API has incidents, and they tend to come at peak times, because peak times are when everyone else is busy too.

This lesson is about the plan for when the model is slow, wrong or gone. The goal is not to never fail. It is that when the model fails, agents lose some speed, not their ability to work.

How a model dependency fails

The first four are loud and fast. The last is quiet and slow, and needs monitoring rather than error handling. This lesson handles the loud ones.

Timeouts and retries that do not make things worse

A timeout protects your service from waiting forever. A retry recovers from brief failures. Together, set badly, they turn a provider incident into your incident: every stuck call holds a worker, and every retry adds load to a provider that is already overloaded.

TiffinGo's rules:

• Per-call timeout: 10 seconds. The p95 is 5.1 seconds, so 10 seconds cuts off only calls that are clearly stuck.
• One retry, only for retryable errors. Network errors, 429s and 5xx responses. Never retry a 400; it is a bug and will fail again.
• Total budget per draft: about 30 seconds, including the repair and the fallback. With a 10-second timeout, that allows the primary and one fallback call, not a long chain. Drafts are generated when the ticket arrives, so this is still far below the queue wait.
• Respect rate-limit hints. The SDK honours the provider's retry-after header on 429s, which is why the client uses the SDK's own single retry rather than a loop of its own.

The client from section 3 also needs to tell the rest of the service when the provider is the problem. It gains one exception type:

1# additions to llm.py2class Unavailable(LLMError):3    """The provider could not serve the call: network, timeout, rate limit or server error."""45RETRYABLE = (anthropic.APIConnectionError, anthropic.RateLimitError, anthropic.InternalServerError)67# inside complete(), around the messages.create call:8#     try:9#         resp = _client.messages.create(...)10#     except RETRYABLE as exc:11#         raise Unavailable(type(exc).__name__) from exc

APIConnectionError covers network failures and timeouts, RateLimitError covers 429, and InternalServerError covers 500 and above, including "overloaded". Only llm.py knows these provider classes; the rest of the service sees llm.Unavailable.

A circuit breaker

When the provider is failing, calling it for every ticket wastes 10 seconds per ticket and adds to the overload. A circuit breaker notices repeated failures and stops calling for a while.

1import time23class CircuitBreaker:4    def __init__(self, failure_limit: int = 5, window_s: float = 30, cooldown_s: float = 60):5        self.failure_limit, self.window_s, self.cooldown_s = failure_limit, window_s, cooldown_s6        self.failures: list[float] = []7        self.opened_at: float | None = None89    def allow(self) -> bool:10        if self.opened_at is None:11            return True12        if time.monotonic() - self.opened_at >= self.cooldown_s:13            self.opened_at, self.failures = None, []  # try the provider again14            return True15        return False1617    def record(self, ok: bool) -> None:18        now = time.monotonic()19        if ok:20            self.failures = []21            return22        self.failures = [t for t in self.failures if now - t < self.window_s] + [now]23        if len(self.failures) >= self.failure_limit:24            self.opened_at = now

Five failures within 30 seconds open the breaker. For the next 60 seconds, calls skip that model entirely. After the cooldown, calls are allowed again; if they fail again, it reopens within seconds. In a multi-worker service, keep this state in a shared store such as Redis, or each worker learns about the outage separately.

The fallback chain

1PRIMARY = load_release("v6")            # medium tier, the normal release2FALLBACK = load_release("v6-fallback")  # same contract on a different, gate-approved model3BREAKERS = {PRIMARY.version: CircuitBreaker(), FALLBACK.version: CircuitBreaker()}45def draft_with_fallback(ticket_text: str, order: dict, minutes_late: int):6    for release in (PRIMARY, FALLBACK):7        breaker = BREAKERS[release.version]8        if not breaker.allow():9            continue10        try:11            draft = draft_for_ticket(release, ticket_text, order, minutes_late)12            breaker.record(ok=True)13            return draft, release.version14        except llm.Unavailable:15            breaker.record(ok=False)   # provider problem: try the next release16        except llm.LLMError:17            return None, release.version  # refusal or truncation: manual path, not an outage18    return None, "degraded"

The fallback is a full prompt release, not a model name swapped at runtime. Prompts often behave differently on a different model, so v6-fallback has its own folder, passed the regression gate, and is re-tested whenever v6 changes. TiffinGo's fallback uses the large-tier model, which scored 116 of 120. It costs more, but only during incidents.

A fallback on the same provider helps when one model is overloaded, which is the most common incident. It does not help when the whole provider is down. A second provider covers that, at the cost of a second adapter in llm.py, a second prompt release to maintain, and a second data-processing agreement. TiffinGo decided the degraded mode below was enough for full outages, and wrote down that decision.

Degraded mode: useful without the model

When both releases fail, the ticket still opens, and much of the pipeline still works, because most of it was never the model. The late-delivery check (R4), the keyword safety net, the R6 and R7 limits, and policy retrieval are all ordinary code.

So the degraded screen shows: a banner saying "Assistant unavailable, drafts will return", the order with paid prices, any R4 refund already computed, any forced escalation, and the three most relevant policy sections for this city. The agent works the ticket by hand, but faster than before the assistant existed. The ticket is also queued, so if it is still open when the provider recovers, a draft appears.