Reliability

An AI request depends on more external systems than a typical web request: one or more model providers, an embedding service, a search index, a re-ranker, guardrail models, and the APIs behind tools. Each one fails sometimes. With six dependencies at 99.9% availability each, the chain is down for about 9 hours a year unless you design for failure.

Failure modes to expectCopy Anchor Link

The worst failure is usually not an error but slowness. A provider that hangs for 60 seconds holds a connection, a worker, and a database connection for that whole time. Enough of those and your whole service stops, even for requests that do not use the slow provider.

TimeoutsCopy Anchor Link

Every network call gets a timeout. No exceptions.

Note the two kinds of model timeout. A total timeout alone is not enough: a stream can deliver its first token quickly and then stall forever. Track time since the last token.

Also propagate a deadline through the request. If the user's request has 90 seconds left, a downstream call should not start with a fresh 120 second timeout.

RetriesCopy Anchor Link

Retry only when the failure is likely temporary and the operation is safe to repeat.

Use exponential backoff with jitter, and a small maximum. For interactive chat, one or two retries is the limit. The user is waiting, and three retries with backoff can take longer than the user's patience.

async function withRetry<T>(fn: () => Promise<T>, { attempts = 2, base = 300 } = {}) {
  for (let i = 0; ; i++) {
    try {
      return await fn();
    } catch (err) {
      if (i >= attempts || !isRetryable(err)) throw err;
      const delay = base * 2 ** i * (0.5 + Math.random()); // jitter
      await sleep(retryAfterMs(err) ?? delay);
    }
  }
}

Avoid retry storms. If every layer retries three times, one failure at the bottom becomes 27 calls. Retry at one layer only, usually the one closest to the failing dependency. The model gateway retries model calls. The orchestrator does not retry the model gateway.

FallbacksCopy Anchor Link

A fallback is a different way to get a useful result when the primary way fails.

Rules for fallbacks:

• Test them. A fallback that has never run in production will fail when you need it. Route a small share of traffic through fallbacks continuously, or run scheduled drills.
• Evaluate them. A fallback model needs its own prompt testing. Prompts tuned for one model can perform badly on another. See Model selection.
• Record them. Log which path served each request, so a quality drop can be traced to fallback usage.

Circuit breakersCopy Anchor Link

When a dependency is failing, retrying every request against it wastes time and adds load to a system that is trying to recover. A circuit breaker stops calling it for a while.

• Closed: calls go through. Track the error rate over a sliding window.
• Open: calls fail immediately or go straight to the fallback. No waiting for timeouts.
• Half-open: after a cool-down, such as 30 seconds, let a few requests through to test recovery.

Put circuit breakers in the model gateway per provider and region, and in the tool service per target system.

Fail open or fail closedCopy Anchor Link

When a guardrail service is down, do you let requests through without the check, or block them?

Make the choice explicit per check and per tenant setting. Never let it be an accident of how an exception is handled.

CancellationCopy Anchor Link

When the user presses stop or closes the tab, stop the work.

• Pass a cancellation signal (AbortSignal in Node, context cancellation in Go, asyncio cancellation in Python) through every call in the request.
• When the client disconnects, cancel the signal.
• The model gateway cancels the provider stream, which stops token billing on most providers.
• Pending tool calls that have not started are dropped. Tool calls already in progress run to completion and are recorded, since a half-done side effect is worse than a finished one.

Backpressure and bulkheadsCopy Anchor Link

• Bounded queues and pools. Limit concurrent model calls per instance. When the limit is reached, queue briefly, then reject with 503, rather than accepting unlimited work and slowing everyone down.
• Bulkheads. Use separate connection pools and worker pools for interactive chat and batch work, so a large batch job cannot starve chat.
• Load shedding. Under extreme load, reject low-priority requests first: batch, then background summarization, then chat.

ChecklistCopy Anchor Link

• Every network call has a timeout, and model streams have an inter-token timeout.
• Deadlines propagate through the request.
• Retries happen at one layer only, with backoff and jitter, and only for retryable errors.
• Every critical dependency has a tested fallback.
• Circuit breakers protect against failing providers and tool targets.
• Each guardrail has an explicit fail-open or fail-closed decision. Permission filtering always fails closed.
• Cancellation reaches the model provider when the user stops or disconnects.
• Interactive and batch work use separate pools.