Rate Limits and Quotas

In a normal web app, a request costs a fraction of a cent. In an AI app, one request can cost several cents to a dollar, depending on the model and how much context it reads. Model providers also cap throughput per API key. Without limits, one script, one runaway agent loop, or one enthusiastic team can exhaust the budget or the provider quota for everyone.

What to limitCopy Anchor Link

Limits at several levels catch different problems. Request limits stop bursts. Token quotas stop slow, expensive overuse. Step limits stop loops that would pass the other two.

Algorithm: token bucketCopy Anchor Link

Use a token bucket for rate limits. Each user has a bucket that holds up to N tokens and refills at a steady rate. Each request takes one token (or, for token quotas, the number of model tokens it will use). If the bucket is empty, the request is rejected.

Why a token bucket: it allows short bursts (a user asks three quick follow-ups), while enforcing an average rate over time. Fixed windows, such as "30 per calendar minute", allow double bursts at window boundaries and feel arbitrary to users.

Implement it in Redis with an atomic script, so concurrent requests across many API servers share one counter:

-- KEYS[1] = bucket key, ARGV = capacity, refill_per_sec, now_ms, cost
local cap, rate, now, cost = tonumber(ARGV[1]), tonumber(ARGV[2]), tonumber(ARGV[3]), tonumber(ARGV[4])
local b = redis.call('HMGET', KEYS[1], 'tokens', 'ts')
local tokens = tonumber(b[1]) or cap
local ts = tonumber(b[2]) or now
tokens = math.min(cap, tokens + (now - ts) / 1000 * rate)
local allowed = tokens >= cost
if allowed then tokens = tokens - cost end
redis.call('HSET', KEYS[1], 'tokens', tokens, 'ts', now)
redis.call('PEXPIRE', KEYS[1], math.ceil(cap / rate * 1000) + 1000)
return { allowed and 1 or 0, tokens }

Token quotas need estimatesCopy Anchor Link

You do not know how many tokens a request will use until it finishes. But you need to decide whether to allow it before you start.

The approach that works:

1. Estimate before the call. Count input tokens with the model's tokenizer after the prompt is built. Add the maximum output tokens you allow for this call.
2. Reserve that estimate from the quota.
3. Reconcile after the call. When the model returns actual usage, refund the difference between reserved and actual.

If you cannot reserve, at least check that the remaining quota is above the estimate. Never let a single request run without a max_tokens limit, or one request can consume a whole daily budget.

Where limits are enforcedCopy Anchor Link

The model gateway is the last line. Even if a bug skips a check upstream, the gateway refuses calls that would exceed a tenant's budget.

Respecting provider limitsCopy Anchor Link

Providers enforce their own limits, in requests per minute and tokens per minute, per key or per project. When you hit them you get 429 responses.

• Track your usage against provider limits in the model gateway, and slow down before you hit them. Queueing a request for 2 seconds is better than failing it.
• Spread load across multiple keys, projects, or regions where the provider and your contract allow it.
• Prioritize: interactive chat gets capacity first. Batch jobs, such as re-embedding and evals, use what remains and back off when chat is busy.
• When the provider returns 429 with a retry-after value, honor it.

What clients receiveCopy Anchor Link

Return 429 Too Many Requests with the standard error body and headers that tell the client what to do:

HTTP/1.1 429 Too Many Requests
Retry-After: 18
RateLimit-Policy: 30;w=60
RateLimit: limit=30, remaining=0, reset=18
Content-Type: application/json

{
  "error": {
    "code": "rate_limited",
    "message": "Too many messages. Try again in 18 seconds.",
    "retryable": true,
    "trace_id": "7f3a9c2e"
  }
}

For quota exhaustion, use code: quota_exceeded and include when it resets. Show both cases clearly in the UI. See Error and empty states.

Usage visibilityCopy Anchor Link

Users and admins handle limits much better when they can see usage before they hit a wall.

• Show admins usage by team, user, and model, with cost.
• Alert tenant admins at 50%, 80%, and 100% of monthly budget.
• Show users a gentle notice near their daily limit, not only when blocked.
• Record usage per message (input tokens, output tokens, model, cost) in the database. It is the basis for every report and chargeback.

ChecklistCopy Anchor Link

• Request rate, token quota, and agent step limits are all in place.
• Rate limiting uses a token bucket in a shared store.
• Token usage is estimated and reserved before calls, and reconciled after.
• Every model call has a max_tokens limit.
• The model gateway enforces tenant budgets as a last line.
• Interactive traffic has priority over batch work for provider capacity.
• 429 responses include Retry-After and a stable error code.
• Admins can see usage and get budget alerts.