How do I set up an LLM gateway for routing requests with fallback models and rate limiting?

A practical LLM gateway sits between your app and model providers, then handles:

• routing to the right model/provider
• fallbacks when a model fails or is overloaded
• rate limiting per user/org/API key
• observability for cost, latency, and errors

Here’s a solid way to set one up.

---

1) Pick an architecture

Typical flow:

Client → API Gateway / LLM Gateway → Router → Model provider(s)

The gateway should maintain:

• request metadata: tenant, user, api_key, route, priority
• policy config: which models are allowed, fallback order, limits
• telemetry: tokens, latency, error rates, retries, cost

You can build this yourself or use an existing gateway layer.

Common options

• Self-hosted
  • LiteLLM Proxy: popular for model routing, fallbacks, budgets, rate limiting
  • OpenRouter-style proxy patterns
  • custom gateway with FastAPI/Node/Go
• Cloud/API gateway
  • Kong, NGINX, Envoy, APISIX for edge rate limiting/auth
  • plus a separate LLM-aware router service

If you want fastest time-to-value, a lot of teams use LiteLLM Proxy for the LLM-specific parts and put Kong/Cloudflare/NGINX in front for network/security controls.

---

2) Define routing policy

Routing usually depends on:

• task type: chat, embeddings, rerank, tool use
• quality tier: cheap vs premium
• context length
• latency target
• tenant/customer tier
• availability
• price
• data residency constraints

Example policy:

• default: gpt-4o-mini
• premium: gpt-4.1
• fallback chain:
  1. gpt-4.1
  2. claude-3.5-sonnet
  3. gpt-4o-mini
• embeddings:
  • text-embedding-3-small primary
  • text-embedding-3-large fallback

Add rules like:

• if prompt > 100k tokens, route to long-context model
• if provider error rate > 5% for 1 minute, fail over
• if tenant budget exhausted, downgrade or reject

---

3) Implement fallback behavior

A good fallback strategy usually checks:

• HTTP 429 / rate limit from provider
• 5xx errors
• timeouts
• connection errors
• model-specific refusal/unavailability

Avoid retrying forever. Use:

• timeouts
• max retries
• exponential backoff
• circuit breakers

Example fallback logic

1. Try primary model
2. If timeout/429/5xx, retry once
3. If still failing, switch to secondary
4. If all fail, return a clean error to the client

Important:

• Don’t fall back from a stronger model to a weaker one if the task requires strict quality.
• Preserve request compatibility across models where possible.

---

4) Add rate limiting

You usually want multiple limit layers:

A. Edge/network limit

Protects infrastructure:

• per IP
• per API key
• per org/tenant

B. LLM usage limit

Protects cost and abuse:

• requests per minute
• tokens per minute
• concurrent requests
• daily budget in dollars

Common algorithms

• Token bucket: good for bursty traffic
• Leaky bucket: smooth traffic
• Sliding window: simple and fair
• Fixed window: easiest but less precise

For LLMs, token-based limits are often more useful than request-based limits.

Example:

• 100 requests/min
• 100,000 input tokens/min
• 20,000 output tokens/min
• 500 concurrent tokens in flight
• daily budget = $50

---

5) Store policies and counters centrally

Use a shared store like:

• Redis for rate limits, counters, circuit breaker state
• Postgres for policy config, tenants, model mappings
• Prometheus/Grafana or Datadog for metrics

You want gateway instances to behave consistently across a cluster, so counters and limit state should be shared.

---

6) Secure the gateway

Add:

• API key auth or OAuth/JWT
• per-tenant model allowlists
• encryption in transit
• optional redaction of sensitive fields
• logging policies to avoid storing prompts if they contain private data
• per-provider secret management via Vault/Secrets Manager

If you’re handling regulated data, make sure you support:

• region-based routing
• provider allowlists
• prompt logging opt-out
• audit logs

---

7) Observe everything

Track:

• request count
• latency by model/provider
• success/error rate
• fallback rate
• token usage
• cost by tenant
• rate-limit hits
• circuit-breaker open/closed status

This lets you answer:

• Which model is cheapest for this workload?
• When is a provider degrading?
• Which tenants are consuming the most spend?

---

8) Example implementation pattern

Simple routing config

routes:
  chat-default:
    primary: openai/gpt-4.1
    fallbacks:
      - anthropic/claude-3.5-sonnet
      - openai/gpt-4o-mini
    timeout_ms: 30000

  embeddings:
    primary: openai/text-embedding-3-small
    fallbacks:
      - openai/text-embedding-3-large
    timeout_ms: 10000

limits:
  per_api_key:
    rpm: 60
    tpm: 100000
  per_tenant:
    daily_budget_usd: 25

Pseudocode for request handling

def handle_request(req):
    tenant = authenticate(req)
    enforce_rate_limits(tenant, req)

    route = pick_route(req)
    for model in [route.primary] + route.fallbacks:
        if circuit_breaker_open(model):
            continue
        try:
            resp = call_model(model, req, timeout=route.timeout_ms)
            record_metrics(model, success=True, req=req, resp=resp)
            return resp
        except RetryableError as e:
            record_metrics(model, success=False, error=e)
            continue

    raise ServiceUnavailable("All models failed")

Rate limiting with Redis

Use Redis keys like:

• rl:{tenant}:{minute}
• tpm:{tenant}:{minute}
• budget:{tenant}:{day}

Increment atomically and reject when thresholds are exceeded.

---

9) If you want a quick production-ready stack

A common stack is:

• Cloudflare / NGINX / Kong at the edge
• LiteLLM Proxy for model routing/fallbacks/budgets
• Redis for limits and cache
• Postgres for config
• Prometheus + Grafana for monitoring

This gives you:

• centralized auth
• per-tenant policy
• retries/fallbacks
• token and cost tracking
• easy provider switching

---

10) Best practices

• Prefer automatic fallback only for transient failures
• Keep fallback chains short
• Use model-specific health checks
• Route by task, not just “best available model”
• Enforce budgets to prevent surprise bills
• Log usage, not raw prompts, when possible
• Add circuit breakers so a failing provider doesn’t cause cascading retries
• Test behavior with simulated 429/5xx/timeouts

---

If you want, I can give you one of these next:

1. a LiteLLM Proxy setup example,
2. a custom FastAPI/Redis gateway example,
3. a Kong/NGINX + LLM router architecture,
4. or a Docker Compose lab you can run locally.