How do I add caching and retries to AI API calls?

A practical way to add both caching and retries to AI API calls is to wrap your model request in a small client layer that:

1. checks a cache first
2. calls the API only on a miss
3. retries transient failures with backoff
4. stores successful responses in cache

1) Cache the request/response pair

Cache by a deterministic key built from:

• model name
• prompt/messages
• temperature/top_p
• tools/function definitions
• any other parameters that change the output

For example, a cache key could be a hash of the full normalized request payload.

Important

Only cache when it’s safe for your use case:

• Good for: repeated prompts, classification, extraction, embeddings, static summaries
• Be careful for: highly dynamic, personalized, or time-sensitive responses

2) Retry only transient errors

Retry on:

• timeouts
• connection resets
• 429 rate limits
• 5xx server errors

Do not blindly retry on:

• bad request / invalid prompt
• auth errors
• permission errors

Use:

• exponential backoff
• jitter
• a max retry count

3) Example pattern in Python

Here’s a simple example using an in-memory cache and retries:

import time
import random
import hashlib
import json
from functools import wraps

cache = {}

def make_cache_key(payload):
    normalized = json.dumps(payload, sort_keys=True, ensure_ascii=False)
    return hashlib.sha256(normalized.encode("utf-8")).hexdigest()

def retry(max_retries=3, base_delay=1.0, exceptions=(TimeoutError, ConnectionError)):
    def decorator(func):
        @wraps(func)
        def wrapper(*args, **kwargs):
            delay = base_delay
            for attempt in range(max_retries + 1):
                try:
                    return func(*args, **kwargs)
                except exceptions as e:
                    if attempt == max_retries:
                        raise
                    sleep_for = delay + random.uniform(0, delay * 0.2)
                    time.sleep(sleep_for)
                    delay *= 2
        return wrapper
    return decorator

@retry(max_retries=3)
def call_ai_api(payload):
    # Replace with your actual API call
    # raise TimeoutError("temporary failure")
    return {"result": "example response"}

def cached_call_ai_api(payload):
    key = make_cache_key(payload)
    if key in cache:
        return cache[key], True  # hit

    response = call_ai_api(payload)
    cache[key] = response
    return response, False  # miss

payload = {
    "model": "my-model",
    "messages": [{"role": "user", "content": "Summarize this text"}],
    "temperature": 0.2
}

result, from_cache = cached_call_ai_api(payload)
print(result, from_cache)

4) If you’re using Redis

Redis is a common choice because it:

• works across processes/servers
• supports TTLs
• is fast

Pseudo-flow:

• build cache key
• GET from Redis
• if found, return it
• otherwise call API
• SETEX with TTL

5) Add TTLs

Use different expiration times depending on the request:

• 5–30 minutes for frequently changing data
• hours/days for stable prompts
• no cache or very short TTL for volatile content

6) Consider idempotency and request deduping

If multiple workers might send the same prompt simultaneously:

• use a distributed lock, or
• use request coalescing so only one call is made and others wait for the result

7) Best practices

• Cache only after successful responses
• Include all model parameters in the cache key
• Log cache hit rate and retry counts
• Add circuit breaking if the API is unstable
• Set timeouts on requests
• Keep retries limited to avoid amplifying outages

8) Minimal architecture

User request
   ↓
Build cache key
   ↓
Cache lookup ── hit → return cached response
   ↓ miss
API call with retries
   ↓ success
Store in cache with TTL
   ↓
Return response

If you want, I can show you:

• a Python example using Redis
• a Node.js example
• or a drop-in wrapper for OpenAI/Anthropic API calls