Why is my fallback routing not switching models when the primary is down?

There are a few common reasons fallback routing doesn’t actually switch models when the primary is down:

1. The primary isn’t being detected as “down”

   • Many systems only fail over on explicit errors/timeouts, not on slow responses or bad outputs.
   • If the primary returns a 200 with an empty/low-quality response, the router may still treat it as healthy.

2. Health checks aren’t configured or aren’t aggressive enough

   • If the router relies on periodic health checks, it may keep sending traffic to the primary until the next check runs.
   • Long intervals, high retry counts, or permissive timeouts can delay failover.

3. Fallback is only triggered on certain error types

   • Some routers only switch on network failures, 5xx errors, or rate limits.
   • They may not fail over on 4xx errors, content-filter refusals, or application-level exceptions unless configured to do so.

4. Caching / sticky routing is pinning traffic to the primary

   • Session affinity, sticky load balancing, or cached route decisions can keep requests on the same model even after it starts failing.

5. The fallback model itself is unavailable or misconfigured

   • If the backup model ID, endpoint, credentials, or region is wrong, the router may appear to “not switch” because the fallback path fails too.

6. Retries are happening before failover

   • Some setups retry the primary several times before considering fallback.
   • This can look like it’s not switching when it’s just retrying the same model.

7. Policy/order configuration is wrong

   • The fallback model may not actually be listed as a backup, or priority order may be reversed.
   • In multi-model routing, the primary may still be weighted too heavily.

8. The system is using client-side routing, but your client never sees the failure

   • If the SDK hides retries internally, or if the application swallows exceptions, your fallback logic may never run.

What to check

• Logs for the exact failure that should trigger failover
• Router config: timeout, retry, health-check interval, failover conditions
• Whether failover is based on transport errors vs application errors
• Whether session affinity or caching is enabled
• Backup model availability and credentials
• Observed request path: primary retried vs actual fallback selected

If you want, paste your routing config or code snippet and I can help pinpoint the exact issue.