Prompt
How do I troubleshoot API errors that only happen in production?
Latest observation
When an API error only shows up in production, the goal is to make the production failure observable without changing behavior too much. A good troubleshooting workflow is:
1. Capture the exact failure
Collect:
- HTTP status code
- Response body
- Request path and method
- Timestamp
- Correlation/request ID
- User/account/tenant ID
- Client version or app release
- Headers that matter, especially auth-related ones
If you already have logs, search by request ID or timestamp first.
2. Compare production vs. local/staging
Check what is different:
- Environment variables
- Secrets / credentials
- Base URLs
- Feature flags
- Database contents
- Third-party service endpoints
- Network rules / firewalls / proxies
- TLS certificates
- Data shape and volume
Production-only issues are often caused by:
- Real data containing edge cases
- Missing permissions
- Expired/rotated credentials
- Rate limits
- Timeouts under load
- Different config values
3. Inspect server-side logs and traces
Look for:
- Stack traces
- Validation errors
- Authentication failures
- Downstream service failures
- Timeouts and retries
- Serialization/deserialization errors
If you have distributed tracing, follow the request through every service it touches.
4. Reproduce with production-like input
Use the same:
- Request payload
- Headers
- Query parameters
- User role/permissions
- Time of day if jobs or quotas are involved
If you can’t safely use production data, anonymize and replay the failing request in staging.
5. Check for silent differences in data
Common production-only causes:
- Unexpected nulls or empty strings
- Larger payloads
- Special characters / encoding issues
- Duplicate records
- Legacy data formats
- Tenant-specific configuration
- Deleted or archived resources
6. Validate auth and permissions
A lot of “mysterious” production API errors are actually:
- Wrong token audience/scopes
- Expired tokens
- Service account permissions
- Missing roles in production RBAC
- CSRF/CORS issues for browser clients
7. Watch for rate limits and downstream dependency failures
Production often hits:
- External API quotas
- Database connection pool exhaustion
- Cache misses
- Circuit breakers
- Queue backlogs
- Resource limits in containers or servers
8. Add temporary diagnostic logging
If needed, add structured logs around the failing path:
- Input validation results
- Key branching decisions
- Downstream request/response metadata
- Retry attempts
- Latency timings
Be careful not to log secrets or sensitive user data.
9. Use feature flags or canary releases
If the issue appeared after a change:
- Roll out to a small percentage
- Compare error rates before full release
- Roll back if needed
- Test one feature flag at a time
10. Monitor metrics
Track:
- Error rate by endpoint
- Latency percentiles
- 4xx vs 5xx split
- Dependency failures
- CPU/memory/thread pool usage
- Queue depth
- DB connections
Patterns often point to the root cause faster than logs alone.
11. Create a minimal reproducer
Once you know the conditions, reduce it to the smallest request or state that still fails. That makes the fix and regression test much easier.
12. Add a regression test
After fixing:
- Add a unit/integration test
- Add a contract test if another service is involved
- Add an alert or dashboard if it’s a recurring production risk
Quick triage checklist
If you need a fast start:
- Find the request ID and exact error
- Check production logs for the same timestamp
- Compare prod config to staging
- Reproduce with the same payload
- Look for auth, rate limit, timeout, or data-shape issues
If you want, I can also give you a step-by-step production API debugging playbook or a checklist for 4xx vs 5xx errors.