For observability and audit trails, an API gateway should give you enough detail to answer:

• What happened?
• Who called what, when, and from where?
• Did it succeed or fail, and why?
• Can I reconstruct the request path and related changes later?

Here are the key features to look for.

1) Structured access logs

You want every request logged in a machine-readable format like JSON.

Include fields such as:

• timestamp
• request ID / correlation ID
• client identity (API key, OAuth client, user, service account)
• source IP / forwarded IP
• method, path, route, upstream service
• status code
• latency / upstream latency
• request size / response size
• auth outcome
• error reason
• tenant / environment / region

Why it matters:

• Supports troubleshooting
• Feeds SIEM, log analytics, and alerting systems
• Makes audit queries easier

2) Correlation and trace propagation

The gateway should:

• generate or pass through a correlation ID
• support W3C Trace Context (traceparent, tracestate)
• optionally integrate with OpenTelemetry

Why it matters:

• Lets you trace one request across gateway, services, queues, and databases
• Essential for distributed debugging and observability

3) Metrics export

The gateway should expose metrics for:

• request rate
• error rate
• latency percentiles
• upstream health
• auth failures
• throttling/rate-limit events
• cache hit rate, if applicable

Integration targets:

• Prometheus
• Datadog
• CloudWatch
• Azure Monitor
• Google Cloud Monitoring

Why it matters:

• Power dashboards and alerts
• Helps spot regressions and outages quickly

4) Distributed tracing integration

Support for tracing should include:

• span creation at ingress/egress
• propagation of trace headers
• ability to annotate spans with route, status, and policy decisions

Why it matters:

• Shows where time is spent
• Helps isolate bottlenecks between gateway and backend services

5) Audit-grade request and response metadata

For audit trails, the gateway should record:

• authenticated principal
• authorization decision
• policy applied
• route selected
• backend chosen
• request outcome
• admin/config changes made to the gateway itself

Important:

• Audit logs should focus on who did what and when
• They should be tamper-resistant and access-controlled

6) Admin action audit logging

The gateway management plane should log:

• configuration changes
• route creation/deletion
• policy changes
• certificate/key changes
• role changes
• login/logout events
• failed admin access attempts

Why it matters:

• Often more important than request logs for compliance
• Helps answer “who changed this and when?”

7) Log retention, immutability, and export

Look for:

• configurable retention periods
• export to immutable storage
• support for write-once or append-only destinations
• archival to object storage
• integration with SIEM or log pipelines

Why it matters:

• Required for compliance and forensic investigations
• Protects audit evidence from tampering

8) Fine-grained access control on logs

The gateway should support:

• RBAC/ABAC for log access
• separate permissions for admins, auditors, and operators
• masking/redaction for sensitive fields

Why it matters:

• Logs often contain tokens, identifiers, or personal data
• Audit data itself must be protected

9) Sensitive data redaction and masking

You should be able to redact:

• Authorization headers
• API keys
• tokens
• PII
• payload fields
• query parameters

Why it matters:

• Prevents accidental leakage in logs and traces
• Helps satisfy privacy/security requirements

10) Request sampling and dynamic verbosity

Useful features:

• configurable log sampling
• debug logging for specific routes/clients
• ability to increase verbosity temporarily
• error-only logging options

Why it matters:

• Keeps observability useful without excessive cost or noise

11) Event notifications and alert hooks

The gateway should emit events for:

• auth failures
• spikes in 4xx/5xx
• rate-limit violations
• backend timeouts
• certificate expiry
• policy violations
• admin changes

Why it matters:

• Enables proactive monitoring and incident response

12) Compliance support

If you’re in a regulated environment, look for:

• audit log integrity controls
• time synchronization support
• retention policies
• separation of duties
• export for eDiscovery / compliance tooling
• support for standards relevant to your industry

---

Minimum recommended feature set

If you want the essentials, make sure the gateway supports:

1. Structured JSON access logs
2. Correlation ID + trace context propagation
3. Metrics export
4. Distributed tracing integration
5. Admin/config audit logs
6. Redaction of sensitive data
7. Log retention and export to SIEM/immutable storage
8. RBAC for log and audit access

---

Questions to ask vendors

• Can logs be exported in JSON with custom fields?
• Does it support OpenTelemetry and W3C trace context?
• Can I redact headers, query params, and payload fields?
• Are admin configuration changes audit logged?
• Can logs be sent to Splunk, Datadog, Elastic, or a SIEM?
• Can I retain logs immutably for compliance?
• Can I tie a request to a specific user/client/service identity?
• Can I search by request ID across logs and traces?

If you want, I can turn this into a vendor evaluation checklist or a must-have vs nice-to-have matrix.