Prompt
How do I trace a single request through APIs and databases?
Latest observation
To trace a single request end-to-end through APIs and databases, you want distributed tracing plus correlated logging.
The core idea
Assign one unique identifier to the request at the edge, then pass it through every service and include it in:
- API requests/responses
- logs
- database queries or query context
- async jobs/messages if the request fans out
That lets you reconstruct the full path later.
Recommended approach
1) Create or propagate a trace ID
When the request first enters your system:
- Generate a trace ID if one doesn’t already exist
- Also create a span ID for that service hop
Common standards:
- W3C Trace Context:
traceparent,tracestate - OpenTelemetry uses this model widely
Example incoming headers:
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-00
2) Pass the trace ID through all services
Every API call between services should include the trace context headers.
Example:
GET /orders/123
traceparent: 00-<trace-id>-<span-id>-01
Each downstream service:
- keeps the same
trace-id - creates a new
span-id - records timing and metadata
3) Instrument your code with tracing
Use an observability SDK such as:
- OpenTelemetry
- Datadog APM
- New Relic
- Jaeger + OpenTelemetry
- Zipkin
Instrumentation should capture:
- request start/end
- service-to-service calls
- database calls
- cache calls
- message queue publishes/consumes
4) Add the trace ID to logs
Structured logs should include:
trace_idspan_id- service name
- request path
- user/session IDs if appropriate
Example log:
{
"level": "info",
"service": "payments",
"trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
"span_id": "00f067aa0ba902b7",
"message": "Charging card",
"order_id": "12345"
}
This makes it easy to search logs for one request.
5) Trace database access
For DB queries:
- wrap queries in spans
- include the same trace context
- log query duration, table/service name, and errors
If your DB supports session/query comments, you can sometimes attach the trace ID there too.
Example SQL comment:
SELECT * FROM orders WHERE id = 12345 /* trace_id=4bf92f3577b34da6a3ce929d0e0e4736 */
Be careful not to leak sensitive info in SQL comments or logs.
6) Use a tracing backend
Send spans to a tracing system:
- Jaeger
- Zipkin
- Grafana Tempo
- Datadog APM
- Honeycomb
- New Relic
Then you can:
- search by trace ID
- view waterfall timelines
- see which service/database call was slow
- inspect errors in the exact request path
Example request flow
- Client calls API Gateway
- Gateway generates
trace_id - Gateway calls
orders-service orders-servicecallspayments-servicepayments-servicequeries PostgreSQL- Every hop emits spans with the same trace ID
- Logs contain the same trace ID
- You search the trace in your tracing UI and correlate logs
If you only want a quick manual method
If you don’t have tracing installed yet:
- Generate a request ID at the entry point
- Include it in every log line
- Forward it in headers between services
- Add it to DB query logs
- Search logs by that ID
This is a simpler version of distributed tracing.
Best practices
- Use OpenTelemetry if you can
- Use structured JSON logs
- Propagate context through sync and async calls
- Never log secrets, tokens, or PII
- Sample traces if volume is high
- Make sure all services use the same trace format
Minimal implementation checklist
- Generate trace ID at ingress
- Propagate
traceparentor custom request ID - Instrument HTTP clients and servers
- Instrument DB calls
- Include trace IDs in logs
- Export spans to a tracing backend
- Search by trace ID to reconstruct one request
If you want, I can show you:
- a Node.js, Python, or Java example, or
- how to do this with OpenTelemetry specifically.