How do I trace a request when it hits Kafka and multiple APIs

To trace a request end-to-end across Kafka + multiple APIs, the standard approach is distributed tracing with correlation IDs / trace context propagated through every hop.

1) Use a single trace context for the whole request

Best practice is to adopt W3C Trace Context or OpenTelemetry:

• trace_id: same for the whole user request
• span_id: unique for each operation
• parent_span_id: links child operations to their parent

This lets you see:

• API A received request
• API A published message to Kafka
• Consumer B picked it up
• B called API C
• C called DB / another service

2) Propagate trace data through HTTP and Kafka

For HTTP APIs

Send trace headers on every request:

• traceparent
• tracestate (optional)

Example:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-00

For Kafka

Include trace context in Kafka message headers.

Typical pattern:

• Producer reads current trace context
• Injects it into Kafka headers
• Consumer extracts it and starts a new child span

If you use OpenTelemetry instrumentation, this is mostly automatic.

3) Create spans at each important boundary

You want spans for:

• Incoming HTTP request
• Kafka produce
• Kafka consume
• Outbound HTTP calls from consumers
• DB calls if relevant

This gives you a timeline of the request flow.

4) Use a tracing backend

Collect traces with:

• OpenTelemetry SDK + Collector
• Backend such as:
  • Jaeger
  • Zipkin
  • Grafana Tempo
  • Datadog / New Relic / Dynatrace

Then you can search by trace_id and see the full path.

5) Add logs with trace IDs

Even with tracing, logs are still useful. Add:

• trace_id
• span_id
• request/user identifiers if allowed

Then you can correlate logs with traces.

Example log line:

{
  "level": "info",
  "message": "Publishing order event to Kafka",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "00f067aa0ba902b7"
}

6) Kafka-specific tracing pattern

A common flow looks like this:

1. Client calls API A
2. API A creates root span
3. API A publishes message to Kafka with trace headers
4. Consumer B receives message, extracts headers
5. Consumer B creates consumer span as child of Kafka span
6. Consumer B calls API C, propagating headers
7. API C continues the trace

7) If you don’t have tracing yet, use a correlation ID

A simpler fallback is a correlation ID:

• Generate one at the first entry point
• Pass it through headers and Kafka message headers
• Log it everywhere

This won’t give you full span timing, but it helps track a request across systems.

8) Recommended implementation stack

If you want the easiest modern approach:

• OpenTelemetry for instrumentation
• OTel Collector for exporting
• Jaeger/Tempo for trace visualization
• Structured logs with trace_id

9) Practical checklist

• Generate trace context at ingress
• Propagate headers on every HTTP call
• Inject/extract headers in Kafka producer/consumer
• Instrument outbound API calls and Kafka operations
• Export traces to a backend
• Put trace_id into logs

10) Tiny example flow

Client -> API A -> Kafka -> Consumer B -> API C
          span1      span2      span3          span4
          trace_id same across all spans

If you want, I can also show:

1. a Kafka + HTTP tracing example in Java/Spring,
2. Node.js, Python, or
3. how to do this with OpenTelemetry end-to-end.