Prompt
How do I troubleshoot broken trace context propagation
Latest observation
Here’s a practical way to troubleshoot broken trace context propagation end-to-end.
1) Confirm the problem is propagation, not sampling
First check whether spans are:
- missing entirely
- present but disconnected
- present only on one service
If traces appear sporadically, it may be:
- head/tail sampling
- exporter issues
- buffering/flush problems
If a downstream service starts a new trace instead of joining the parent trace, that usually means trace context headers were not received, not recognized, or were overwritten.
2) Verify headers are being injected on the outbound request
Check the client-side instrumentation or middleware and confirm outbound requests include trace headers.
Common header formats:
- W3C Trace Context
traceparenttracestate
- Baggage
baggage
- B3
x-b3-traceidx-b3-spanidx-b3-sampledx-b3-parentspanid
- Jaeger
uber-trace-id
Useful checks:
- Enable debug logging in the tracing SDK/agent.
- Log outbound request headers in a non-production environment.
- Use a proxy like mitmproxy, Charles, or an ingress/service mesh debug tool.
Look for:
- headers missing entirely
- wrong casing or format
- multiple conflicting propagation formats
- headers set on the wrong request object
- headers lost after redirect/retry/rebuild of request
3) Verify headers survive the network path
Even if the client injects them, intermediaries may drop them.
Check:
- API gateways
- load balancers
- reverse proxies
- CDNs
- service meshes
- browser CORS behavior for web apps
Common failure modes:
- proxy strips unknown headers
- CORS preflight blocks custom headers
- HTTP/2 to HTTP/1 translation quirks
- gRPC metadata not forwarded properly
- header size limits truncating baggage or trace headers
- case sensitivity assumptions in custom code
If using browser-based requests:
- ensure server sends
Access-Control-Allow-Headersfor trace headers - ensure
traceparent/baggageare allowed if used cross-origin
4) Verify the server extracts context correctly
On the downstream service, confirm instrumentation is:
- enabled
- using the same propagator format as the client
- executed before app code creates new spans
Common issues:
- service is using W3C while client uses B3 only
- custom framework middleware runs after span creation
- server instrumentation disabled or not auto-initialized
- request context is read from the wrong thread/task/local storage
- async boundary loses context
Signs of extraction failure:
- downstream trace ID differs from upstream
- parent span ID is empty
- trace starts at server ingress every time
5) Make sure both sides use compatible propagation standards
Propagation can fail if services don’t agree on format.
Examples:
- Client injects W3C, server extracts only B3
- One service injects multiple formats, another only extracts one
- Mixed libraries with different defaults
Fix by:
- standardizing on W3C Trace Context where possible
- or configuring multi-format injection/extraction consistently
If you’re in a polyglot environment, verify:
- Java, Go, Python, Node, .NET all use the same propagator
- mesh/collector settings don’t rewrite headers unexpectedly
6) Check context loss across async/concurrency boundaries
This is one of the most common causes.
Propagation breaks when code crosses:
- thread pools
- async/await
- callbacks
- message queues
- background jobs
- reactive streams
- coroutines
- goroutines/tasks without context passing
What to check:
- Is the current span stored in a thread-local/context-local mechanism that doesn’t flow automatically?
- Are you creating new tasks without capturing/restoring context?
- Are you using a framework helper that preserves context?
Symptoms:
- outgoing calls from worker threads have no parent span
- spans inside async callbacks appear as separate roots
Fixes vary by language:
- use framework-supported context propagation utilities
- avoid manual thread hopping without context capture
- ensure executor wrappers / context managers are enabled
7) Check message queues and event-driven systems separately
Trace propagation over Kafka, RabbitMQ, SQS, Pub/Sub, etc. requires explicit header/message attribute propagation.
Verify:
- producer injects trace context into message headers/attributes
- consumer extracts it before starting child spans
- headers are preserved by the queue and not rewritten
- DLQ/retry paths still carry context
Common mistakes:
- using HTTP instrumentation assumptions for messaging systems
- serializing messages without metadata
- queue limit truncates headers
8) Inspect logs and correlate IDs
Enable tracing logs at debug level and compare:
- upstream trace ID
- outbound injected headers
- inbound extracted headers
- span parent/child relationships
If available, use:
- OpenTelemetry debug logs
- agent diagnostics
- collector debug logs
- SDK verbose export logs
A good test is:
- Start a request in service A
- Note trace ID/span ID in logs
- Confirm service B receives same trace ID and parent span ID matches A’s span ID
9) Use a minimal reproducible test
Build a tiny two-service test:
- Service A makes one HTTP call to Service B
- no retries, no proxies, no auth gateway initially
- enable tracing on both
- verify a single connected trace
Then add complexity one layer at a time:
- proxies
- async execution
- service mesh
- messaging
- browser CORS
- retries/redirects
This isolates where context breaks.
10) Common root causes checklist
Quick checklist:
- tracing enabled on both services
- same propagation format on both sides
- outbound request headers contain trace context
- headers not stripped by proxy/gateway
- CORS allows headers if browser-based
- async/thread context preserved
- queue/message headers preserved
- no conflicting manual span creation
- SDK/agent versions compatible
- sampling not mistaken for propagation failure
11) If you’re using OpenTelemetry specifically
Things to verify:
- propagator configured correctly
- auto-instrumentation installed and loaded early enough
- context API used correctly in your language
- exporter working independently of propagation
- resource attributes aren’t the issue
- collector isn’t modifying headers unexpectedly
Typical W3C config examples:
- use
tracecontext,baggage - optionally include
b3only if you need compatibility
12) Fast diagnostic approach
If you need to troubleshoot quickly:
- Enable debug logs in client and server SDKs.
- Capture one outbound request and verify trace headers are present.
- Capture one inbound request and verify the same trace ID arrives.
- Confirm parent span ID matches the caller span.
- If it breaks, determine whether it’s:
- missing injection
- header stripping
- extraction mismatch
- async context loss
If you want, I can also give you:
- a language-specific troubleshooting guide (Java, Node.js, Python, Go, .NET)
- an OpenTelemetry checklist
- or a step-by-step packet/header inspection workflow for your stack.