Prompt
How do I instrument services with OpenTelemetry for observability?
Latest observation
Instrumenting services with OpenTelemetry (OTel) means adding traces, metrics, and logs to your app so you can observe behavior across services in a consistent way.
1) Core idea
OpenTelemetry has 3 parts:
- Instrumentation: code or libraries that generate telemetry
- OTel SDK: collects and batches telemetry in your app
- Exporter / Collector: sends telemetry to a backend such as Jaeger, Tempo, Prometheus, Grafana, Datadog, New Relic, etc.
A common setup is:
Service → OpenTelemetry SDK → OpenTelemetry Collector → Observability backend
2) What to instrument
Start with the highest-value signals:
- Traces: request flow across services
- Metrics: latency, error rate, throughput, CPU, memory, queue depth
- Logs: events and errors, ideally correlated with trace/span IDs
A good minimum:
- Incoming HTTP/gRPC requests
- Outgoing HTTP/gRPC calls
- Database queries
- Message queue publish/consume operations
- Important business operations
3) Recommended approach
A. Use auto-instrumentation first
For common frameworks/libraries, OTel often provides auto-instrumentation.
Examples:
- Java: agent-based auto-instrumentation
- Node.js: OTel instrumentation packages
- Python: instrumentations for Flask, FastAPI, requests, etc.
- .NET: built-in integrations and packages
- Go: manual instrumentation is more common, but there are helpers
This gives you quick value without changing much code.
B. Add manual instrumentation for business logic
Auto-instrumentation usually won’t capture domain-specific steps like:
- “payment authorization”
- “fraud check”
- “order finalized”
Add spans around these operations manually.
4) Basic tracing concepts
A trace represents one end-to-end request. A span is one unit of work within a trace.
Important fields:
trace_idspan_idparent_span_idstatus(OK,ERROR)- attributes/tags like:
http.methodhttp.routedb.systemmessaging.systemuser.idortenant.idonly if safe and not sensitive
5) Typical implementation steps
Step 1: Add SDK and instrumentation libraries
Install the OpenTelemetry packages for your language/framework.
Step 2: Initialize the tracer/meter/logger provider
Configure:
- service name
- resource attributes
- sampler
- exporter endpoint
- batching
- propagation format
Step 3: Capture incoming requests
Create a root span for each request and attach context.
Step 4: Propagate context across service boundaries
Make sure trace context is sent in outgoing requests using standard headers:
traceparenttracestate
This is what links spans across services.
Step 5: Add manual spans for key internal operations
Wrap business logic with spans and record errors.
Step 6: Export to an OTel Collector
Prefer sending telemetry to a collector instead of directly to the backend.
Step 7: Visualize and alert
Use your backend to:
- inspect traces
- build SLO dashboards
- alert on latency/error-rate thresholds
- correlate logs with traces
6) Example: Python with FastAPI
from fastapi import FastAPI
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
resource = Resource.create({"service.name": "orders-api"})
provider = TracerProvider(resource=resource)
trace.set_tracer_provider(provider)
span_exporter = OTLPSpanExporter(endpoint="otel-collector:4317", insecure=True)
provider.add_span_processor(BatchSpanProcessor(span_exporter))
tracer = trace.get_tracer(__name__)
app = FastAPI()
@app.get("/checkout")
def checkout():
with tracer.start_as_current_span("process_checkout") as span:
span.set_attribute("order.type", "web")
# business logic here
return {"status": "ok"}
For real apps, also add auto-instrumentation:
opentelemetry-instrumentation-fastapiopentelemetry-instrumentation-requests
7) Example: Node.js with Express
const express = require('express');
const { NodeTracerProvider } = require('@opentelemetry/sdk-trace-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-grpc');
const { BatchSpanProcessor } = require('@opentelemetry/sdk-trace-base');
const { Resource } = require('@opentelemetry/resources');
const { trace } = require('@opentelemetry/api');
const provider = new NodeTracerProvider({
resource: new Resource({ 'service.name': 'orders-api' })
});
provider.addSpanProcessor(
new BatchSpanProcessor(new OTLPTraceExporter({ url: 'http://otel-collector:4317' }))
);
provider.register();
const tracer = trace.getTracer('app');
const app = express();
app.get('/checkout', (req, res) => {
const span = tracer.startSpan('process_checkout');
try {
// business logic
res.json({ status: 'ok' });
} catch (err) {
span.recordException(err);
span.setStatus({ code: 2, message: err.message });
res.status(500).send('error');
} finally {
span.end();
}
});
app.listen(3000);
In practice, use @opentelemetry/sdk-node plus auto-instrumentations.
8) Propagation between services
This is critical. If Service A calls Service B, B should continue the same trace.
Common propagation is W3C Trace Context:
traceparenttracestate
Most OTel auto-instrumentation libraries handle this for HTTP/gRPC automatically.
If you do manual outbound requests, ensure headers are injected/extracted.
9) Metrics to add
Good service metrics:
- request count
- request latency histogram
- error count
- queue lag
- retry count
- DB connection pool usage
Use histograms for latency, not just averages.
Example dimensions:
- route
- status code
- method
- service name
- dependency name
Be careful with high-cardinality labels like user IDs or full URLs.
10) Logging with trace correlation
Add trace context to logs so you can jump from a log line to a trace.
Include:
trace_idspan_idservice.name- request path
- severity
Many logging libraries support this via MDC/log context or OTel log correlation.
11) Best practices
- Instrument edges first: inbound/outbound requests, queues, DB
- Use span names that are stable and meaningful
- Keep attributes low-cardinality
- Never log secrets or sensitive PII
- Sample intelligently in high-traffic systems
- Prefer the Collector over direct backend export
- Add dashboards and alerts tied to user-visible SLOs
- Standardize service names and environment tags (
dev,staging,prod)
12) Common pitfalls
- Forgetting context propagation, causing broken traces
- Creating too many spans
- High-cardinality attributes causing backend cost issues
- Instrumenting everything before defining what you need to observe
- Not correlating logs with traces
- Using averages only for latency instead of percentiles/histograms
13) Practical rollout plan
- Deploy OTel Collector
- Add service name/resource metadata
- Enable auto-instrumentation for HTTP/gRPC
- Export traces to backend
- Add metrics for latency/error rate
- Correlate logs with trace IDs
- Add manual spans for key business workflows
- Build alerts and SLO dashboards
If you want, I can give you a language-specific setup for Java, Python, Node.js, Go, or .NET, or a Docker/Kubernetes OpenTelemetry Collector example.