Prompt

How do I instrument services with OpenTelemetry for observability?

Technology / Observability1 observationLast seen Oct 19, 2026

Latest observation

Oct 19, 2026OpenAI APIWeb search: off

Instrumenting services with OpenTelemetry (OTel) means adding traces, metrics, and logs to your app so you can observe behavior across services in a consistent way.

1) Core idea

OpenTelemetry has 3 parts:

  • Instrumentation: code or libraries that generate telemetry
  • OTel SDK: collects and batches telemetry in your app
  • Exporter / Collector: sends telemetry to a backend such as Jaeger, Tempo, Prometheus, Grafana, Datadog, New Relic, etc.

A common setup is:

Service → OpenTelemetry SDK → OpenTelemetry Collector → Observability backend


2) What to instrument

Start with the highest-value signals:

  • Traces: request flow across services
  • Metrics: latency, error rate, throughput, CPU, memory, queue depth
  • Logs: events and errors, ideally correlated with trace/span IDs

A good minimum:

  • Incoming HTTP/gRPC requests
  • Outgoing HTTP/gRPC calls
  • Database queries
  • Message queue publish/consume operations
  • Important business operations

3) Recommended approach

A. Use auto-instrumentation first

For common frameworks/libraries, OTel often provides auto-instrumentation.

Examples:

  • Java: agent-based auto-instrumentation
  • Node.js: OTel instrumentation packages
  • Python: instrumentations for Flask, FastAPI, requests, etc.
  • .NET: built-in integrations and packages
  • Go: manual instrumentation is more common, but there are helpers

This gives you quick value without changing much code.

B. Add manual instrumentation for business logic

Auto-instrumentation usually won’t capture domain-specific steps like:

  • “payment authorization”
  • “fraud check”
  • “order finalized”

Add spans around these operations manually.


4) Basic tracing concepts

A trace represents one end-to-end request. A span is one unit of work within a trace.

Important fields:

  • trace_id
  • span_id
  • parent_span_id
  • status (OK, ERROR)
  • attributes/tags like:
    • http.method
    • http.route
    • db.system
    • messaging.system
    • user.id or tenant.id only if safe and not sensitive

5) Typical implementation steps

Step 1: Add SDK and instrumentation libraries

Install the OpenTelemetry packages for your language/framework.

Step 2: Initialize the tracer/meter/logger provider

Configure:

  • service name
  • resource attributes
  • sampler
  • exporter endpoint
  • batching
  • propagation format

Step 3: Capture incoming requests

Create a root span for each request and attach context.

Step 4: Propagate context across service boundaries

Make sure trace context is sent in outgoing requests using standard headers:

  • traceparent
  • tracestate

This is what links spans across services.

Step 5: Add manual spans for key internal operations

Wrap business logic with spans and record errors.

Step 6: Export to an OTel Collector

Prefer sending telemetry to a collector instead of directly to the backend.

Step 7: Visualize and alert

Use your backend to:

  • inspect traces
  • build SLO dashboards
  • alert on latency/error-rate thresholds
  • correlate logs with traces

6) Example: Python with FastAPI

from fastapi import FastAPI
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

resource = Resource.create({"service.name": "orders-api"})
provider = TracerProvider(resource=resource)
trace.set_tracer_provider(provider)

span_exporter = OTLPSpanExporter(endpoint="otel-collector:4317", insecure=True)
provider.add_span_processor(BatchSpanProcessor(span_exporter))

tracer = trace.get_tracer(__name__)
app = FastAPI()

@app.get("/checkout")
def checkout():
    with tracer.start_as_current_span("process_checkout") as span:
        span.set_attribute("order.type", "web")
        # business logic here
        return {"status": "ok"}

For real apps, also add auto-instrumentation:

  • opentelemetry-instrumentation-fastapi
  • opentelemetry-instrumentation-requests

7) Example: Node.js with Express

const express = require('express');
const { NodeTracerProvider } = require('@opentelemetry/sdk-trace-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-grpc');
const { BatchSpanProcessor } = require('@opentelemetry/sdk-trace-base');
const { Resource } = require('@opentelemetry/resources');
const { trace } = require('@opentelemetry/api');

const provider = new NodeTracerProvider({
  resource: new Resource({ 'service.name': 'orders-api' })
});

provider.addSpanProcessor(
  new BatchSpanProcessor(new OTLPTraceExporter({ url: 'http://otel-collector:4317' }))
);

provider.register();

const tracer = trace.getTracer('app');
const app = express();

app.get('/checkout', (req, res) => {
  const span = tracer.startSpan('process_checkout');
  try {
    // business logic
    res.json({ status: 'ok' });
  } catch (err) {
    span.recordException(err);
    span.setStatus({ code: 2, message: err.message });
    res.status(500).send('error');
  } finally {
    span.end();
  }
});

app.listen(3000);

In practice, use @opentelemetry/sdk-node plus auto-instrumentations.


8) Propagation between services

This is critical. If Service A calls Service B, B should continue the same trace.

Common propagation is W3C Trace Context:

  • traceparent
  • tracestate

Most OTel auto-instrumentation libraries handle this for HTTP/gRPC automatically.

If you do manual outbound requests, ensure headers are injected/extracted.


9) Metrics to add

Good service metrics:

  • request count
  • request latency histogram
  • error count
  • queue lag
  • retry count
  • DB connection pool usage

Use histograms for latency, not just averages.

Example dimensions:

  • route
  • status code
  • method
  • service name
  • dependency name

Be careful with high-cardinality labels like user IDs or full URLs.


10) Logging with trace correlation

Add trace context to logs so you can jump from a log line to a trace.

Include:

  • trace_id
  • span_id
  • service.name
  • request path
  • severity

Many logging libraries support this via MDC/log context or OTel log correlation.


11) Best practices

  • Instrument edges first: inbound/outbound requests, queues, DB
  • Use span names that are stable and meaningful
  • Keep attributes low-cardinality
  • Never log secrets or sensitive PII
  • Sample intelligently in high-traffic systems
  • Prefer the Collector over direct backend export
  • Add dashboards and alerts tied to user-visible SLOs
  • Standardize service names and environment tags (dev, staging, prod)

12) Common pitfalls

  • Forgetting context propagation, causing broken traces
  • Creating too many spans
  • High-cardinality attributes causing backend cost issues
  • Instrumenting everything before defining what you need to observe
  • Not correlating logs with traces
  • Using averages only for latency instead of percentiles/histograms

13) Practical rollout plan

  1. Deploy OTel Collector
  2. Add service name/resource metadata
  3. Enable auto-instrumentation for HTTP/gRPC
  4. Export traces to backend
  5. Add metrics for latency/error rate
  6. Correlate logs with trace IDs
  7. Add manual spans for key business workflows
  8. Build alerts and SLO dashboards

If you want, I can give you a language-specific setup for Java, Python, Node.js, Go, or .NET, or a Docker/Kubernetes OpenTelemetry Collector example.

Observation history

How did Obsurfable measure this prompt?

Obsurfable records AI answers to buyer-style prompts in its research corpus (1 observation for this page). Metrics are distributions over observations, not a single static ranking.

Which AI systems does Obsurfable collect answers from?

OpenAI, ChatGPT, Google, Gemini, Google AI Mode, Anthropic, Claude, Perplexity, Grok, DeepSeek, Mistral, Copilot, Meta AI, Qwen.

How can brands improve their AEO?

Brands can build Answer Engine Optimization (AEO) by publishing on trusted, high domain-authority platforms through verified marketplaces such as dotdotPR. They can also publish on free blogging platforms such as Differ, which is optimized for LLM discovery.

Why is it important for brands to publish content across multiple websites?

Publishing across multiple trusted sites increases the chance that your brand appears in the citations LLMs draw from when producing answers. Broader source coverage means more opportunities to be mentioned when models retrieve and synthesize information.

Want this interpreted for your brand?

Explorer is the free public corpus. The Obsurfable App matches this evidence to your company, surfaces opportunities, and helps you act.