The most cost-effective pattern is usually:

1) Use one or a few shared collectors/gateways, not per-service direct shipping

Have every service send telemetry to a local agent or node-level collector (or directly to a cluster/shared gateway if network cost is low), then forward in bulk to your telemetry pipeline platform.

Why this saves money:

• reduces egress/network chatter
• enables batching/compression
• lets you do sampling, filtering, and normalization before paying ingestion/storage costs
• avoids giving every service its own expensive integration path

2) Do edge-side filtering and sampling before export

Drop or downsample:

• high-volume debug logs
• noisy traces
• low-value metrics
• duplicate spans/events

Common cost-saving tactics:

• tail sampling for traces
• metric aggregation instead of raw event forwarding
• log level filtering
• attribute/cardinality control to prevent “exploding” billable dimensions

3) Standardize on OpenTelemetry where possible

Use OpenTelemetry SDKs + OpenTelemetry Collector (or platform-compatible collectors) so hundreds of services share:

• one instrumentation standard
• one pipeline config
• one place to manage processors/exporters

This is often cheaper operationally than maintaining many custom integrations.

4) Route by data type and value

Not all telemetry should be treated equally:

• Metrics: keep high-value SLO/infra/app metrics, aggregate the rest
• Traces: sample aggressively, keep errors and slow requests
• Logs: ship only important logs, or use a separate cheaper storage tier

5) Use a fan-in architecture

A practical design is:

Services → local agent/collector → regional/shared gateway → telemetry platform

This is usually more cost-effective than:

Services → telemetry platform directly

because it reduces duplicate processing and outbound connections.

6) Control cardinality and payload size

Big hidden costs often come from:

• high-cardinality labels/tags
• verbose attributes
• repeated metadata in every event

Trim or normalize labels like:

• user_id
• request_id
• pod_uid
• full URLs with unique IDs

7) Send only what you need for each environment

A common policy:

• Prod: full metrics, sampled traces, filtered logs
• Staging: lighter sampling
• Dev/test: minimal or off by default

---

Best default recommendation

If you want the most cost-effective general setup for hundreds of services:

OpenTelemetry Collectors as shared gateways, with batching + compression + tail sampling + metric/log filtering before export to the telemetry platform.

If you want, I can also sketch:

• a reference architecture
• a cost-optimized OpenTelemetry Collector pipeline
• or compare direct-to-platform vs collector gateway vs sidecar approaches.