Prompt
I'm building a high-volume telemetry pipeline and need to keep costs down, how should I design it?
Latest observation
For a high-volume telemetry pipeline with cost control as a top priority, design around three goals:
- Reduce data volume early
- Use cheap storage for what you keep
- Scale compute only where needed
Here’s a practical design approach.
1) Start with an event contract
Define a strict schema for telemetry events so you avoid noisy, inconsistent payloads.
- Use versioned schemas with backward compatibility
- Keep fields flat and compact
- Avoid sending redundant metadata
- Prefer numeric codes/enums over long strings where possible
- Separate:
- high-cardinality fields you actually need
- debug-only fields that can be sampled or dropped
If every event is bloated by even 100 bytes, costs add up quickly at scale.
2) Filter and aggregate at the edge
The cheapest byte is the one you never ingest.
At the producer or edge collector:
- Drop unneeded events
- Sample low-value events
- Aggregate counters, histograms, and summaries locally
- Batch small events into larger payloads
- Compress before sending if latency allows
Good examples:
- Send 1-minute aggregates instead of every raw gauge update
- Keep full fidelity only for errors, anomalies, or sampled traces
- Use adaptive sampling when volume spikes
3) Use a buffer layer to decouple ingestion
Put a durable message bus between producers and downstream systems.
Typical choices:
- Kafka / Redpanda
- Kinesis
- Pub/Sub
- Azure Event Hubs
Why:
- Smooths bursts
- Prevents downstream overload
- Lets you fan out to multiple consumers
- Allows replay if a sink fails
Cost tips:
- Keep retention short on the hot buffer
- Partition efficiently
- Avoid over-replication unless required
- Right-size throughput and brokers
4) Separate hot, warm, and cold paths
Not all telemetry needs the same treatment.
Hot path
For real-time alerts, dashboards, and anomaly detection:
- Keep only recent data in fast systems
- Use streaming compute for a subset of events
- Store rollups in a fast analytics DB
Warm path
For recent analytics:
- Write to an OLAP store or warehouse
- Keep only needed dimensions and measures
- Partition by time and maybe tenant/service
Cold path
For long-term retention:
- Store raw or semi-raw events in object storage
- Use compressed columnar formats like Parquet or ORC
- Partition by date/service/region
- Query via external table engines only when needed
Object storage is usually the cheapest place to keep large volumes.
5) Prefer columnar storage and compression
For analytics and retention:
- Use Parquet or ORC
- Compress with ZSTD or Snappy depending on latency/cost tradeoff
- Partition sensibly:
date=YYYY-MM-DDservice=...region=...
Avoid tiny files. Compact them regularly.
6) Use stream processing only where necessary
Streaming compute is powerful but can become expensive if you process everything.
Use it for:
- alerting
- anomaly detection
- deduplication
- enrichment
- windowed aggregations
Avoid:
- expensive joins on every event
- heavy per-event enrichment if it can be deferred
- replicating batch ETL in streaming form unless required
A good pattern is:
- Stream processor handles only critical transformations
- Everything else is batch processed later
7) Control cardinality
High-cardinality dimensions can explode storage and query costs.
Examples:
- user_id
- request_id
- container_id
- trace_id
- exact URL paths
Mitigations:
- Keep them only when needed
- Hash or bucket them
- Normalize paths
- Move request-level detail to sampled traces or logs
- Use them as debug attributes, not always-on dimensions
8) Build retention policies from the start
Define how long each data class lives.
Example:
- Raw events: 7–30 days
- Aggregates: 90–365 days
- Compliance-critical subsets: longer
- Debug logs: short retention unless flagged
Automate lifecycle policies:
- move old data to colder tiers
- expire unnecessary data
- compact partitions
- delete PII when retention ends
9) Keep query costs under control
Analytics costs often come from queries, not ingestion.
To reduce them:
- Precompute common rollups
- Materialize frequent dashboards
- Partition and cluster tables well
- Avoid
SELECT * - Use approximate aggregates when acceptable
- Cache popular queries/results
- Restrict ad hoc access to raw data
10) Make observability economical
Ironically, telemetry systems can generate expensive telemetry about themselves.
Track:
- ingest rate
- dropped events
- queue lag
- compression ratio
- storage growth
- query scan size
- cost per million events
- cardinality growth
Then set budgets and alerts on those metrics.
A cost-efficient reference architecture
A common low-cost pattern looks like this:
Producers → edge filtering/sampling/batching → durable message bus → stream processor →
- hot store for alerts/monitoring
- object storage for raw archive
- warehouse/OLAP for analytics rollups
Plus:
- lifecycle policies
- schema registry
- compaction jobs
- query cost controls
Practical defaults
If you want a starting point:
- Sampling: 1–10% for verbose events, 100% for errors
- Batch size: optimize for network efficiency without too much latency
- Compression: ZSTD for storage, Snappy if CPU is a concern
- Retention: short for raw, long for aggregates
- Storage: object storage for raw, OLAP for queried summaries
- Schema: strict and versioned
- Processing: aggregate early, enrich later
Biggest cost traps
Avoid these if you want costs low:
- ingesting everything at full fidelity
- unbounded cardinality
- tiny files in object storage
- long retention of raw data
- overusing streaming compute
- expensive cross-joins on event streams
- treating every event as equally valuable
If you want, I can also give you:
- a reference architecture diagram
- a cloud-specific design for AWS/GCP/Azure
- a cost-optimized schema strategy for telemetry events