How do I set up a market data engine for real-time price feeds on a 24/7 exchange?

To set up a market data engine for real-time price feeds on a 24/7 exchange, think in terms of ingestion → normalization → distribution → persistence → monitoring. The exact design depends on whether you’re building for trading, analytics, or public data APIs, but a solid low-latency architecture usually looks like this:

---

1) Define your feed requirements

Start by answering:

• Asset types: spot, futures, options, perps?
• Market data types:
  • trades
  • best bid/ask (BBO)
  • order book depth
  • OHLCV candles
  • funding / index / mark prices
• Latency target:
  • sub-millisecond, low-millisecond, or “near real-time”
• Throughput:
  • symbols count
  • messages/sec at peak
• Availability target:
  • 24/7 means no maintenance window, so you need active failover and rolling upgrades

---

2) Use a layered architecture

A. Feed handlers / ingestion layer

Connect to exchange/native feeds via:

• WebSocket for public streaming data
• FIX / binary protocols if the exchange offers them
• REST only for snapshot/bootstrap or reconciliation, not live streaming

Responsibilities:

• maintain connections
• authenticate if needed
• auto-reconnect
• resubscribe after disconnect
• detect sequence gaps
• request snapshots when needed

Best practice:

• Run one dedicated feed handler per exchange or feed type
• Keep the handler stateless where possible, and push normalized messages downstream

---

B. Normalization layer

Exchanges often have different schemas and message formats. Normalize them into an internal schema like:

• trade
• book_update
• book_snapshot
• ticker
• candle
• funding_rate

Include canonical fields:

• exchange
• symbol
• event_time
• receive_time
• sequence
• price
• size
• side
• best_bid
• best_ask

This layer should also:

• convert prices/sizes to consistent precision
• map symbol names to internal instrument IDs
• enforce time ordering when possible

---

C. Order book builder

If you need depth data, maintain an in-memory book per symbol.

Typical flow:

1. get initial snapshot
2. apply incremental updates in sequence order
3. detect missing sequence numbers
4. resync if a gap is found

Implementation tips:

• use efficient in-memory structures
• separate bid and ask ladders
• keep only the depth you need, e.g. top 10/50/100
• use lock-free or low-lock designs if throughput is high

For 24/7 exchanges, books never “close,” so you need:

• periodic sanity checks
• gap recovery
• heartbeats/timeouts
• stale feed detection

---

D. Message bus / event distribution

Use a fast internal transport between ingestion and consumers:

• Kafka if you want durability, replay, and horizontal scaling
• NATS / Redis Streams / Pulsar for lower-latency pub/sub patterns
• in-process channels if everything is in one service and ultra-low latency matters

Common pattern:

• ingest raw feed
• normalize
• publish to internal topics:
  • market.trades
  • market.book.l2
  • market.ticker
  • market.candles

If you need both low latency and replayability, a hybrid approach works well:

• live path via pub/sub
• durable copy to Kafka/object storage

---

E. Persistence layer

Store data for:

• historical analysis
• recovery
• auditing
• backtesting

Options:

• Time-series DB: ClickHouse, TimescaleDB, InfluxDB
• Columnar warehouse: ClickHouse is especially popular for market data
• Object storage: Parquet files in S3/GCS for cheap long-term retention

Store:

• raw messages
• normalized events
• aggregated candles
• reference data

Important:

• keep raw data immutable
• version your schemas
• store event time and receive time separately

---

3) Build for exchange-specific reliability

Because the exchange is 24/7:

Connection management

• heartbeat/ping-pong
• exponential backoff reconnect
• circuit breaker for repeated failures
• multiple endpoint failover if supported

Sequence integrity

• track message IDs/sequence numbers
• detect dropped packets or missed updates
• trigger snapshot reload on inconsistency

Time sync

• sync all nodes with NTP or preferably PTP if ultra-low latency
• use monotonic clocks for latency measurement
• don’t rely only on system wall-clock time

Idempotency

Make consumers able to handle duplicates gracefully:

• dedupe by (exchange, symbol, sequence)
• or by (event_id) if available

---

4) Separate hot path and cold path

A common mistake is doing too much in the live processing path.

Hot path

• connect
• parse
• normalize
• update state
• publish latest data

Keep it minimal and fast.

Cold path

• persistence
• analytics
• alerting
• resampling candles
• auditing

Use asynchronous handoff so the hot path isn’t blocked by slow storage or downstream consumers.

---

5) Consider your data products

You may want to expose:

• WebSocket API for your clients
• REST API for snapshots and historical queries
• gRPC/internal API for internal services
• push notifications for alerts

For public consumers, provide:

• current ticker
• top-of-book
• full depth snapshots at intervals
• incremental book deltas

---

6) Monitoring and alerting are critical

For 24/7 operation, observe:

Health metrics

• connection status
• message lag
• heartbeat age
• reconnect count
• sequence gap count
• queue depth
• consumer lag

Data quality metrics

• stale symbols
• missing trades
• book desync incidents
• abnormal spread changes
• outlier prices

Alerts

• feed stalled
• missing heartbeats
• rapid reconnects
• gap detected
• downstream backlog growing

Use Prometheus + Grafana, Datadog, or similar.

---

7) Deployment and resilience

High availability

• run multiple instances in different AZs/regions
• active-passive or active-active depending on your consistency needs
• use leader election for stateful book builders if needed

Rolling upgrades

• support graceful shutdown
• drain in-flight messages
• checkpoint state if possible
• replay from durable logs after restart

Backpressure handling

If downstream slows down:

• buffer with limits
• drop non-critical derived data first
• never let live ingestion block indefinitely

---

8) Security and compliance

Even for public market data:

• secure credentials for authenticated feeds
• rotate secrets
• restrict network access
• audit data sources and redistribution rights

Some exchanges have licensing restrictions on redistribution of market data, so confirm:

• what data you can store
• what you can redistribute
• whether you need display or redistribution agreements

---

9) A practical reference architecture

A common production setup:

1. Feed handler service

   • connects to exchange WebSocket/FIX
   • parses and sequence-checks messages

2. Normalizer service

   • converts messages to internal schema

3. Book builder service

   • maintains in-memory order books

4. Event bus

   • Kafka or NATS for internal dissemination

5. Storage pipeline

   • writes raw + normalized data to ClickHouse/S3

6. API service

   • serves latest prices, book snapshots, history

7. Monitoring

   • Prometheus/Grafana + alerting

---

10) Suggested tech stack

A good default stack:

• Language: Rust, Go, or Java for low-latency reliability
• Streaming: Kafka or NATS
• Storage: ClickHouse + S3/Parquet
• APIs: gRPC + WebSocket
• Monitoring: Prometheus + Grafana
• Deploy: Kubernetes or systemd depending on scale

If ultra-low latency is key:

• Rust/Java with careful GC control
• avoid heavyweight serialization in the hot path
• use binary protocols internally where possible

---

11) Minimal implementation checklist

• Exchange feed subscriptions
• Snapshot + incremental book sync
• Sequence gap detection
• Auto-reconnect/resubscribe
• Internal normalized event schema
• Durable event storage
• Real-time API endpoints
• Monitoring/alerting
• Failover and restart recovery
• Data licensing review

---

If you want, I can also provide:

1. a concrete architecture diagram,
2. a Python/Go/Rust sample implementation, or
3. a production-ready Kafka + WebSocket design for a 24/7 exchange feed.