How do I set up a streaming quotes API for low-latency portfolio valuation and charting analytics?

To set up a streaming quotes API for low-latency portfolio valuation and charting analytics, you want an architecture that separates:

1. Market data ingestion
2. Normalization and fan-out
3. Real-time storage / cache
4. Portfolio valuation engine
5. Charting / analytics API
6. Client delivery via streaming

Here’s a practical blueprint.

---

1) Define the latency target and quote type

First decide:

• Latency target: sub-100 ms, sub-500 ms, or “near real time”
• Asset classes: equities, options, FX, crypto, futures
• Quote source: SIP, exchange direct feeds, broker feed, aggregator
• Quote granularity:
  • trades
  • NBBO / top-of-book
  • full depth order book
• Update semantics:
  • every tick
  • only on price change
  • batched updates every 100–250 ms

For portfolio valuation and charting, most systems use:

• Top-of-book + last trade
• Incremental updates
• Event-driven streaming rather than polling

---

2) High-level architecture

A common low-latency design:

Market Data Vendors
        |
        v
Feed Handlers / Adapters
        |
        v
Normalizer + Symbol Mapper
        |
        v
Message Bus / Stream
   (Kafka, Redpanda, NATS, Pulsar)
        |
        +--------------------------+
        |                          |
        v                          v
Real-time Cache               Analytics Engine
(Redis / Aerospike)            (valuation, bars, indicators)
        |                          |
        v                          v
Streaming API Gateway        Historical Store
(WebSocket/SSE/gRPC)         (ClickHouse, TimescaleDB, S3)
        |
        v
Clients (web, mobile, services)

---

3) Ingest quotes efficiently

a) Use vendor adapters

Write a dedicated adapter for each feed or vendor API:

• WebSocket feed handler
• TCP multicast handler
• FIX market data handler
• REST polling fallback if needed

These adapters should:

• authenticate
• subscribe/unsubscribe to symbols
• decode vendor payloads
• convert to a canonical quote format

b) Normalize into one schema

Create a standard event structure like:

{
  "ts": 1730000000123,
  "symbol": "AAPL",
  "bid": 189.12,
  "ask": 189.15,
  "bid_size": 100,
  "ask_size": 200,
  "last": 189.13,
  "last_size": 50,
  "source": "vendor_x",
  "seq": 9823741
}

Include:

• exchange/source timestamp
• ingest timestamp
• sequence number
• symbol mapping
• currency / venue if relevant

c) Handle symbol mapping carefully

You’ll need a symbol master:

• canonical symbol
• exchange-specific symbol
• corporate actions / splits
• venue-specific identifiers
• lot size / tick size

This is critical for correct valuation.

---

4) Stream the data internally

For low latency, don’t make every consumer call the vendor directly. Use an internal stream.

Good options

• NATS: very low latency, simple pub/sub
• Redpanda/Kafka: durable event log, great for replay
• Pulsar: scalable but more operational overhead

Recommendation

• If you want ultra-low latency and simple pub/sub, start with NATS
• If you need durable replay and analytics pipelines, use Kafka or Redpanda
• Many teams use:
  • NATS for live fan-out
  • Kafka/Redpanda for persistence and replay

---

5) Maintain a real-time quote cache

Use an in-memory cache for the latest quote per symbol.

Common choices

• Redis: easy, fast enough for many systems
• Aerospike: very fast, scalable
• in-process memory + replication: fastest, but more complex

Store:

• latest bid/ask/last
• timestamp
• sequence number
• venue/source
• precomputed fields like mid, spread

Example derived values:

• mid = (bid + ask) / 2
• spread = ask - bid
• mark = mid or last trade depending on your valuation policy

For portfolio valuation, use a clear pricing policy:

• If bid/ask exists: use mid
• If only last trade exists: use last
• If stale beyond threshold: mark as stale and flag it

---

6) Build the portfolio valuation engine

This engine subscribes to quote updates and recalculates only affected positions.

Data model

For each portfolio:

• positions: symbol → quantity
• optional cost basis
• currency
• instrument type

Valuation logic

When a quote update arrives for AAPL:

1. find all portfolios holding AAPL
2. compute new market value:
   • position_value = qty * mark_price
3. update:
   • total portfolio value
   • unrealized P&L
   • exposure by sector/asset class
   • risk metrics if needed

Make it incremental

Do not recalculate the entire portfolio on every tick unless it’s tiny.

Use:

• symbol-to-position index
• symbol-to-portfolio reverse index
• incremental aggregation

This keeps latency low.

---

7) Charting analytics pipeline

For charting, you usually don’t want every raw tick in the UI. Instead generate:

• 1s bars
• 1m bars
• 5m bars
• optional indicators:
  • VWAP
  • moving averages
  • RSI
  • volume

Approach

1. ingest ticks
2. aggregate into OHLCV bars
3. publish updated bars to clients
4. store historical bars separately

You can compute bars in-stream using:

• stream processors
• a lightweight aggregation service
• Kafka Streams / Flink / Materialize if needed

---

8) API design

A. Streaming endpoint

Use one of:

• WebSocket
• Server-Sent Events (SSE)
• gRPC streaming

For market data and live portfolio updates, WebSockets are the most common for browsers.

Example WebSocket messages:

Subscribe

{
  "type": "subscribe",
  "symbols": ["AAPL", "MSFT", "SPY"],
  "channels": ["quotes", "bars", "portfolio"]
}

Quote update

{
  "type": "quote",
  "symbol": "AAPL",
  "bid": 189.12,
  "ask": 189.15,
  "last": 189.13,
  "ts": 1730000000123
}

Portfolio update

{
  "type": "portfolio_value",
  "portfolio_id": "p123",
  "total_value": 1054321.88,
  "daily_pnl": 1234.56,
  "ts": 1730000000155
}

Bar update

{
  "type": "bar",
  "symbol": "AAPL",
  "interval": "1m",
  "open": 189.00,
  "high": 189.30,
  "low": 188.95,
  "close": 189.13,
  "volume": 54210,
  "ts": 1730000060000
}

---

9) Reduce latency further

Key techniques

• keep hot paths in memory
• avoid synchronous database writes in the update path
• batch non-critical writes
• use binary serialization if possible:
  • MessagePack
  • Protobuf
  • FlatBuffers
• compress only if network bandwidth is the bottleneck
• colocate ingestion, cache, and valuation services in the same region/AZ
• use connection pooling and persistent sockets
• avoid per-tick heavy computation

Important

If you’re serving browser clients:

• WebSocket + JSON is easiest
• gRPC streaming is faster but less browser-friendly

---

10) Persistence strategy

You need both:

• latest state
• historical replay

Store:

• raw ticks: object storage or log system
• normalized quotes: Kafka/Redpanda
• bars: TimescaleDB / ClickHouse
• portfolio snapshots: Postgres or a time-series store

Recommended stores

• Redis: latest quote cache
• ClickHouse: fast analytics on historical ticks/bars
• Postgres: portfolios, users, permissions
• S3/GCS: raw archives

---

11) Reliability and correctness

Add these controls

• sequence gap detection
• stale quote detection
• feed reconnect logic
• duplicate suppression
• idempotent event handling
• fallback pricing rules
• market hours awareness
• corporate action adjustments
• per-symbol throttling if data is too noisy

Health metrics

Track:

• ingest lag
• quote age
• dropped messages
• cache hit rate
• portfolio recompute time
• websocket client count
• fan-out latency

---

12) Security and access control

For a production finance API:

• authenticate clients with JWT/OAuth2
• authorize symbol-level access if needed
• restrict entitlements by vendor agreement
• audit subscription changes
• encrypt data in transit with TLS
• rate-limit public endpoints

---

13) Suggested implementation stack

A solid pragmatic stack:

Option A: Simple and fast to ship

• Ingestion: Python or Go
• Stream bus: NATS or Redpanda
• Cache: Redis
• API: Go or Node.js WebSocket server
• Historical storage: ClickHouse
• Portfolio DB: Postgres

Option B: Higher-scale streaming system

• Ingestion: Go/C++
• Bus: Kafka/Redpanda
• Stream processing: Flink/Kafka Streams
• Cache: Redis/Aerospike
• API gateway: Go
• Storage: ClickHouse + S3 + Postgres

---

14) Minimal flow example

1. Vendor feed delivers AAPL quote
2. Adapter normalizes it
3. Publish to internal stream
4. Update latest quote cache
5. Recompute impacted portfolios
6. Update chart bars
7. Push updates to subscribed WebSocket clients

This gives:

• low latency
• reactivity
• replayability
• easy scaling

---

15) Practical recommendations

If you’re starting from scratch:

• Use WebSockets for client delivery
• Use Redis for latest quote cache
• Use NATS or Redpanda internally
• Use Go for low-latency services
• Use ClickHouse for historical analytics
• Compute valuation incrementally, not by full recompute
• Use mid-price for valuation unless your policy dictates otherwise

---

16) Example service breakdown

You can split into microservices:

• feed-service: vendor adapters
• quote-normalizer: canonical schema and symbol mapping
• quote-cache: latest state store
• valuation-service: portfolio pricing
• bar-aggregator: OHLCV generation
• stream-api: WebSocket/SSE fan-out
• history-service: query past data

---

If you want, I can also provide:

1. a reference architecture diagram
2. a Go or Python implementation skeleton
3. a WebSocket API design
4. a Redis + Kafka/Redpanda schema
5. a production checklist for market-data systems