How do I set up a tick data platform for pricing portfolios and monitoring market moves during market hours?

Setting up a tick-data platform for portfolio pricing and live market monitoring is mostly about combining:

1. reliable market data ingestion,
2. normalized storage,
3. fast pricing/risk services, and
4. real-time monitoring/alerting.

Here’s a practical blueprint.

---

1) Define the use case clearly

Before choosing tools, decide:

• Asset classes: equities, options, futures, FX, rates, crypto?
• Data types:
  • trades
  • top-of-book quotes
  • full depth/order book
  • reference data
  • corporate actions
  • FX rates / curves / vol surfaces
• Latency target:
  • near-real-time dashboarding (seconds)
  • intraday pricing/risk (sub-second to seconds)
  • low-latency trading (milliseconds or less)
• Coverage:
  • only live market hours?
  • pre/post-market?
  • historical replay/backtesting?

This affects architecture and cost a lot.

---

2) Core architecture

A common setup looks like this:

Market data vendors / exchanges
→ Ingestion layer
→ Normalization + validation
→ Message bus / stream
→ Tick store + reference store
→ Pricing engine / risk engine
→ Dashboard + alerting + APIs

Key components

A. Ingestion

Connect to:

• exchange feeds or vendor feeds (Bloomberg, Refinitiv, ICE, CME, NYSE, Nasdaq, Polygon, etc.)
• internal pricing sources
• reference/master data feeds

Responsibilities:

• reconnect logic
• sequence-gap detection
• deduplication
• timestamping
• failover to backup feed if needed

B. Normalization

Convert all feeds into a common schema:

• instrument ID
• timestamp (exchange time + receipt time)
• bid/ask/last size/price
• trade condition flags
• venue/source
• currency
• corporate action adjustments if applicable

Use a canonical symbol mapping service:

• ticker → internal instrument_id
• maintain mapping across vendors and contract rolls

C. Stream layer

Use a pub/sub or streaming platform for live distribution:

• Kafka
• Redpanda
• NATS
• Pulsar

This lets pricing services, dashboards, and alerting subscribe independently.

D. Storage

You typically want two stores:

1. Hot store for recent ticks and fast queries
   • ClickHouse, QuestDB, TimescaleDB, kdb+
2. Cold/object store for long-term archive
   • S3 / GCS / Azure Blob in Parquet format

For market-hours monitoring, the hot store matters most.

E. Pricing/risk engine

This service consumes live ticks and computes:

• latest prices / mid / VWAP
• portfolio PV
• P&L
• Greeks / sensitivities
• exposure and stress moves
• breach checks

For speed, cache:

• latest market state
• reference curves/vols
• portfolio positions
• instrument metadata

F. Dashboard and alerts

Build a live dashboard that shows:

• market movers
• portfolio value
• top P&L contributors
• bid/ask changes
• unusual volume
• threshold breaches
• stale prices / feed issues

Alerting can go to:

• Slack / Teams
• email
• pager/incident system

---

3) Data model you should store

At minimum, keep these entities:

Market data ticks

• instrument_id
• event_time
• receive_time
• bid_price
• bid_size
• ask_price
• ask_size
• last_price
• last_size
• trade_id
• venue
• sequence_num
• flags

Reference data

• symbol
• asset class
• exchange
• currency
• multiplier
• tick size
• expiry
• strike
• option type
• underlying
• corporate action adjustments

Portfolio data

• account / book
• position
• average cost
• risk factors
• strategy / desk
• limits

Derived data

• mid price
• spread
• intraday return
• volatility estimate
• VWAP
• live P&L
• Greeks
• factor exposures

---

4) Pricing portfolios correctly

Pricing portfolios from tick data depends on the asset class.

Equities

• price with last traded price or mid price
• use bid/ask for conservative valuation
• corporate actions matter a lot

Options

• derive underlying mid price
• pull vol surface / implied vols
• price with Black-Scholes, binomial, or more advanced models
• recompute Greeks on every meaningful market move

Futures

• price using last or mid
• handle contract rolls
• multipliers and expiration are critical

Bonds / rates / OTC

• often need curves, quotes, and interpolation
• tick data alone is not enough; need curve updates and reference pricing logic

Practical rule

For intraday portfolio monitoring:

• use mid for neutral valuation
• use bid/ask if you need conservative or executable marks
• store both, plus the selected mark policy

---

5) Monitoring market moves during market hours

You’ll usually want real-time features like:

• top gainers/losers by % move
• largest spread widening
• largest volume spikes
• biggest portfolio contributors to P&L
• instruments with stale quotes
• instruments with large jumps vs. previous close
• correlation/factor move detection
• market regime shift alerts

Common calculations

• return_1m, return_5m, return_intraday
• z-score vs. recent history
• spread_bps
• volume_vs_average
• price_change_vs_prev_close
• P&L_by_position, P&L_by_sector

Alert examples

• “Position X moved > 2% in 5 minutes”
• “Bid/ask spread widened 3x”
• “Price stale for > 30 seconds”
• “Portfolio P&L down > $250k intraday”
• “Feed sequence gaps detected”

---

6) Suggested technology stack

A solid modern stack could be:

Ingestion

• Python for quick development
• C++/Java/Go for low-latency feed handling

Stream processing

• Kafka / Redpanda
• Flink / Spark Streaming / Kafka Streams

Storage

• ClickHouse for fast tick analytics
• PostgreSQL for reference/portfolio data
• S3 + Parquet for archives

Pricing service

• Python for quant logic, if latency is acceptable
• C++ if you need speed
• Redis for caching live latest values

Visualization

• Grafana
• Superset
• custom React dashboard

Orchestration/ops

• Docker + Kubernetes
• Airflow / Dagster for batch jobs
• Prometheus + Loki + OpenTelemetry for observability

---

7) Critical design considerations

A. Timestamping

Always store:

• exchange/event timestamp
• receive timestamp
• processing timestamp

This helps detect feed delays and measure latency.

B. Symbol mapping

This is one of the hardest parts. Maintain a master instrument service with:

• corporate actions
• ticker changes
• contract rolls
• exchange-specific identifiers

C. Data quality

Implement checks:

• missing ticks
• out-of-order ticks
• impossible prices
• duplicate messages
• crossed markets
• stale values

D. Resilience

You need:

• multiple feeds if possible
• replay/recovery
• idempotent processing
• backfill capability from historical data

E. Performance

For live monitoring:

• keep the latest state in memory
• update derived metrics incrementally
• avoid querying raw ticks for every dashboard refresh

---

8) A practical implementation pattern

A good pattern is:

1. Ingest raw feed
2. Normalize into canonical tick format
3. Publish to Kafka
4. Persist raw + normalized tick
5. Update in-memory latest price cache
6. Revalue portfolio positions
7. Publish derived metrics
8. Render dashboard + trigger alerts

This keeps the system scalable and reduces coupling.

---

9) Build vs buy

Build if:

• you need custom portfolio valuation
• you cover niche asset classes
• you need control over data quality and latency

Buy if:

• you mainly need market data distribution and charts
• you don’t want to manage exchange entitlements and feed handlers
• you need faster time-to-market

Many firms do a hybrid:

• buy market data
• build internal pricing/risk and monitoring layer

---

10) Minimal viable architecture

If you want something simple to start:

• Feed source: vendor API
• Ingest service: Python/Go
• Message bus: Kafka or Redpanda
• Storage: ClickHouse + PostgreSQL
• Pricing: Python service with cached latest prices
• Dashboard: Grafana or custom web app
• Alerts: Slack/email

This is enough for intraday pricing and monitoring for many desks.

---

11) Recommended rollout plan

Phase 1

• ingest live ticks for a small universe
• store normalized ticks
• build latest-price cache
• show live dashboard

Phase 2

• add portfolio positions
• compute live mark-to-market and P&L
• add alerts for stale feeds and big movers

Phase 3

• add options/curves/vol surfaces
• add historical replay/backtesting
• add failover feeds and stronger data quality controls

---

12) Questions to answer before implementation

If you want the right design, answer these:

1. Which asset classes?
2. How many instruments?
3. How many ticks/sec at peak?
4. How fresh do valuations need to be?
5. Do you need order book depth or just trades/quotes?
6. Do you need historical replay?
7. What’s your preferred cloud/on-prem setup?
8. What portfolio models do you need?

---

If you want, I can turn this into one of these next:

1. a reference architecture diagram,
2. a database schema for tick + portfolio data, or
3. a Python/Kafka/ClickHouse starter implementation.