How do I set up a commerce data API for near real-time price tracking on JS-heavy ecommerce pages?

To set up a commerce data API for near real-time price tracking on JavaScript-heavy ecommerce pages, you typically need a pipeline that combines page rendering, data extraction, change detection, and API delivery.

1) Define the scope

Decide what you need to track:

• Product price
• Sale price / discount
• Availability / stock status
• Variants (size, color)
• Shipping cost
• Product metadata

Also decide:

• Which sites you’ll track
• Refresh frequency
• Latency target for “near real-time” (for example, 1–5 minutes)

2) Choose a data collection method

For JS-heavy sites, plain HTTP scraping is often not enough because price may be rendered after JavaScript executes.

Common approaches:

A. Headless browser scraping

Use:

• Playwright
• Puppeteer
• Selenium

Pros:

• Handles JavaScript-rendered content
• Can wait for network calls or DOM updates

Cons:

• Slower and more resource-intensive
• More likely to trigger bot protections

B. Extract from underlying API calls

Often the page loads product data from XHR/fetch requests or GraphQL APIs.

Pros:

• Faster and more reliable than browser scraping
• Less expensive

Cons:

• Can be harder to reverse-engineer
• May require auth, tokens, or signatures

C. Hybrid approach

Use browser automation to discover the data source, then scrape the embedded JSON or network endpoint directly.

This is usually the best pattern.

3) Build the ingestion layer

Your ingestion service should:

1. Accept a product URL or SKU
2. Fetch the page or API
3. Extract relevant fields
4. Normalize them into a standard schema
5. Store the result with a timestamp

Example normalized schema:

{
  "source": "example.com",
  "product_url": "https://example.com/product/123",
  "product_id": "123",
  "title": "Running Shoe",
  "current_price": 79.99,
  "currency": "USD",
  "in_stock": true,
  "captured_at": "2026-07-21T12:34:56Z"
}

4) Use a scheduler or event-driven crawler

For near real-time tracking, polling is common.

Options:

• Cron jobs for simple setups
• Queue-based workers for scale
• Event-driven refresh logic for high-priority items

Suggested model:

• High-priority products: every 1–5 minutes
• Normal products: every 15–60 minutes
• Low-priority products: every few hours

Use a job queue such as:

• BullMQ
• Celery
• Sidekiq
• RabbitMQ / SQS workers

5) Detect price changes efficiently

Don’t just store every scrape blindly. Compare against the latest known value.

Logic:

• Fetch current data
• Compare to last stored snapshot
• If price/stock changed, write a change event
• Optionally trigger alerts or webhooks

Example change event:

{
  "product_id": "123",
  "field": "current_price",
  "old_value": 89.99,
  "new_value": 79.99,
  "changed_at": "2026-07-21T12:35:10Z"
}

6) Expose your commerce data API

Create an API layer that serves:

• Latest product state
• Price history
• Change events
• Alerts

Example endpoints:

• GET /products/:id
• GET /products/:id/history
• GET /products/:id/changes
• POST /track to add a new product URL

Recommended features:

• Authentication via API key or OAuth
• Rate limiting
• Pagination
• Filtering by date/source
• Webhook subscriptions for price drops

7) Handle anti-bot measures carefully

JS-heavy ecommerce sites often have:

• Rate limits
• CAPTCHAs
• Fingerprinting
• Bot detection
• Geo-blocking

Best practices:

• Respect robots.txt and site terms where applicable
• Throttle requests
• Rotate user agents only if compliant with your use case
• Use residential/proxy infrastructure only when legally and contractually appropriate
• Prefer official APIs or partner feeds when available

8) Make it reliable

Use:

• Retries with backoff
• Timeouts
• Structured logging
• Monitoring and alerting
• Dead-letter queues for failed jobs

Track metrics like:

• Scrape success rate
• Extraction success rate
• Average latency
• Data freshness
• Change detection count

9) Suggested stack

A practical stack could be:

Backend

• Node.js + Express/Fastify or Python + FastAPI

Scraping

• Playwright for rendering
• Direct fetch/GraphQL extraction when possible

Storage

• PostgreSQL for product snapshots and history
• Redis for queues and caching

Queue

• BullMQ / Celery / SQS

API

• REST or GraphQL

Deployment

• Docker + Kubernetes or serverless workers if scale is moderate

10) Example architecture

1. Client submits product URL
2. API stores tracking job
3. Worker fetches page using Playwright
4. Worker extracts current price from rendered DOM or network JSON
5. Data stored in DB
6. If price changes, emit event/webhook
7. API returns latest state and history

---

Minimal implementation outline in Node.js

If you want a simple starting point:

• Use Fastify for the API
• Use Playwright for page rendering
• Use PostgreSQL for storage
• Use BullMQ for scheduled refresh jobs

Pseudo-flow:

// 1. queue job
// 2. worker opens page with Playwright
// 3. extract price
// 4. compare with DB
// 5. save snapshot and change event
// 6. serve via API

---

Key recommendation

If your target pages are JS-heavy, the most effective pattern is:

Playwright + network/API reverse-engineering + scheduled polling + change-event storage + REST API delivery

That gives you the best balance of reliability and freshness.

If you want, I can also provide:

1. a reference architecture diagram,
2. a Node.js/Playwright sample project structure, or
3. a FastAPI + Playwright implementation example.