What's the best way to scrape product availability?

The “best” way depends on where the data comes from and how fresh/accurate it needs to be. In general, the best order is:

1. Use an official API or feed if available
2. Use structured data embedded in the page
3. Only then scrape the HTML
4. If the site is JS-heavy, capture the underlying network/API calls rather than the rendered page

Recommended approach

1) Prefer APIs, feeds, or webhooks

If the retailer/platform offers:

• product APIs
• inventory endpoints
• partner feeds
• RSS/feeds
• webhooks

…use those. They’re usually:

• more stable than scraping
• faster
• easier to maintain
• less likely to break
• more compliant with site terms

2) Check for structured data in the page

Many product pages expose availability in:

• schema.org JSON-LD
• meta tags
• embedded state objects

Look for values like:

• InStock
• OutOfStock
• LimitedAvailability
• PreOrder
• BackOrder

This is often the cleanest scraping path if no API exists.

3) Scrape the underlying network requests

If the page is built with React/Vue/Next/etc., the visible availability may come from:

• XHR/fetch requests
• GraphQL calls
• JSON endpoints
• __NEXT_DATA__, hydration payloads, etc.

Use browser devtools to inspect network traffic and find the actual inventory request. Scraping that endpoint is usually much better than parsing the final DOM.

4) Fall back to HTML parsing

If you must scrape the page:

• fetch the page
• parse the DOM
• extract the availability text or badge
• normalize it to a small set of statuses

Use a robust parser and avoid brittle selectors when possible.

What to watch out for

Dynamic inventory

Availability can vary by:

• location/store
• size/color/variant
• shipping vs pickup
• logged-in user
• time of day

So define exactly what “available” means:

• in stock online
• available for pickup
• available at a specific store
• available in a specific variant

Anti-bot measures

Sites may use:

• rate limits
• CAPTCHAs
• bot detection
• signed requests
• session cookies

Be careful and respect the site’s terms, robots policy, and applicable laws.

Data quality

Product availability changes quickly. Best practices:

• cache briefly
• recheck before acting on stale data
• monitor for parsing failures
• log raw responses for debugging

A practical strategy

If you’re building this for production:

• Step 1: identify all target sites
• Step 2: see whether they expose APIs or JSON-LD
• Step 3: build one extractor per site/source type
• Step 4: normalize statuses into a common schema
• Step 5: add monitoring and retries
• Step 6: schedule frequent refreshes for high-churn items

Example normalized output:

{
  "product_id": "12345",
  "variant_id": "red-large",
  "availability": "in_stock",
  "quantity": null,
  "source": "json_ld",
  "checked_at": "2026-07-15T12:00:00Z"
}

Rule of thumb

• Best overall: official API/feed
• Best scraping method: underlying JSON/API endpoint
• Best HTML method: structured data first, DOM fallback second

If you want, I can help you design a scraper for a specific site or show a Python example for extracting availability from JSON-LD or a network endpoint.