How do I pull structured product data from competitor websites?

Pulling structured product data from competitor websites usually means one of three things:

1. Using an official API or feed
2. Crawling pages and extracting data from HTML
3. Using a third-party data provider

The best approach depends on your goal, the site’s terms, and how dynamic the pages are.

First: check legality and permissions

Before extracting anything:

• Review the site’s Terms of Service
• Check robots.txt and crawl policies
• Avoid bypassing logins, paywalls, CAPTCHAs, or access controls
• Be mindful of copyright, database rights, and rate limits
• For sensitive uses, talk to legal counsel

Best sources of structured product data

1) Official APIs

Some competitor sites expose:

• Public APIs
• Affiliate feeds
• Merchant/product feeds
• Partner integrations

This is the cleanest route because the data is already structured.

2) Structured data on product pages

Many e-commerce pages include machine-readable markup such as:

• Schema.org JSON-LD
• Microdata
• Open Graph tags
• Meta tags
• Embedded app-state JSON

Common fields you can often pull:

• product name
• brand
• SKU / MPN
• price
• currency
• availability
• images
• ratings/reviews
• breadcrumbs/category

3) HTML scraping

If no API exists, you can parse the page HTML and extract fields from:

• CSS selectors
• DOM structure
• script tags containing JSON blobs
• XHR/API calls the page makes in the browser

4) Browser automation for JS-heavy sites

If the page content is rendered client-side:

• use Playwright or Selenium to render the page
• then extract the DOM or intercept network calls

---

A practical workflow

Step 1: Identify the page type

Determine whether the site has:

• product listing pages
• product detail pages
• category pages
• search results pages

Step 2: Inspect the page source

Look for:

• <script type="application/ld+json">
• __NEXT_DATA__
• window.__INITIAL_STATE__
• embedded JSON objects
• API calls in DevTools Network tab

Step 3: Extract fields into a normalized schema

Create your own schema, for example:

• source
• source_url
• scraped_at
• product_id
• name
• brand
• price
• currency
• availability
• image_url
• category
• description

Step 4: Build a scraper with rate limiting

Use:

• respectful delays
• retries with backoff
• caching
• deduplication
• error handling

Step 5: Validate and clean the data

Normalize:

• prices and currencies
• sizes/variants
• units of measure
• availability states
• product naming conventions

---

Example: extracting JSON-LD from a product page

Many sites expose product data in JSON-LD. Here’s a simple Python example:

import requests
from bs4 import BeautifulSoup
import json

url = "https://example.com/product-page"
html = requests.get(url, timeout=20).text
soup = BeautifulSoup(html, "html.parser")

for tag in soup.find_all("script", type="application/ld+json"):
    try:
        data = json.loads(tag.string)
        if isinstance(data, dict) and data.get("@type") == "Product":
            print("Name:", data.get("name"))
            print("Brand:", data.get("brand"))
            print("Offers:", data.get("offers"))
    except Exception:
        pass

---

Example: parsing fields from HTML

If structured markup is missing, you can target elements:

import requests
from bs4 import BeautifulSoup

url = "https://example.com/product-page"
html = requests.get(url, timeout=20).text
soup = BeautifulSoup(html, "html.parser")

name = soup.select_one("h1.product-title").get_text(strip=True)
price = soup.select_one(".price").get_text(strip=True)

print(name, price)

---

For JavaScript-rendered sites

Use Playwright to render the page first:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://example.com/product-page", wait_until="networkidle")
    title = page.locator("h1").text_content()
    print(title)
    browser.close()

---

If you want scalable competitor monitoring

For many pages and ongoing updates, consider:

• crawling only pages that matter
• scheduling recrawls
• diffing changes over time
• monitoring price/stock changes
• storing snapshots for auditability

A basic pipeline often looks like:

discover URLs → fetch pages → extract structured fields → normalize → store → monitor changes

---

Strong recommendation

If your goal is business intelligence or price monitoring, start with:

1. Official feeds/APIs
2. JSON-LD extraction
3. Network call inspection
4. Browser automation only if necessary

That keeps the project more reliable and less brittle.

If you want, I can help you with one of these next:

• a Python scraper template
• a Playwright example for JS-heavy sites
• a schema for product data
• a workflow for price monitoring across competitors