Prompt
How to extract product data from multiple domains
Latest observation
Nov 20, 2026
OpenAI APIWeb search: off
There are a few good ways to extract product data from multiple domains, depending on scale, legality, and whether you need real-time or batch data.
1) Best options by use case
A. Use official APIs if available
Many e-commerce platforms and marketplaces expose APIs for products, pricing, inventory, etc.
Pros
- Most reliable
- Clean structured data
- Less likely to break
- Usually allowed
Cons
- Rate limits
- Requires auth
- Not every site has one
Examples:
- Shopify Storefront/Admin APIs
- WooCommerce REST API
- Magento APIs
- Amazon/eBay/Walmart partner or seller APIs
B. Web scraping
If there’s no API, scrape the product pages.
Typical data to extract
- Product name
- Price
- Currency
- SKU / model
- Availability
- Description
- Images
- Categories
- Ratings/reviews
Common data sources on pages
- HTML elements
- JSON-LD structured data (
<script type="application/ld+json">) - Open Graph tags
- Embedded JS state objects
C. Use feeds or sitemaps
Some sites publish:
- XML sitemaps
- Product feeds
- RSS / CSV exports
These can be easier than scraping page-by-page.
2) Architecture for multiple domains
If you need to extract data from many domains, use a modular pipeline:
Step 1: Discover URLs
- Crawl site sitemap
- Follow category pages
- Use known product listing pages
Step 2: Classify site type
Determine whether the domain is:
- Shopify
- WooCommerce
- Magento
- Custom site
- Marketplace
This helps decide extraction logic.
Step 3: Use a domain-specific extractor
Create a scraper template per domain or platform:
- Extract from JSON-LD if present
- Fallback to CSS/XPath selectors
- Fallback to browser automation for JS-rendered pages
Step 4: Normalize data
Map all products into a common schema:
{
"domain": "",
"url": "",
"name": "",
"price": "",
"currency": "",
"sku": "",
"availability": "",
"description": "",
"images": [],
"brand": "",
"category": ""
}
Step 5: Store and deduplicate
Use:
- PostgreSQL / MySQL for structured storage
- MongoDB for flexible schemas
- Deduping by URL, SKU, GTIN, or product title + brand
3) Recommended tools
Python
requests/httpxfor HTTPBeautifulSoup/lxmlfor parsing HTMLscrapyfor crawling large numbers of pagesplaywrightorseleniumfor JavaScript-heavy sitespydanticfor data validation
For scale
- Scrapy + Redis queue
- Celery / Kafka for distributed jobs
- Playwright on a worker pool for dynamic pages
4) Extraction strategy
Prefer this order:
- API
- JSON-LD
- Meta tags / embedded data
- HTML selectors
- Headless browser rendering
Example JSON-LD product data often looks like:
<script type="application/ld+json">
{
"@type": "Product",
"name": "Example Shoe",
"offers": {
"@type": "Offer",
"price": "79.99",
"priceCurrency": "USD",
"availability": "https://schema.org/InStock"
}
}
</script>
5) Handling many domains safely
- Respect
robots.txt - Add delays and retries
- Use realistic user-agent headers
- Avoid aggressive concurrency
- Handle CAPTCHA or blocks appropriately
- Check legal/ToS requirements
6) Example approach in Python
A simple pattern is:
- Fetch page
- Parse JSON-LD
- If missing, parse HTML selectors
- Save normalized record
If you want, I can provide:
- a Python script for multi-domain product scraping,
- a Scrapy project structure,
- or a database schema + pipeline for storing product data.