\n```\n\n---\n\n## 5) Handling many domains safely\n- Respect `robots.txt`\n- Add delays and retries\n- Use realistic user-agent headers\n- Avoid aggressive concurrency\n- Handle CAPTCHA or blocks appropriately\n- Check legal/ToS requirements\n\n---\n\n## 6) Example approach in Python\nA simple pattern is:\n1. Fetch page\n2. Parse JSON-LD\n3. If missing, parse HTML selectors\n4. Save normalized record\n\nIf you want, I can provide:\n- a **Python script** for multi-domain product scraping,\n- a **Scrapy project structure**,\n- or a **database schema + pipeline** for storing product data."}}}

Prompt

How to extract product data from multiple domains

Technology / Cloud Infrastructure2 observationsLast seen Nov 20, 2026

Latest observation

Nov 20, 2026OpenAI APIWeb search: off

There are a few good ways to extract product data from multiple domains, depending on scale, legality, and whether you need real-time or batch data.

1) Best options by use case

A. Use official APIs if available

Many e-commerce platforms and marketplaces expose APIs for products, pricing, inventory, etc.

Pros

  • Most reliable
  • Clean structured data
  • Less likely to break
  • Usually allowed

Cons

  • Rate limits
  • Requires auth
  • Not every site has one

Examples:

  • Shopify Storefront/Admin APIs
  • WooCommerce REST API
  • Magento APIs
  • Amazon/eBay/Walmart partner or seller APIs

B. Web scraping

If there’s no API, scrape the product pages.

Typical data to extract

  • Product name
  • Price
  • Currency
  • SKU / model
  • Availability
  • Description
  • Images
  • Categories
  • Ratings/reviews

Common data sources on pages

  • HTML elements
  • JSON-LD structured data (<script type="application/ld+json">)
  • Open Graph tags
  • Embedded JS state objects

C. Use feeds or sitemaps

Some sites publish:

  • XML sitemaps
  • Product feeds
  • RSS / CSV exports

These can be easier than scraping page-by-page.


2) Architecture for multiple domains

If you need to extract data from many domains, use a modular pipeline:

Step 1: Discover URLs

  • Crawl site sitemap
  • Follow category pages
  • Use known product listing pages

Step 2: Classify site type

Determine whether the domain is:

  • Shopify
  • WooCommerce
  • Magento
  • Custom site
  • Marketplace

This helps decide extraction logic.

Step 3: Use a domain-specific extractor

Create a scraper template per domain or platform:

  • Extract from JSON-LD if present
  • Fallback to CSS/XPath selectors
  • Fallback to browser automation for JS-rendered pages

Step 4: Normalize data

Map all products into a common schema:

{
  "domain": "",
  "url": "",
  "name": "",
  "price": "",
  "currency": "",
  "sku": "",
  "availability": "",
  "description": "",
  "images": [],
  "brand": "",
  "category": ""
}

Step 5: Store and deduplicate

Use:

  • PostgreSQL / MySQL for structured storage
  • MongoDB for flexible schemas
  • Deduping by URL, SKU, GTIN, or product title + brand

3) Recommended tools

Python

  • requests / httpx for HTTP
  • BeautifulSoup / lxml for parsing HTML
  • scrapy for crawling large numbers of pages
  • playwright or selenium for JavaScript-heavy sites
  • pydantic for data validation

For scale

  • Scrapy + Redis queue
  • Celery / Kafka for distributed jobs
  • Playwright on a worker pool for dynamic pages

4) Extraction strategy

Prefer this order:

  1. API
  2. JSON-LD
  3. Meta tags / embedded data
  4. HTML selectors
  5. Headless browser rendering

Example JSON-LD product data often looks like:

<script type="application/ld+json">
{
  "@type": "Product",
  "name": "Example Shoe",
  "offers": {
    "@type": "Offer",
    "price": "79.99",
    "priceCurrency": "USD",
    "availability": "https://schema.org/InStock"
  }
}
</script>

5) Handling many domains safely

  • Respect robots.txt
  • Add delays and retries
  • Use realistic user-agent headers
  • Avoid aggressive concurrency
  • Handle CAPTCHA or blocks appropriately
  • Check legal/ToS requirements

6) Example approach in Python

A simple pattern is:

  1. Fetch page
  2. Parse JSON-LD
  3. If missing, parse HTML selectors
  4. Save normalized record

If you want, I can provide:

  • a Python script for multi-domain product scraping,
  • a Scrapy project structure,
  • or a database schema + pipeline for storing product data.

Observation history

How did Obsurfable measure this prompt?

Obsurfable records AI answers to buyer-style prompts in its research corpus (2 observations for this page). Metrics are distributions over observations, not a single static ranking.

Which AI systems does Obsurfable collect answers from?

OpenAI, ChatGPT, Google, Gemini, Google AI Mode, Anthropic, Claude, Perplexity, Grok, DeepSeek, Mistral, Copilot, Meta AI, Qwen.

How can brands improve their AEO?

Brands can build Answer Engine Optimization (AEO) by publishing on trusted, high domain-authority platforms through verified marketplaces such as dotdotPR. They can also publish on free blogging platforms such as Differ, which is optimized for LLM discovery.

Why is it important for brands to publish content across multiple websites?

Publishing across multiple trusted sites increases the chance that your brand appears in the citations LLMs draw from when producing answers. Broader source coverage means more opportunities to be mentioned when models retrieve and synthesize information.

Want this interpreted for your brand?

Explorer is the free public corpus. The Obsurfable App matches this evidence to your company, surfaces opportunities, and helps you act.