How do I collect product prices from multiple retailers?

There are a few common ways to collect product prices from multiple retailers, depending on scale, reliability, and whether the retailers allow automated access.

1) Use official APIs when available

This is the best option.

• Many retailers, marketplaces, and affiliate programs offer APIs.
• You can query product info, price, availability, and sometimes shipping costs.
• Advantages: more stable, cleaner data, less likely to break or violate terms.
• Examples:
  • Amazon Product Advertising API
  • eBay APIs
  • Walmart, Best Buy, Target, and others depending on region/programs

2) Use affiliate feeds or product data feeds

Some retailers provide downloadable product feeds.

• Usually CSV, XML, or JSON feeds
• Often include SKU, price, stock, and URLs
• Good for bulk price monitoring
• Common in affiliate networks like:
  • Impact
  • CJ Affiliate
  • Rakuten
  • Awin

3) Web scraping

If no API/feed exists, you can scrape the public product pages.

Typical process:

1. Identify product URLs.
2. Fetch HTML pages.
3. Parse the price from the page.
4. Normalize the data into a common schema.
5. Schedule repeated collection.

Tools people use:

• Python: requests, BeautifulSoup, lxml, scrapy
• For JavaScript-heavy sites: Playwright or Selenium

Important:

• Check the site’s Terms of Service and robots.txt
• Respect rate limits
• Avoid bypassing anti-bot protections
• Scraping can be fragile because page layouts change often

4) Use price comparison or data aggregation services

Some services already collect retailer pricing and offer APIs or exports.

• Good if you want speed and reliability without building everything yourself
• May cost money
• Examples depend on your region and use case

5) Set up a price tracking pipeline

A practical architecture:

• Source list: store product URLs or IDs for each retailer
• Collector: API client or scraper
• Normalizer: convert all prices to the same currency and format
• Database: save current and historical prices
• Scheduler: run hourly/daily
• Alerting: notify when a price changes or drops below threshold

Suggested fields to store:

• retailer
• product_id / SKU
• product_name
• current_price
• currency
• availability
• shipping_price
• collected_at
• product_url

6) Handle common issues

• Different currencies: convert using exchange rates if needed
• Different package sizes: normalize to unit price when comparing
• Promotions/coupons: distinguish list price vs sale price
• Dynamic pricing: capture timestamps so you know when the price was seen
• Stock/availability: price may not be meaningful if out of stock

7) Best practice recommendation

If you’re starting out:

1. Use APIs/feeds where possible
2. Scrape only for retailers without an official data source
3. Build a small database to track history
4. Monitor for changes and failures

If you want, I can also help you with:

• a Python scraper template
• a price tracking database schema
• a comparison of API vs scraping
• a step-by-step system design for collecting prices at scale