What's the best way to collect competitor pricing data?

The best approach is usually a mix of automated collection + manual validation, with the exact method depending on how many competitors, how often prices change, and what sources are available.

Good options, from simplest to most scalable

1. Manual monitoring

Best if you only track a few competitors or a small set of products.

• Check competitor websites directly
• Record prices in a spreadsheet
• Use browser bookmarks, alerts, or saved searches

Pros: cheap, simple, low setup
Cons: slow, hard to scale, error-prone

2. Web scraping / site crawling

Best for large-scale, repeatable price tracking.

• Use scraping tools or custom scripts to pull product pages, category pages, or search results
• Extract price, stock status, discounts, shipping, SKU, and timestamps
• Store in a database or spreadsheet
• Schedule daily or hourly runs

Pros: scalable, automated, consistent
Cons: can break if site layout changes; may be restricted by robots.txt, terms of service, or anti-bot measures

3. Price intelligence / monitoring tools

Best if you want faster deployment without building your own system.

Examples of what these tools do:

• Track competitor prices
• Detect promotions and discounts
• Monitor out-of-stock changes
• Send alerts when prices change

Pros: easy to use, less maintenance
Cons: subscription cost, less customization

4. Marketplaces, feeds, and public sources

If competitors sell on marketplaces or publish feeds:

• Marketplace listings
• Public APIs
• Product feeds
• Emails/newsletters/promotions
• Affiliate/product comparison sites

Pros: often more reliable than scraping websites
Cons: may not cover everything you need

Best practice: combine methods

A strong setup usually looks like this:

1. Define the exact products/SKUs you want to monitor
2. Choose the most reliable source for each competitor
3. Automate collection
4. Normalize the data
   • price
   • currency
   • shipping
   • discounts
   • bundle differences
5. Validate samples manually
6. Set alerts for major changes
7. Track history over time, not just current prices

Data points worth collecting

To make competitor pricing useful, capture more than just the listed price:

• Product name
• SKU / model / variant
• Listed price
• Sale price
• Original price / strike-through price
• Shipping cost
• Availability / stock status
• Coupon or promo details
• Timestamp
• Source URL
• Region/currency

Important cautions

• Check website terms of service and robots.txt
• Avoid collecting personal data
• Use reasonable request rates to prevent overloading sites
• Be careful with dynamic or login-protected content

My recommendation

If you want the best balance of accuracy and scalability:

• Use automated scraping or a price monitoring platform
• Keep a manual review step for exceptions
• Store historical data
• Track total landed price rather than sticker price alone

If you want, I can also help you design a competitor pricing tracker based on your business size, number of competitors, and technical setup.