How do I gather pricing data from ecommerce sites without constant bans?

To do this reliably, focus on being a good crawler and reducing load, not “evading” protections. The biggest causes of bans are high request rates, repetitive patterns, and ignoring site rules.

Safer ways to collect pricing data

1. Use official sources first

   • Public APIs, affiliate/product feeds, merchant APIs, sitemap feeds, or data partnerships.
   • Check robots.txt and site terms. If scraping is disallowed, don’t.

2. Throttle heavily

   • Low request rates per domain.
   • Randomize delays a bit, but don’t try to disguise abuse.
   • Avoid parallel bursts to the same site.

3. Cache aggressively

   • Store pages locally and only re-fetch when needed.
   • Use HTTP validators: ETag, If-None-Match, Last-Modified, If-Modified-Since.

4. Identify your crawler

   • Use a clear, honest User-Agent with contact info.
   • Some sites will whitelist or tolerate known bots.

5. Request less data

   • Prefer product JSON endpoints, structured data (schema.org), or page fragments if legitimately available.
   • Avoid fetching images, scripts, or unnecessary assets.

6. Be polite with retries

   • Handle 429/503 with exponential backoff.
   • Stop on repeated errors instead of hammering.

7. Distribute load responsibly

   • Spread requests across time, not just across IPs.
   • Multiple IPs to bypass bans is risky and often against terms.

8. Monitor block signals

   • Track status codes, CAPTCHA pages, HTML changes, and response times.
   • Reduce crawl rate automatically when you see warning signs.

Practical architecture

• Scheduler: decides when each product/page should be refreshed.
• Fetcher: low-concurrency HTTP client with caching and backoff.
• Parser: extracts price, currency, availability, and timestamp.
• Change detector: only reprocesses when content changes.
• Compliance layer: respects robots/terms and blocks prohibited targets.

A simple rule of thumb

If you’re getting banned, your crawl is usually:

• too fast,
• too repetitive,
• too broad,
• or targeting pages the site doesn’t want automated access to.

If you want, I can help you design a polite price-monitoring pipeline or show a sample crawler with rate limiting, caching, and backoff.