unhappy with browser scraping tools that fail on infinite scroll

If you’re unhappy with browser scraping tools that break on infinite scroll, the core issue is usually that they’re treating the page like a static document instead of a dynamic app.

What typically works better:

• Use a real browser automation layer that can scroll, wait for network activity, and observe DOM changes.
• Detect loading triggers instead of blindly scrolling:
  • scroll to bottom
  • wait for new items to appear
  • repeat until no new content loads
• Intercept API calls when possible. Infinite scroll UIs often fetch data from JSON endpoints, which is more reliable than scraping rendered HTML.
• Add termination logic:
  • no new items after N scrolls
  • repeated responses
  • end-of-feed marker
  • max time / max pages
• Handle lazy loading and virtualization:
  • some pages remove old DOM nodes as you scroll
  • you may need to extract data incrementally as it appears

If you want, I can help with any of these:

1. A robust infinite-scroll scraper design
2. A Playwright/Puppeteer example
3. A Python/Selenium example
4. A way to extract the underlying API instead of scraping the UI
5. Debugging a specific site or scraper that’s failing

If you share the site type or your current tool, I can suggest a more reliable approach.