Prompt
How do I automate infinite scroll pages and extract all the results?
Latest observation
To automate infinite scroll pages, the usual approach is:
- Open the page in a browser automation tool
- Scroll down repeatedly
- Wait for new content to load
- Detect when no more new items appear
- Extract the data from all loaded items
You can do this with tools like Playwright, Selenium, or Puppeteer. Playwright is usually the easiest and most reliable for modern sites.
Basic strategy
1) Scroll until the page stops loading new content
Most infinite-scroll pages load more items when you reach near the bottom.
Typical loop:
- measure current page height or number of items
- scroll to bottom
- wait for network/content to settle
- compare with previous height/item count
- stop when nothing changes after a few attempts
Example with Playwright (Python)
from playwright.sync_api import sync_playwright
import time
def scroll_and_collect(page, item_selector):
items_seen = set()
stable_rounds = 0
last_count = 0
while stable_rounds < 3:
# Collect current items
elements = page.locator(item_selector)
count = elements.count()
for i in range(count):
text = elements.nth(i).inner_text()
items_seen.add(text)
if count == last_count:
stable_rounds += 1
else:
stable_rounds = 0
last_count = count
# Scroll down
page.mouse.wheel(0, 2000)
time.sleep(2)
return list(items_seen)
with sync_playwright() as p:
browser = p.chromium.launch(headless=True)
page = browser.new_page()
page.goto("https://example.com/infinite-scroll")
results = scroll_and_collect(page, ".result-item")
print(results)
browser.close()
More robust approach
Sometimes simply scrolling isn’t enough because:
- the site loads via API calls
- content is virtualized
- the page uses a “Load more” button
- scrolling triggers lazy-loading images, not data loading
- the DOM reuses elements, so item count doesn’t grow
In those cases, you should:
Option A: Intercept the network API
This is often the best solution.
Steps:
- open DevTools manually once
- inspect XHR/fetch requests made while scrolling
- find the API endpoint returning JSON
- call that endpoint directly in Python/Node
This is much faster and more reliable than UI scrolling.
Example idea:
import requests
url = "https://example.com/api/items?page=1"
headers = {"User-Agent": "Mozilla/5.0"}
data = requests.get(url, headers=headers).json()
print(data)
If the API needs cursor-based pagination, keep requesting with the next cursor until no more results are returned.
Detecting when to stop scrolling
Common stop conditions:
- page height no longer changes
- number of items no longer increases
- same content appears repeatedly
- API returns empty array / no next cursor
- “No more results” appears on the page
Example condition:
if new_count == old_count:
unchanged_rounds += 1
else:
unchanged_rounds = 0
Stop after 2–3 unchanged rounds to avoid false positives.
If there is a “Load more” button
Some pages are not true infinite scroll. They just hide a button at the bottom.
In that case:
- scroll to the button
- click it
- wait for new items
- repeat until the button disappears
Playwright example:
while page.locator("button:has-text('Load more')").count() > 0:
page.locator("button:has-text('Load more')").click()
page.wait_for_timeout(1500)
Things to watch out for
1) Virtualized lists
Some sites only render a small number of rows at once and recycle DOM nodes.
In that case, extracting from the DOM after scrolling may miss items. Better to:
- capture data from network responses, or
- extract fields immediately as items appear
2) Rate limits / anti-bot protection
Be polite:
- add delays
- avoid excessive requests
- use realistic browser behavior
- respect site terms and robots rules
3) Dynamic content timing
Don’t rely on fixed sleeps alone if possible. Prefer waiting for:
- a network request to finish
- a specific element to appear
- item count to increase
Recommended workflow
- Try to find the backend API first
- If not possible, use Playwright scrolling
- Use a loop + stop condition
- Extract results incrementally
- Save to file as you go
If you want, I can give you:
- a Python Playwright script for infinite scroll,
- a Selenium version,
- or help you inspect the API for a specific site.