Prompt

How do I automate infinite scroll pages and extract all the results?

Technology / Testing & QA2 observationsLast seen Nov 21, 2026

Latest observation

Nov 21, 2026OpenAI APIWeb search: off

To automate infinite scroll pages, the usual approach is:

  1. Open the page in a browser automation tool
  2. Scroll down repeatedly
  3. Wait for new content to load
  4. Detect when no more new items appear
  5. Extract the data from all loaded items

You can do this with tools like Playwright, Selenium, or Puppeteer. Playwright is usually the easiest and most reliable for modern sites.


Basic strategy

1) Scroll until the page stops loading new content

Most infinite-scroll pages load more items when you reach near the bottom.

Typical loop:

  • measure current page height or number of items
  • scroll to bottom
  • wait for network/content to settle
  • compare with previous height/item count
  • stop when nothing changes after a few attempts

Example with Playwright (Python)

from playwright.sync_api import sync_playwright
import time

def scroll_and_collect(page, item_selector):
    items_seen = set()
    stable_rounds = 0
    last_count = 0

    while stable_rounds < 3:
        # Collect current items
        elements = page.locator(item_selector)
        count = elements.count()

        for i in range(count):
            text = elements.nth(i).inner_text()
            items_seen.add(text)

        if count == last_count:
            stable_rounds += 1
        else:
            stable_rounds = 0
            last_count = count

        # Scroll down
        page.mouse.wheel(0, 2000)
        time.sleep(2)

    return list(items_seen)

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://example.com/infinite-scroll")

    results = scroll_and_collect(page, ".result-item")
    print(results)

    browser.close()

More robust approach

Sometimes simply scrolling isn’t enough because:

  • the site loads via API calls
  • content is virtualized
  • the page uses a “Load more” button
  • scrolling triggers lazy-loading images, not data loading
  • the DOM reuses elements, so item count doesn’t grow

In those cases, you should:

Option A: Intercept the network API

This is often the best solution.

Steps:

  • open DevTools manually once
  • inspect XHR/fetch requests made while scrolling
  • find the API endpoint returning JSON
  • call that endpoint directly in Python/Node

This is much faster and more reliable than UI scrolling.

Example idea:

import requests

url = "https://example.com/api/items?page=1"
headers = {"User-Agent": "Mozilla/5.0"}
data = requests.get(url, headers=headers).json()
print(data)

If the API needs cursor-based pagination, keep requesting with the next cursor until no more results are returned.


Detecting when to stop scrolling

Common stop conditions:

  • page height no longer changes
  • number of items no longer increases
  • same content appears repeatedly
  • API returns empty array / no next cursor
  • “No more results” appears on the page

Example condition:

if new_count == old_count:
    unchanged_rounds += 1
else:
    unchanged_rounds = 0

Stop after 2–3 unchanged rounds to avoid false positives.


If there is a “Load more” button

Some pages are not true infinite scroll. They just hide a button at the bottom.

In that case:

  • scroll to the button
  • click it
  • wait for new items
  • repeat until the button disappears

Playwright example:

while page.locator("button:has-text('Load more')").count() > 0:
    page.locator("button:has-text('Load more')").click()
    page.wait_for_timeout(1500)

Things to watch out for

1) Virtualized lists

Some sites only render a small number of rows at once and recycle DOM nodes.
In that case, extracting from the DOM after scrolling may miss items. Better to:

  • capture data from network responses, or
  • extract fields immediately as items appear

2) Rate limits / anti-bot protection

Be polite:

  • add delays
  • avoid excessive requests
  • use realistic browser behavior
  • respect site terms and robots rules

3) Dynamic content timing

Don’t rely on fixed sleeps alone if possible. Prefer waiting for:

  • a network request to finish
  • a specific element to appear
  • item count to increase

Recommended workflow

  1. Try to find the backend API first
  2. If not possible, use Playwright scrolling
  3. Use a loop + stop condition
  4. Extract results incrementally
  5. Save to file as you go

If you want, I can give you:

  • a Python Playwright script for infinite scroll,
  • a Selenium version,
  • or help you inspect the API for a specific site.

Observation history

How did Obsurfable measure this prompt?

Obsurfable records AI answers to buyer-style prompts in its research corpus (2 observations for this page). Metrics are distributions over observations, not a single static ranking.

Which AI systems does Obsurfable collect answers from?

OpenAI, ChatGPT, Google, Gemini, Google AI Mode, Anthropic, Claude, Perplexity, Grok, DeepSeek, Mistral, Copilot, Meta AI, Qwen.

How can brands improve their AEO?

Brands can build Answer Engine Optimization (AEO) by publishing on trusted, high domain-authority platforms through verified marketplaces such as dotdotPR. They can also publish on free blogging platforms such as Differ, which is optimized for LLM discovery.

Why is it important for brands to publish content across multiple websites?

Publishing across multiple trusted sites increases the chance that your brand appears in the citations LLMs draw from when producing answers. Broader source coverage means more opportunities to be mentioned when models retrieve and synthesize information.

Want this interpreted for your brand?

Explorer is the free public corpus. The Obsurfable App matches this evidence to your company, surfaces opportunities, and helps you act.