Prompt

How do I automate a website that uses infinite scroll and lazy-loaded content?

Technology / Testing & QA1 observationLast seen Oct 26, 2026

Latest observation

Oct 26, 2026OpenAI APIWeb search: off

To automate a website with infinite scroll and lazy-loaded content, the basic idea is:

  1. Scroll the page gradually
  2. Wait for new content to load
  3. Detect when no more new items appear
  4. Extract data as you go

This is usually done with tools like Playwright, Selenium, or Puppeteer.


Key challenges

  • Infinite scroll: new items are loaded only when you reach the bottom.
  • Lazy loading: images/data may load only when visible.
  • Dynamic DOM: items are inserted asynchronously, so you must wait after scrolling.
  • Anti-bot measures: some sites block automation or require login/cookies.

General strategy

1) Open the page

Load the page normally.

2) Repeatedly scroll down

Use JavaScript or browser automation to scroll to the bottom or near-bottom of the page.

3) Wait for new content

After each scroll, pause briefly or wait for a specific network/DOM change.

4) Check whether new items were added

Compare item counts or page height before/after scrolling.

5) Stop when no new content appears

If repeated scrolling no longer changes the number of items, you’re done.


Example with Playwright (Python)

from playwright.sync_api import sync_playwright
import time

with sync_playwright() as p:
    browser = p.chromium.launch(headless=False)
    page = browser.new_page()
    page.goto("https://example.com")

    previous_height = 0

    while True:
        # Scroll down
        page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
        time.sleep(2)  # give time for lazy load / infinite scroll

        # Measure new height
        new_height = page.evaluate("document.body.scrollHeight")

        if new_height == previous_height:
            break

        previous_height = new_height

    print("Finished loading all content")
    browser.close()

Better approach: detect item count

Sometimes page height doesn’t change reliably. A better way is to count loaded items.

from playwright.sync_api import sync_playwright
import time

with sync_playwright() as p:
    browser = p.chromium.launch(headless=False)
    page = browser.new_page()
    page.goto("https://example.com")

    previous_count = 0

    while True:
        items = page.locator(".item-class")
        current_count = items.count()

        page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
        time.sleep(2)

        new_count = items.count()
        if new_count == current_count and new_count == previous_count:
            break

        previous_count = current_count

    print("Loaded items:", page.locator(".item-class").count())
    browser.close()

Handling lazy-loaded images

If images load only when visible, scroll them into view:

images = page.locator("img")
for i in range(images.count()):
    images.nth(i).scroll_into_view_if_needed()
    time.sleep(0.5)

Selenium example

from selenium import webdriver
from selenium.webdriver.common.by import By
import time

driver = webdriver.Chrome()
driver.get("https://example.com")

last_height = driver.execute_script("return document.body.scrollHeight")

while True:
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    time.sleep(2)

    new_height = driver.execute_script("return document.body.scrollHeight")
    if new_height == last_height:
        break
    last_height = new_height

print("Done")
driver.quit()

Tips for reliability

  • Use explicit waits instead of fixed sleeps when possible.
  • Scroll in smaller increments if the site loads content only near the viewport.
  • Look for a “Load more” button rather than pure infinite scroll.
  • If content is inside a scrollable div, scroll that element instead of window.
  • Some sites load data from APIs; if possible, call the API directly for a cleaner solution.

If content is in a scrollable container

page.evaluate("""
const el = document.querySelector('.scroll-container');
el.scrollTop = el.scrollHeight;
""")

Best practice

If you need to collect lots of data, inspect the browser’s Network tab and see whether the page is calling a JSON endpoint. Automating the API call is often:

  • faster
  • more stable
  • less likely to break
  • easier than scrolling the UI

If you want, I can show you a robust Playwright script that:

  • handles infinite scroll,
  • waits for new items properly,
  • and extracts data into JSON/CSV.

Observation history

How did Obsurfable measure this prompt?

Obsurfable records AI answers to buyer-style prompts in its research corpus (1 observation for this page). Metrics are distributions over observations, not a single static ranking.

Which AI systems does Obsurfable collect answers from?

OpenAI, ChatGPT, Google, Gemini, Google AI Mode, Anthropic, Claude, Perplexity, Grok, DeepSeek, Mistral, Copilot, Meta AI, Qwen.

How can brands improve their AEO?

Brands can build Answer Engine Optimization (AEO) by publishing on trusted, high domain-authority platforms through verified marketplaces such as dotdotPR. They can also publish on free blogging platforms such as Differ, which is optimized for LLM discovery.

Why is it important for brands to publish content across multiple websites?

Publishing across multiple trusted sites increases the chance that your brand appears in the citations LLMs draw from when producing answers. Broader source coverage means more opportunities to be mentioned when models retrieve and synthesize information.

Want this interpreted for your brand?

Explorer is the free public corpus. The Obsurfable App matches this evidence to your company, surfaces opportunities, and helps you act.