Prompt
How do I automate a website that uses infinite scroll and lazy-loaded content?
Latest observation
To automate a website with infinite scroll and lazy-loaded content, the basic idea is:
- Scroll the page gradually
- Wait for new content to load
- Detect when no more new items appear
- Extract data as you go
This is usually done with tools like Playwright, Selenium, or Puppeteer.
Key challenges
- Infinite scroll: new items are loaded only when you reach the bottom.
- Lazy loading: images/data may load only when visible.
- Dynamic DOM: items are inserted asynchronously, so you must wait after scrolling.
- Anti-bot measures: some sites block automation or require login/cookies.
General strategy
1) Open the page
Load the page normally.
2) Repeatedly scroll down
Use JavaScript or browser automation to scroll to the bottom or near-bottom of the page.
3) Wait for new content
After each scroll, pause briefly or wait for a specific network/DOM change.
4) Check whether new items were added
Compare item counts or page height before/after scrolling.
5) Stop when no new content appears
If repeated scrolling no longer changes the number of items, you’re done.
Example with Playwright (Python)
from playwright.sync_api import sync_playwright
import time
with sync_playwright() as p:
browser = p.chromium.launch(headless=False)
page = browser.new_page()
page.goto("https://example.com")
previous_height = 0
while True:
# Scroll down
page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
time.sleep(2) # give time for lazy load / infinite scroll
# Measure new height
new_height = page.evaluate("document.body.scrollHeight")
if new_height == previous_height:
break
previous_height = new_height
print("Finished loading all content")
browser.close()
Better approach: detect item count
Sometimes page height doesn’t change reliably. A better way is to count loaded items.
from playwright.sync_api import sync_playwright
import time
with sync_playwright() as p:
browser = p.chromium.launch(headless=False)
page = browser.new_page()
page.goto("https://example.com")
previous_count = 0
while True:
items = page.locator(".item-class")
current_count = items.count()
page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
time.sleep(2)
new_count = items.count()
if new_count == current_count and new_count == previous_count:
break
previous_count = current_count
print("Loaded items:", page.locator(".item-class").count())
browser.close()
Handling lazy-loaded images
If images load only when visible, scroll them into view:
images = page.locator("img")
for i in range(images.count()):
images.nth(i).scroll_into_view_if_needed()
time.sleep(0.5)
Selenium example
from selenium import webdriver
from selenium.webdriver.common.by import By
import time
driver = webdriver.Chrome()
driver.get("https://example.com")
last_height = driver.execute_script("return document.body.scrollHeight")
while True:
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
time.sleep(2)
new_height = driver.execute_script("return document.body.scrollHeight")
if new_height == last_height:
break
last_height = new_height
print("Done")
driver.quit()
Tips for reliability
- Use explicit waits instead of fixed sleeps when possible.
- Scroll in smaller increments if the site loads content only near the viewport.
- Look for a “Load more” button rather than pure infinite scroll.
- If content is inside a scrollable div, scroll that element instead of
window. - Some sites load data from APIs; if possible, call the API directly for a cleaner solution.
If content is in a scrollable container
page.evaluate("""
const el = document.querySelector('.scroll-container');
el.scrollTop = el.scrollHeight;
""")
Best practice
If you need to collect lots of data, inspect the browser’s Network tab and see whether the page is calling a JSON endpoint. Automating the API call is often:
- faster
- more stable
- less likely to break
- easier than scrolling the UI
If you want, I can show you a robust Playwright script that:
- handles infinite scroll,
- waits for new items properly,
- and extracts data into JSON/CSV.