How can I scrape a website that uses infinite scrolling?

To scrape a website with infinite scrolling, you usually need to simulate the page’s loading behavior or, better yet, find the underlying API request that fetches the additional content.

Best approach: inspect network requests

1. Open the site in your browser.
2. Use DevTools → Network.
3. Scroll down until more content loads.
4. Look for requests to:
   • JSON endpoints
   • GraphQL endpoints
   • XHR/fetch calls
5. Reproduce those requests in code.

This is usually much faster and more reliable than controlling the browser’s scroll position.

---

Option 1: Use the underlying API directly

If the site loads data from an API, you can often scrape with requests:

import requests

url = "https://example.com/api/items?page=2"
headers = {
    "User-Agent": "Mozilla/5.0"
}

resp = requests.get(url, headers=headers)
data = resp.json()
print(data)

If the API requires pagination tokens or cursors, you’ll need to extract and pass those along.

---

Option 2: Use a browser automation tool

If the site renders content only after scrolling and you can’t easily find an API, use tools like:

• Selenium
• Playwright
• Puppeteer

Example with Selenium

from selenium import webdriver
from selenium.webdriver.common.by import By
import time

driver = webdriver.Chrome()
driver.get("https://example.com")

last_height = driver.execute_script("return document.body.scrollHeight")

while True:
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    time.sleep(2)

    new_height = driver.execute_script("return document.body.scrollHeight")
    if new_height == last_height:
        break
    last_height = new_height

items = driver.find_elements(By.CSS_SELECTOR, ".item")
for item in items:
    print(item.text)

driver.quit()

---

Option 3: Scroll until no new items appear

With infinite scroll, you typically loop:

1. Scroll to bottom
2. Wait for new content
3. Check whether more items appeared
4. Stop when nothing new loads

You can track:

• page height
• number of loaded items
• a “loading” spinner disappearing
• absence of network requests

---

Practical tips

• Add delays to avoid being blocked.
• Use realistic User-Agent and headers.
• Handle lazy-loaded images separately if needed.
• If content is behind login, you may need to preserve cookies/session.
• Be mindful of the site’s terms of service and robots.txt.

---

Recommended strategy

1. Check the network tab first
2. If possible, scrape the API
3. Otherwise, use Playwright/Selenium to scroll and collect content

If you want, I can show you:

• a Playwright example
• how to find the API endpoint
• or how to scrape a specific infinite-scroll page