How do I scrape paginated websites?

Scraping paginated websites usually means: fetch one page, find the “next page” link or page number, repeat until there are no more pages. The exact approach depends on how the site paginates.

1) First identify the pagination type

Common patterns:

A. URL-based pagination

Examples:

• https://example.com/products?page=2
• https://example.com/articles?offset=20
• https://example.com/search/page/3

This is the easiest case: you just change the page number or offset in the request URL.

B. Link-based pagination

The page contains a “Next” button/link in the HTML:

<a href="/items?page=2">Next</a>

You fetch the page, parse the next link, and follow it until it disappears.

C. API-backed pagination

The page loads data from an endpoint returning JSON, often with:

• page
• limit
• offset
• cursor
• next_cursor

This is often better to scrape than HTML, because it’s cleaner and more stable.

D. Infinite scroll / lazy load

The website loads more results as you scroll, usually via API requests. You can:

• inspect network requests in browser dev tools,
• find the underlying API,
• call it directly.

---

2) Basic scraping loop

Example: URL with ?page=

import requests
from bs4 import BeautifulSoup

base_url = "https://example.com/items"

page = 1
all_items = []

while True:
    url = f"{base_url}?page={page}"
    response = requests.get(url, timeout=10)
    response.raise_for_status()

    soup = BeautifulSoup(response.text, "html.parser")
    items = soup.select(".item")

    if not items:
        break

    for item in items:
        all_items.append(item.get_text(strip=True))

    # stop if there is no next page
    next_link = soup.select_one("a.next")
    if not next_link:
        break

    page += 1

---

3) Following “Next” links

If the site provides a next-page URL in the HTML, use that instead of guessing page numbers:

import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin

url = "https://example.com/items"
all_items = []

while url:
    r = requests.get(url, timeout=10)
    r.raise_for_status()

    soup = BeautifulSoup(r.text, "html.parser")

    for item in soup.select(".item"):
        all_items.append(item.get_text(strip=True))

    next_a = soup.select_one("a[rel='next'], a.next")
    url = urljoin(url, next_a["href"]) if next_a else None

This is more robust because it follows the site’s actual navigation.

---

4) Handling offset-based pagination

Sometimes the URL uses offsets rather than page numbers:

offset = 0
limit = 20

while True:
    url = f"https://example.com/items?offset={offset}&limit={limit}"
    r = requests.get(url)
    r.raise_for_status()

    soup = BeautifulSoup(r.text, "html.parser")
    items = soup.select(".item")

    if not items:
        break

    # process items...
    offset += limit

---

5) Scraping API pagination

If you find an API endpoint returning JSON:

import requests

url = "https://example.com/api/items?page=1"
all_items = []

while url:
    data = requests.get(url, timeout=10).json()
    all_items.extend(data["results"])
    url = data.get("next")   # often a full URL or None

Or cursor-based pagination:

cursor = None
all_items = []

while True:
    params = {}
    if cursor:
        params["cursor"] = cursor

    data = requests.get("https://example.com/api/items", params=params).json()
    all_items.extend(data["results"])

    cursor = data.get("next_cursor")
    if not cursor:
        break

---

6) Avoid common issues

Rate limiting / blocking

• Add delays between requests:

import time
time.sleep(1)

• Use a proper User-Agent
• Keep request volume reasonable
• Respect robots.txt and the site’s terms

Duplicate or missing results

• Track item IDs/URLs to avoid duplicates
• Stop only when you truly hit the end, not just when a page is empty due to an error

JavaScript-rendered pages

If content is not in the HTML response, use:

• the site’s API directly, or
• a browser automation tool like Playwright or Selenium.

---

7) Practical strategy

When scraping paginated sites, do this:

1. Open the page in your browser
2. Check whether page changes are in the URL
3. Inspect network requests for JSON APIs
4. Prefer direct API scraping if available
5. If not, parse HTML and follow the “next” link
6. Add retries, delays, and deduplication

---

8) A simple reusable pattern

import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin
import time

def scrape_paginated(start_url, item_selector, next_selector):
    url = start_url
    results = []

    while url:
        r = requests.get(url, timeout=10)
        r.raise_for_status()

        soup = BeautifulSoup(r.text, "html.parser")
        results.extend([x.get_text(" ", strip=True) for x in soup.select(item_selector)])

        next_link = soup.select_one(next_selector)
        url = urljoin(url, next_link["href"]) if next_link and next_link.get("href") else None

        time.sleep(1)

    return results

---

If you want, I can show you how to scrape a specific paginated site using:

• requests + BeautifulSoup,
• Scrapy,
• or Playwright for JavaScript-heavy pages.