Why is pagination not being followed in my scraper?

Pagination usually “isn’t being followed” for one of these reasons:

1) The site loads pages with JavaScript

If the next-page content is fetched dynamically in the browser, a simple requests/BeautifulSoup scraper will only see the initial HTML.
Check: Open page source vs. browser dev tools → Network tab.
Fix: Use the site’s underlying API endpoint, or a browser automation tool like Playwright/Selenium.

2) You’re not updating the page URL or request parameter correctly

Often pagination is driven by:

• ?page=2
• /page/2/
• offset=20
• a “next” token/cursor

If your scraper keeps requesting the same URL, it will keep getting the first page.
Check: Log each request URL you make.

3) Your “next page” selector is wrong

The HTML element you’re clicking/extracting may not be the real next link, or it may be disabled on the first page.
Check: Print the extracted href/text for the next button/link.

4) Relative URLs aren’t being resolved correctly

A link like /products?page=2 must be turned into the full URL.
Fix: Use urljoin(base_url, href).

5) You’re stopping too early

Your loop may break because:

• “next” link isn’t found
• duplicate-page detection is too aggressive
• exceptions are swallowed
• the scraper thinks it reached the last page

Check: Add debug prints for each loop iteration and break condition.

6) The site requires headers/cookies/session state

Some sites only return paginated content if you preserve:

• cookies
• session tokens
• User-Agent / Referer headers

Fix: Use a persistent session and include the required headers.

---

Quick debugging checklist

Print these on every iteration:

• current URL
• response status code
• first 200 characters of HTML
• extracted next-page link/URL
• whether the loop decides to continue

---

Example pattern for HTML link pagination

import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin

url = "https://example.com/items"
session = requests.Session()

while url:
    print("Fetching:", url)
    r = session.get(url, headers={"User-Agent": "Mozilla/5.0"})
    print("Status:", r.status_code)

    soup = BeautifulSoup(r.text, "html.parser")
    # process items here...

    next_link = soup.select_one("a.next")
    if next_link and next_link.get("href"):
        url = urljoin(url, next_link["href"])
    else:
        url = None

If you want, paste your scraper code and a sample page URL, and I can point out the exact pagination issue.