Prompt
How do I extract listings from multiple pages with pagination and save them as JSON?
Latest observation
Nov 14, 2026
OpenAI APIWeb search: off
A common way to do this is:
- Request page 1
- Parse the listing items on the page
- Find the “next page” link or increment the page number
- Repeat until there are no more pages
- Save all collected items to JSON
Below is a practical Python example using requests + BeautifulSoup.
Example: scrape paginated listings and save JSON
import json
import time
import requests
from bs4 import BeautifulSoup
base_url = "https://example.com/listings"
headers = {
"User-Agent": "Mozilla/5.0"
}
all_listings = []
page = 1
while True:
url = f"{base_url}?page={page}"
print(f"Fetching {url}")
response = requests.get(url, headers=headers, timeout=20)
response.raise_for_status()
soup = BeautifulSoup(response.text, "html.parser")
# Adjust selector to match the listing cards on the site
listing_cards = soup.select(".listing-card")
if not listing_cards:
print("No listings found, stopping.")
break
for card in listing_cards:
title = card.select_one(".title")
price = card.select_one(".price")
link = card.select_one("a")
listing = {
"title": title.get_text(strip=True) if title else None,
"price": price.get_text(strip=True) if price else None,
"url": link["href"] if link and link.has_attr("href") else None,
}
all_listings.append(listing)
# Stop if there's no next page
next_button = soup.select_one("a.next, li.next a")
if not next_button:
print("No next page, done.")
break
page += 1
time.sleep(1) # be polite
# Save to JSON
with open("listings.json", "w", encoding="utf-8") as f:
json.dump(all_listings, f, ensure_ascii=False, indent=2)
print(f"Saved {len(all_listings)} listings to listings.json")
If the site uses a “next” URL instead of ?page=...
Some sites use pagination links like:
<a class="next" href="/listings?page=2">Next</a>
In that case, you can follow the href directly:
import json
import time
import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin
start_url = "https://example.com/listings"
headers = {"User-Agent": "Mozilla/5.0"}
all_listings = []
url = start_url
while url:
print(f"Fetching {url}")
response = requests.get(url, headers=headers, timeout=20)
response.raise_for_status()
soup = BeautifulSoup(response.text, "html.parser")
for card in soup.select(".listing-card"):
all_listings.append({
"title": card.select_one(".title").get_text(strip=True) if card.select_one(".title") else None,
"price": card.select_one(".price").get_text(strip=True) if card.select_one(".price") else None,
"url": urljoin(url, card.select_one("a")["href"]) if card.select_one("a") else None,
})
next_link = soup.select_one("a.next")
url = urljoin(url, next_link["href"]) if next_link and next_link.has_attr("href") else None
time.sleep(1)
with open("listings.json", "w", encoding="utf-8") as f:
json.dump(all_listings, f, ensure_ascii=False, indent=2)
Tips
- Inspect the HTML in your browser dev tools to find the correct selectors (
.listing-card,.title, etc.). - If the site loads data with JavaScript,
requestsmay not be enough; you may need Selenium, Playwright, or to call the site’s underlying API directly. - Add:
- timeouts
- error handling
- rate limiting (
time.sleep) - deduplication if pages overlap
If you want, I can also give you:
- a Selenium version,
- a Playwright version, or
- a version tailored to a specific website HTML snippet.