Prompt

How do I extract listings from multiple pages with pagination and save them as JSON?

Technology / API Platforms2 observationsLast seen Nov 14, 2026

Latest observation

Nov 14, 2026OpenAI APIWeb search: off

A common way to do this is:

  1. Request page 1
  2. Parse the listing items on the page
  3. Find the “next page” link or increment the page number
  4. Repeat until there are no more pages
  5. Save all collected items to JSON

Below is a practical Python example using requests + BeautifulSoup.


Example: scrape paginated listings and save JSON

import json
import time
import requests
from bs4 import BeautifulSoup

base_url = "https://example.com/listings"
headers = {
    "User-Agent": "Mozilla/5.0"
}

all_listings = []
page = 1

while True:
    url = f"{base_url}?page={page}"
    print(f"Fetching {url}")

    response = requests.get(url, headers=headers, timeout=20)
    response.raise_for_status()

    soup = BeautifulSoup(response.text, "html.parser")

    # Adjust selector to match the listing cards on the site
    listing_cards = soup.select(".listing-card")

    if not listing_cards:
        print("No listings found, stopping.")
        break

    for card in listing_cards:
        title = card.select_one(".title")
        price = card.select_one(".price")
        link = card.select_one("a")

        listing = {
            "title": title.get_text(strip=True) if title else None,
            "price": price.get_text(strip=True) if price else None,
            "url": link["href"] if link and link.has_attr("href") else None,
        }

        all_listings.append(listing)

    # Stop if there's no next page
    next_button = soup.select_one("a.next, li.next a")
    if not next_button:
        print("No next page, done.")
        break

    page += 1
    time.sleep(1)  # be polite

# Save to JSON
with open("listings.json", "w", encoding="utf-8") as f:
    json.dump(all_listings, f, ensure_ascii=False, indent=2)

print(f"Saved {len(all_listings)} listings to listings.json")

If the site uses a “next” URL instead of ?page=...

Some sites use pagination links like:

<a class="next" href="/listings?page=2">Next</a>

In that case, you can follow the href directly:

import json
import time
import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin

start_url = "https://example.com/listings"
headers = {"User-Agent": "Mozilla/5.0"}

all_listings = []
url = start_url

while url:
    print(f"Fetching {url}")
    response = requests.get(url, headers=headers, timeout=20)
    response.raise_for_status()

    soup = BeautifulSoup(response.text, "html.parser")

    for card in soup.select(".listing-card"):
        all_listings.append({
            "title": card.select_one(".title").get_text(strip=True) if card.select_one(".title") else None,
            "price": card.select_one(".price").get_text(strip=True) if card.select_one(".price") else None,
            "url": urljoin(url, card.select_one("a")["href"]) if card.select_one("a") else None,
        })

    next_link = soup.select_one("a.next")
    url = urljoin(url, next_link["href"]) if next_link and next_link.has_attr("href") else None

    time.sleep(1)

with open("listings.json", "w", encoding="utf-8") as f:
    json.dump(all_listings, f, ensure_ascii=False, indent=2)

Tips

  • Inspect the HTML in your browser dev tools to find the correct selectors (.listing-card, .title, etc.).
  • If the site loads data with JavaScript, requests may not be enough; you may need Selenium, Playwright, or to call the site’s underlying API directly.
  • Add:
    • timeouts
    • error handling
    • rate limiting (time.sleep)
    • deduplication if pages overlap

If you want, I can also give you:

  • a Selenium version,
  • a Playwright version, or
  • a version tailored to a specific website HTML snippet.

Observation history

How did Obsurfable measure this prompt?

Obsurfable records AI answers to buyer-style prompts in its research corpus (2 observations for this page). Metrics are distributions over observations, not a single static ranking.

Which AI systems does Obsurfable collect answers from?

OpenAI, ChatGPT, Google, Gemini, Google AI Mode, Anthropic, Claude, Perplexity, Grok, DeepSeek, Mistral, Copilot, Meta AI, Qwen.

How can brands improve their AEO?

Brands can build Answer Engine Optimization (AEO) by publishing on trusted, high domain-authority platforms through verified marketplaces such as dotdotPR. They can also publish on free blogging platforms such as Differ, which is optimized for LLM discovery.

Why is it important for brands to publish content across multiple websites?

Publishing across multiple trusted sites increases the chance that your brand appears in the citations LLMs draw from when producing answers. Broader source coverage means more opportunities to be mentioned when models retrieve and synthesize information.

Want this interpreted for your brand?

Explorer is the free public corpus. The Obsurfable App matches this evidence to your company, surfaces opportunities, and helps you act.