What's the best way to scrape websites into JSON?

The “best” way depends on how the site is built and what you want from the data, but the most reliable pattern is:

1. Use the site’s own API if available

   • Best for stability, speed, and clean JSON.
   • Check network requests in DevTools for XHR/fetch calls.

2. If no API, fetch HTML and extract structured data

   • Use a crawler/scraper plus parsers like:
     • Python: requests + BeautifulSoup / lxml
     • JS/TS: fetch/axios + cheerio
   • Normalize extracted fields into JSON.

3. For JavaScript-heavy sites, use a headless browser

   • Playwright or Puppeteer render the page and let you wait for content.
   • Often you can still intercept the underlying API calls and skip browser rendering.

4. Design a schema first

   • Decide what JSON should look like before scraping.
   • Example:

     {
       "title": "",
       "price": 0,
       "url": "",
       "images": [],
       "metadata": {}
     }
     

5. Handle pagination, deduping, and errors

   • Follow next-page links or page cursors.
   • Save raw HTML or source URLs for debugging.
   • Validate output with a schema.

6. Respect robots.txt and site terms

   • Rate limit requests, identify your bot, and avoid overloading servers.

Practical recommendation

• Static sites: requests + BeautifulSoup/lxml
• Dynamic sites: Playwright, but prefer network/API extraction when possible
• Large-scale crawling: Scrapy (Python) or a crawler with queueing/retries

Minimal Python example

import requests
from bs4 import BeautifulSoup

url = "https://example.com"
html = requests.get(url, timeout=10).text
soup = BeautifulSoup(html, "html.parser")

data = {
    "title": soup.title.get_text(strip=True) if soup.title else None,
    "headings": [h.get_text(strip=True) for h in soup.select("h1, h2")]
}

print(data)

If you want, I can suggest the best stack for your specific target site type: static HTML, JS app, or API-backed.