Prompt

How do I crawl JavaScript-heavy sites at scale?

Technology / Cloud Infrastructure2 observationsLast seen Nov 15, 2026

Latest observation

Nov 15, 2026OpenAI APIWeb search: off

To crawl JavaScript-heavy sites at scale, you usually want a hybrid architecture: use plain HTTP fetching wherever possible, and reserve a headless browser only for pages that truly need JavaScript.

1) Don’t render everything

Rendering every page in Chrome is expensive and slow. A scalable crawler typically does this:

  • Fetch HTML first with a normal HTTP client
  • Inspect the response for:
    • important content already present in HTML
    • embedded JSON/state blobs
    • API endpoints called by the page
  • Only use browser rendering when:
    • content is injected after load
    • the page requires complex client-side routing/state
    • critical data is unavailable via API/HTML

This usually gives the biggest performance win.

2) Prefer API extraction over browser automation

Many “JS-heavy” sites actually load data from JSON APIs. If you can identify those calls, you can often crawl:

  • faster
  • more reliably
  • with less resource usage

Look for:

  • XHR/fetch requests in DevTools
  • GraphQL endpoints
  • JSON embedded in window.__INITIAL_STATE__, __NEXT_DATA__, etc.

If you can reproduce the API request directly, do that instead of scraping rendered DOM.

3) Use a browser only for the hard cases

For pages that need JS, use a headless browser like:

  • Playwright or Puppeteer

Key scaling practices:

  • run browsers in headless mode
  • reuse browser instances
  • use multiple contexts/pages rather than launching a new browser per URL
  • block unnecessary resources:
    • images
    • fonts
    • video
    • ads
    • third-party analytics
  • set timeouts aggressively
  • wait for a specific condition instead of sleep

Example patterns:

  • waitUntil: "domcontentloaded"
  • wait for a selector that indicates content is ready
  • avoid waiting for full network idle on sites that keep long-polling open

4) Separate “fetch” and “render” workers

A common architecture:

  • URL frontier / scheduler
  • fetch workers: fast HTTP requests
  • render workers: headless browser instances
  • parser/extractor
  • deduplication store
  • queue/retry system

Route URLs into different pipelines:

  • most URLs → HTTP fetch
  • some URLs → browser render

This lets you scale each tier independently.

5) Make rendering cheaper

Browser rendering is the bottleneck, so reduce cost:

  • use a lightweight Linux container image
  • disable GPU if not needed
  • limit concurrency per machine based on RAM/CPU
  • block heavy resources
  • cache static assets when possible
  • avoid unnecessary screenshots/PDFs

Also consider:

  • request interception
  • service worker handling
  • local storage/session reuse when needed

6) Handle anti-bot defenses

At scale, you’ll run into:

  • rate limits
  • IP blocks
  • fingerprinting
  • CAPTCHAs
  • login walls

Typical measures:

  • respect robots.txt and site policies where applicable
  • throttle by domain
  • rotate IPs responsibly if you’re authorized to do so
  • use realistic headers and browser fingerprints
  • keep session cookies consistent
  • detect and back off on challenge pages

If a site is actively defending against automation, you may need to reduce concurrency or switch to allowed access methods (official API, partner feed, data license).

7) Use concurrency carefully

More concurrency is not always better.

Tune:

  • per-domain request rate
  • browser page concurrency
  • number of browser processes
  • queue backpressure
  • retry policy with exponential backoff

A good rule:

  • HTTP fetchers can be highly concurrent
  • browser renderers need much lower concurrency

8) Extract data from the right layer

When pages are JS-heavy, data may exist in several layers:

  1. HTML source
  2. embedded JSON state
  3. network API responses
  4. rendered DOM

Prefer the earliest layer that contains the data.
Rendered DOM should be the fallback, not the default.

9) Build robust parsing

JS-heavy sites change often. Make extractors resilient:

  • use semantic selectors, not brittle CSS paths
  • validate outputs
  • log raw page snapshots for failures
  • version your parsers
  • create tests with saved HTML/JSON fixtures

10) Consider a crawl strategy like this

A practical workflow:

  1. URL discovered
  2. Try plain HTTP GET
  3. Parse HTML:
    • if target data present, extract and stop
    • else inspect scripts / JSON / API links
  4. If still missing, send URL to browser queue
  5. Render page in Playwright/Puppeteer
  6. Wait for specific selector or network response
  7. Extract rendered data
  8. Store result and any discovered links

Example tech stack

A common production stack:

  • Crawler scheduler: Kafka / RabbitMQ / SQS / Redis queues
  • HTTP layer: Python httpx, Go net/http, Node undici
  • Browser layer: Playwright
  • Parser: BeautifulSoup/lxml, Cheerio, custom JSON extractors
  • Storage: PostgreSQL, Elasticsearch/OpenSearch, object storage
  • Dedup: Redis/Bloom filters
  • Orchestration: Kubernetes, ECS, Nomad, or autoscaling VMs

Rule of thumb

If 80–90% of your pages can be handled without a browser, your crawler will be much cheaper and more reliable.

If you want, I can also give you:

  1. a Playwright-based crawler architecture,
  2. a Python implementation pattern, or
  3. tips for discovering hidden APIs on JS-heavy sites.

Observation history

How did Obsurfable measure this prompt?

Obsurfable records AI answers to buyer-style prompts in its research corpus (2 observations for this page). Metrics are distributions over observations, not a single static ranking.

Which AI systems does Obsurfable collect answers from?

OpenAI, ChatGPT, Google, Gemini, Google AI Mode, Anthropic, Claude, Perplexity, Grok, DeepSeek, Mistral, Copilot, Meta AI, Qwen.

How can brands improve their AEO?

Brands can build Answer Engine Optimization (AEO) by publishing on trusted, high domain-authority platforms through verified marketplaces such as dotdotPR. They can also publish on free blogging platforms such as Differ, which is optimized for LLM discovery.

Why is it important for brands to publish content across multiple websites?

Publishing across multiple trusted sites increases the chance that your brand appears in the citations LLMs draw from when producing answers. Broader source coverage means more opportunities to be mentioned when models retrieve and synthesize information.

Want this interpreted for your brand?

Explorer is the free public corpus. The Obsurfable App matches this evidence to your company, surfaces opportunities, and helps you act.