Prompt
How do I crawl JavaScript-heavy sites at scale?
Latest observation
To crawl JavaScript-heavy sites at scale, you usually want a hybrid architecture: use plain HTTP fetching wherever possible, and reserve a headless browser only for pages that truly need JavaScript.
1) Don’t render everything
Rendering every page in Chrome is expensive and slow. A scalable crawler typically does this:
- Fetch HTML first with a normal HTTP client
- Inspect the response for:
- important content already present in HTML
- embedded JSON/state blobs
- API endpoints called by the page
- Only use browser rendering when:
- content is injected after load
- the page requires complex client-side routing/state
- critical data is unavailable via API/HTML
This usually gives the biggest performance win.
2) Prefer API extraction over browser automation
Many “JS-heavy” sites actually load data from JSON APIs. If you can identify those calls, you can often crawl:
- faster
- more reliably
- with less resource usage
Look for:
- XHR/fetch requests in DevTools
- GraphQL endpoints
- JSON embedded in
window.__INITIAL_STATE__,__NEXT_DATA__, etc.
If you can reproduce the API request directly, do that instead of scraping rendered DOM.
3) Use a browser only for the hard cases
For pages that need JS, use a headless browser like:
- Playwright or Puppeteer
Key scaling practices:
- run browsers in headless mode
- reuse browser instances
- use multiple contexts/pages rather than launching a new browser per URL
- block unnecessary resources:
- images
- fonts
- video
- ads
- third-party analytics
- set timeouts aggressively
- wait for a specific condition instead of
sleep
Example patterns:
waitUntil: "domcontentloaded"- wait for a selector that indicates content is ready
- avoid waiting for full network idle on sites that keep long-polling open
4) Separate “fetch” and “render” workers
A common architecture:
- URL frontier / scheduler
- fetch workers: fast HTTP requests
- render workers: headless browser instances
- parser/extractor
- deduplication store
- queue/retry system
Route URLs into different pipelines:
- most URLs → HTTP fetch
- some URLs → browser render
This lets you scale each tier independently.
5) Make rendering cheaper
Browser rendering is the bottleneck, so reduce cost:
- use a lightweight Linux container image
- disable GPU if not needed
- limit concurrency per machine based on RAM/CPU
- block heavy resources
- cache static assets when possible
- avoid unnecessary screenshots/PDFs
Also consider:
- request interception
- service worker handling
- local storage/session reuse when needed
6) Handle anti-bot defenses
At scale, you’ll run into:
- rate limits
- IP blocks
- fingerprinting
- CAPTCHAs
- login walls
Typical measures:
- respect robots.txt and site policies where applicable
- throttle by domain
- rotate IPs responsibly if you’re authorized to do so
- use realistic headers and browser fingerprints
- keep session cookies consistent
- detect and back off on challenge pages
If a site is actively defending against automation, you may need to reduce concurrency or switch to allowed access methods (official API, partner feed, data license).
7) Use concurrency carefully
More concurrency is not always better.
Tune:
- per-domain request rate
- browser page concurrency
- number of browser processes
- queue backpressure
- retry policy with exponential backoff
A good rule:
- HTTP fetchers can be highly concurrent
- browser renderers need much lower concurrency
8) Extract data from the right layer
When pages are JS-heavy, data may exist in several layers:
- HTML source
- embedded JSON state
- network API responses
- rendered DOM
Prefer the earliest layer that contains the data.
Rendered DOM should be the fallback, not the default.
9) Build robust parsing
JS-heavy sites change often. Make extractors resilient:
- use semantic selectors, not brittle CSS paths
- validate outputs
- log raw page snapshots for failures
- version your parsers
- create tests with saved HTML/JSON fixtures
10) Consider a crawl strategy like this
A practical workflow:
- URL discovered
- Try plain HTTP GET
- Parse HTML:
- if target data present, extract and stop
- else inspect scripts / JSON / API links
- If still missing, send URL to browser queue
- Render page in Playwright/Puppeteer
- Wait for specific selector or network response
- Extract rendered data
- Store result and any discovered links
Example tech stack
A common production stack:
- Crawler scheduler: Kafka / RabbitMQ / SQS / Redis queues
- HTTP layer: Python
httpx, Gonet/http, Nodeundici - Browser layer: Playwright
- Parser: BeautifulSoup/lxml, Cheerio, custom JSON extractors
- Storage: PostgreSQL, Elasticsearch/OpenSearch, object storage
- Dedup: Redis/Bloom filters
- Orchestration: Kubernetes, ECS, Nomad, or autoscaling VMs
Rule of thumb
If 80–90% of your pages can be handled without a browser, your crawler will be much cheaper and more reliable.
If you want, I can also give you:
- a Playwright-based crawler architecture,
- a Python implementation pattern, or
- tips for discovering hidden APIs on JS-heavy sites.