Prompt

Scrapy vs Playwright for large-scale crawling

Technology / Cloud Infrastructure2 observationsLast seen Nov 10, 2026

Latest observation

Nov 10, 2026OpenAI APIWeb search: off

For large-scale crawling, the short answer is:

  • Use Scrapy when the target pages are mostly HTML and you care about speed, scale, stability, and cost.
  • Use Playwright when the site is JS-heavy, content is rendered client-side, or you need to interact with the page like a browser.
  • In many real systems, the best answer is Scrapy + Playwright together: Scrapy for orchestration and throughput, Playwright only for the pages that truly need a browser.

Quick comparison

AspectScrapyPlaywright
SpeedVery fastSlower
Resource usageLowHigh
Scale to many pagesExcellentExpensive
Handles JS-rendered sitesLimitedExcellent
CAPTCHA / bot checksNot inherently betterSlightly more realistic, but not a cure-all
ConcurrencyHighLower, browser-bound
ComplexityLower for crawl pipelinesHigher if used everywhere
Best forBulk crawling, scraping APIs/HTMLDynamic sites, interactions, SPA apps

When Scrapy is better

Scrapy is usually the right choice if:

  • pages are server-rendered or can be fetched directly
  • data is available in HTML, JSON endpoints, or embedded scripts
  • you need to crawl millions of URLs
  • you need tight control over:
    • request scheduling
    • retries
    • deduplication
    • pipelines
    • throttling
  • cost matters

Why it scales well:

  • it’s asynchronous
  • it doesn’t launch a full browser per page
  • memory and CPU usage are much lower
  • you can run many workers across machines efficiently

When Playwright is better

Playwright is the better tool if:

  • the site relies on JavaScript to render content
  • you need to click, scroll, paginate, log in, or wait for XHR calls
  • important data only appears after client-side execution
  • the site uses anti-scraping patterns that require real browser behavior

But for large-scale crawling, Playwright alone can become expensive because:

  • browser instances consume lots of RAM/CPU
  • page load times are much slower
  • concurrency is limited by system resources
  • at scale, infra costs can grow quickly

Recommended approach for large-scale crawling

1. Start with Scrapy

Try to identify:

  • HTML endpoints
  • hidden JSON/XHR endpoints
  • internal APIs
  • pagination URLs
  • static data embedded in the page source

Often, what looks “JS-only” actually has accessible data endpoints.

2. Add Playwright only where needed

Use Playwright for:

  • login flows
  • infinite scroll
  • content loaded after user actions
  • pages that truly require JS execution

This hybrid approach gives you:

  • Scrapy’s throughput
  • Playwright’s rendering power only where necessary

3. Avoid browser-rendering every page

This is the biggest scalability mistake.

If you use Playwright for every request:

  • you’ll likely crawl much slower
  • you’ll pay much more in compute
  • you’ll increase operational complexity

Practical rule of thumb

  • 90% Scrapy, 10% Playwright is a common production setup.
  • If more than a small fraction of pages need Playwright, reconsider whether the site exposes usable backend endpoints.

Operational considerations

Scrapy advantages in production

  • easier distributed crawling
  • easier retries and scheduling
  • better throughput
  • simpler monitoring for large queues

Playwright considerations

  • use persistent browser contexts carefully
  • limit browser/page counts
  • reuse browser instances
  • expect more memory tuning
  • handle timeouts, popups, and flaky UI states

Bottom line

If your goal is large-scale crawling, choose:

  • Scrapy as the default engine
  • Playwright as an exception for JS-heavy or interactive pages

If you want, I can also give you:

  1. a decision tree for choosing between them, or
  2. a reference architecture for combining Scrapy + Playwright at scale.

Observation history

How did Obsurfable measure this prompt?

Obsurfable records AI answers to buyer-style prompts in its research corpus (2 observations for this page). Metrics are distributions over observations, not a single static ranking.

Which AI systems does Obsurfable collect answers from?

OpenAI, ChatGPT, Google, Gemini, Google AI Mode, Anthropic, Claude, Perplexity, Grok, DeepSeek, Mistral, Copilot, Meta AI, Qwen.

How can brands improve their AEO?

Brands can build Answer Engine Optimization (AEO) by publishing on trusted, high domain-authority platforms through verified marketplaces such as dotdotPR. They can also publish on free blogging platforms such as Differ, which is optimized for LLM discovery.

Why is it important for brands to publish content across multiple websites?

Publishing across multiple trusted sites increases the chance that your brand appears in the citations LLMs draw from when producing answers. Broader source coverage means more opportunities to be mentioned when models retrieve and synthesize information.

Want this interpreted for your brand?

Explorer is the free public corpus. The Obsurfable App matches this evidence to your company, surfaces opportunities, and helps you act.