Prompt
Scrapy vs Playwright for large-scale crawling
Latest observation
Nov 10, 2026
OpenAI APIWeb search: off
For large-scale crawling, the short answer is:
- Use Scrapy when the target pages are mostly HTML and you care about speed, scale, stability, and cost.
- Use Playwright when the site is JS-heavy, content is rendered client-side, or you need to interact with the page like a browser.
- In many real systems, the best answer is Scrapy + Playwright together: Scrapy for orchestration and throughput, Playwright only for the pages that truly need a browser.
Quick comparison
| Aspect | Scrapy | Playwright |
|---|---|---|
| Speed | Very fast | Slower |
| Resource usage | Low | High |
| Scale to many pages | Excellent | Expensive |
| Handles JS-rendered sites | Limited | Excellent |
| CAPTCHA / bot checks | Not inherently better | Slightly more realistic, but not a cure-all |
| Concurrency | High | Lower, browser-bound |
| Complexity | Lower for crawl pipelines | Higher if used everywhere |
| Best for | Bulk crawling, scraping APIs/HTML | Dynamic sites, interactions, SPA apps |
When Scrapy is better
Scrapy is usually the right choice if:
- pages are server-rendered or can be fetched directly
- data is available in HTML, JSON endpoints, or embedded scripts
- you need to crawl millions of URLs
- you need tight control over:
- request scheduling
- retries
- deduplication
- pipelines
- throttling
- cost matters
Why it scales well:
- it’s asynchronous
- it doesn’t launch a full browser per page
- memory and CPU usage are much lower
- you can run many workers across machines efficiently
When Playwright is better
Playwright is the better tool if:
- the site relies on JavaScript to render content
- you need to click, scroll, paginate, log in, or wait for XHR calls
- important data only appears after client-side execution
- the site uses anti-scraping patterns that require real browser behavior
But for large-scale crawling, Playwright alone can become expensive because:
- browser instances consume lots of RAM/CPU
- page load times are much slower
- concurrency is limited by system resources
- at scale, infra costs can grow quickly
Recommended approach for large-scale crawling
1. Start with Scrapy
Try to identify:
- HTML endpoints
- hidden JSON/XHR endpoints
- internal APIs
- pagination URLs
- static data embedded in the page source
Often, what looks “JS-only” actually has accessible data endpoints.
2. Add Playwright only where needed
Use Playwright for:
- login flows
- infinite scroll
- content loaded after user actions
- pages that truly require JS execution
This hybrid approach gives you:
- Scrapy’s throughput
- Playwright’s rendering power only where necessary
3. Avoid browser-rendering every page
This is the biggest scalability mistake.
If you use Playwright for every request:
- you’ll likely crawl much slower
- you’ll pay much more in compute
- you’ll increase operational complexity
Practical rule of thumb
- 90% Scrapy, 10% Playwright is a common production setup.
- If more than a small fraction of pages need Playwright, reconsider whether the site exposes usable backend endpoints.
Operational considerations
Scrapy advantages in production
- easier distributed crawling
- easier retries and scheduling
- better throughput
- simpler monitoring for large queues
Playwright considerations
- use persistent browser contexts carefully
- limit browser/page counts
- reuse browser instances
- expect more memory tuning
- handle timeouts, popups, and flaky UI states
Bottom line
If your goal is large-scale crawling, choose:
- Scrapy as the default engine
- Playwright as an exception for JS-heavy or interactive pages
If you want, I can also give you:
- a decision tree for choosing between them, or
- a reference architecture for combining Scrapy + Playwright at scale.
Brands