Why is concurrent Puppeteer scraping slow in Node.js?

Concurrent Puppeteer scraping is usually slow not because Node.js cannot run async work, but because every “concurrent” task still runs inside a heavyweight Chromium browser. Multiple tabs compete for the same browser process’s CPU, memory, network, and internal locks, so throughput often plateaus after only a handful of pages.

Main bottlenecks

• Chromium is the real bottleneck. Opening 50 pages in one browser means 50 page loads, DOM parses, JavaScript executions, and networkidle waits competing inside one Chromium instance—not 50 independent Node tasks.
• Shared browser context and cache contention. Pages created from the same browser share Chromium state; concurrent requests for the same scripts or assets can contend on cache locks, serializing work that looks parallel in your code.
• Heavy resources and rendering. Images, fonts, CSS, videos, trackers, and scripts consume bandwidth and renderer CPU even when you only need text from the DOM.
• waitUntil: 'networkidle2' adds fixed delay. It waits for network quiet, so every page can incur an extra idle timeout even after the content you need is already available.
• Memory pressure. Each Chromium page can consume roughly 30–80 MB, and rendering pages may use 100–400 MB. Too many tabs cause swapping, GC pauses, crashes, or slower rendering.
• Unbounded Promise.all. await Promise.all(urls.map(scrape)) launches every scrape at once; it does not create a queue. That can exhaust file descriptors, increase timeouts, and trigger rate limits.
• Leaked pages/contexts. If a page is not closed on an error path, its renderer resources remain allocated, causing memory growth and eventual slowdowns.

How to make it faster

Change	Why it helps
Use a concurrency pool of about 5–10 pages per browser	Avoids saturating Chromium, RAM, and the target site; beyond this, added tabs often cost memory without improving throughput. webscrapingapi+1
Block images, media, fonts, and unnecessary CSS via request interception	Removes work that does not contribute to most text/data scraping. webscrapingapi
Use domcontentloaded or wait for a specific selector instead of networkidle2	Avoids waiting for unrelated network activity to settle. plainenglish
Reuse one browser, but close every page in finally	Browser startup is expensive; leaked pages accumulate memory. clawengine+1
Recycle the browser periodically	Prevents gradual Chromium state/memory degradation. clawengine
Run several browser processes for CPU-heavy workloads	Distributes rendering and parsing across processes/cores instead of concentrating them in one Chromium instance. plainenglish
Use fetch + Cheerio for static pages	Avoids launching a browser entirely; Puppeteer is best reserved for JavaScript-rendered or interaction-dependent pages. crawlora

Practical pattern

Use a small worker pool rather than mapping every URL to a Puppeteer task:

import pLimit from 'p-limit';

const limit = pLimit(6);

const results = await Promise.all(
  urls.map(url => limit(() => scrapePage(url)))
);

Inside scrapePage, open a page, disable unneeded resources, wait only for the selector you need, extract the data, and close the page in a finally block. This keeps concurrency bounded while still overlapping network latency across pages.

In short: Node’s event loop handles many waiting tasks well, but Puppeteer’s parallelism is bounded by Chromium. The biggest wins usually come from reducing page weight, replacing broad network-idle waits, limiting concurrency, and using a non-browser scraper where possible.