Why is Apify so slow on large crawl jobs?

Apify can feel slow on large crawl jobs for a few common reasons, and most of them come down to how crawling scales, not just Apify itself.

Main reasons large jobs slow down

1. Site-side limits

• Many websites throttle requests, block bots, or slow responses when they detect heavy traffic.
• If pages start returning 429s, retries, or long response times, your crawl will drag.

2. Browser automation overhead

• If you’re using Playwright/Puppeteer, each page is expensive:
  • starting browsers
  • rendering JS
  • loading images/scripts
  • waiting for network idle
• At scale, this is much slower than plain HTTP crawling.

3. Too much waiting in the scraper

• Common bottlenecks:
  • waitUntil: networkidle
  • fixed sleeps
  • waiting for selectors that never appear
  • loading unnecessary resources
• These compound heavily over thousands of pages.

4. Actor concurrency is too low or misconfigured

• If concurrency is conservative, you may be underutilizing available capacity.
• If it’s too high, you may trigger throttling and actually slow down overall.
• The “right” concurrency depends on the site and whether you’re using browser or HTTP.

5. Dataset/storage overhead

• Writing lots of records, especially large objects, can become a bottleneck.
• Frequent flushes, logging, or pushing huge items one-by-one may slow throughput.

6. Memory/CPU constraints

• Crawls can become CPU-bound, especially with browser automation and heavy parsing.
• If the run is memory constrained, you may see garbage collection pressure or process slowdown.

7. Queue churn and retries

• Large crawls often include:
  • duplicate URLs
  • redirects
  • bad links
  • retries on failures
• Those add a lot of hidden time.

8. Proxy/network latency

• If you use proxies, especially residential ones, requests are usually slower.
• Geographic distance and proxy quality can matter a lot.

---

How to speed it up

• Prefer HTTP crawling over browser automation when possible.
• Remove unnecessary waits and resource loading.
• Tune concurrency gradually upward while monitoring block/error rates.
• Use autoscaled crawling carefully rather than blindly maximizing.
• Minimize data written per item and batch where possible.
• Deduplicate URLs early.
• Avoid over-logging.
• Use site-specific throttling instead of hammering everything equally.

---

Quick rule of thumb

If a crawl is slow:

• Fast pages but many of them → likely concurrency/storage/queue tuning.
• Slow individual pages → likely browser overhead, waits, or site throttling.
• Random slowdowns and retries → likely blocking, proxies, or server-side limits.

If you want, I can help you diagnose a specific Apify crawl setup and tell you where the bottleneck likely is.