How do I crawl millions of pages without building infra?

If you want to crawl millions of pages without building infrastructure, the easiest path is to use a managed crawling/extraction service rather than running your own browser fleet, proxies, queues, retries, and storage.

Best options

1) Managed web scraping APIs

These handle:

• fetching pages
• rotating IPs / proxies
• retries / rate limits
• browser rendering for JS sites
• anti-bot handling
• sometimes extraction to structured JSON

Examples:

• Apify
• Zyte
• Bright Data
• ScrapingBee
• Zenscrape
• Oxylabs

Good when you need scale fast and don’t want ops work.

2) Cloud crawl platforms

These are closer to “run a crawler as a service”:

• define URLs, rules, and extraction
• they run workers for you
• can export to S3, BigQuery, webhook, CSV, JSON, etc.

Examples:

• Apify actors
• Zyte API / Scrapy Cloud
• Diffbot for extraction-heavy use cases

3) Headless browser orchestration services

If pages require JS rendering:

• use a service with managed Playwright/Puppeteer
• you write the crawl logic, they manage the compute

Examples:

• Browserless
• Apify
• Zyte

---

A practical approach

For millions of pages, don’t think “single crawl job.” Think:

1. Seed URLs
2. Distribute crawl tasks
3. Deduplicate URLs
4. Throttle by domain
5. Store raw HTML + metadata
6. Extract data separately
7. Resume from checkpoints

Managed platforms usually give you most of this.

---

If you want the simplest setup

A common low-infra stack is:

• Apify or Zyte for crawling
• S3 / GCS for raw page storage
• BigQuery / Postgres / Elasticsearch for processed data
• Airflow / Make / simple scripts only for orchestration, if needed

---

How to choose

Choose managed scraping if:

• you need millions of pages
• sites are dynamic or protected
• you want to avoid proxy/browser maintenance
• speed matters more than lowest possible cost

Choose your own crawler if:

• you have a dedicated infra team
• crawl logic is highly custom
• cost at massive scale outweighs convenience

---

Important considerations

• Respect robots.txt and site terms
• Rate limit per domain
• Expect failures and retries
• Store raw responses so you can reprocess without recrawling
• Use incremental crawling after the initial bulk crawl

---

Recommended starting point

If you want minimal engineering effort:

1. Use Apify or Zyte
2. Start with a small crawl of 10k–50k pages
3. Measure:
   • success rate
   • cost per 1k pages
   • JS rendering needs
   • anti-bot friction
4. Scale to millions once stable

If you want, I can recommend the best service based on your site type:

• static pages
• JS-heavy pages
• e-commerce
• news/content sites
• login-required sites