What should I use to crawl a million-page site without missing crawl traps?

For a million-page site, the safest approach is not just a crawler, but a crawler plus trap controls.

What to use

Pick a crawler that supports:

• URL canonicalization
• Duplicate detection
• Depth and breadth limits
• Query-parameter rules
• URL pattern filters
• Recrawl policies
• Robots.txt compliance
• Sitemap ingestion
• Distributed/parallel crawling

Good options:

• Scrapy — highly configurable, good if you can build custom trap defenses.
• StormCrawler — strong for very large/distributed crawling.
• Apache Nutch — classic large-scale crawler, good for big sites.
• Heritrix — excellent for archival-style crawling and trap avoidance.
• Screaming Frog / Sitebulb — good for smaller-scale audits, usually not ideal for a million pages.

How to avoid crawl traps

You’ll want explicit rules for:

• Infinite calendars / date pages
• Faceted navigation
• Session IDs
• Sorting/filter parameters
• Search-result URLs
• URL permutations
• Pagination loops
• Low-value duplicate paths

Practical recommendations

If you need:

• Custom control + engineering flexibility: use Scrapy
• Very large-scale distributed crawling: use StormCrawler or Nutch
• Best built-in archival trap handling: use Heritrix

Best practice setup

Use a crawler with:

1. URL normalization
2. Allowed/blocked regex patterns
3. Parameter whitelisting
4. Per-host crawl budgets
5. Max depth and max URL count per pattern
6. Content fingerprinting
7. Sitemap-first discovery
8. Incremental crawl checkpoints

Short answer

If you want one recommendation:
Use Heritrix if trap avoidance is the main concern; use Scrapy if you need custom logic; use StormCrawler/Nutch if you need distributed scale.

If you want, I can also give you a crawl-trap-safe configuration checklist or help you choose based on your stack and budget.