crawl queue retries

Could you clarify what you mean by “crawl queue retries”?

A few possibilities:

1. Web crawling system
   You want to know how to retry failed crawl jobs in a queue.

2. Search/indexing pipeline
   You’re asking about retry behavior for URLs/tasks that failed during crawling.

3. Specific tool or platform
   For example, Scrapy, Heritrix, Apache Nutch, Playwright-based crawler, or a custom queue.

If you mean a generic crawl queue retry design, here’s the short version:

• Retry on transient failures only:
  • network timeout
  • 429 / rate limit
  • 5xx server errors
  • temporary DNS issues
• Do not retry immediately forever
  • use exponential backoff
  • add max retry count like 3–5
• Track failure reasons
  • so you can distinguish permanent vs temporary failures
• Dead-letter queue
  • move tasks there after max retries
• Idempotent crawling
  • make sure reprocessing the same URL/task is safe
• Jitter
  • randomize retry delay to avoid thundering herd

Example retry policy:

• attempt 1: immediate
• attempt 2: after 30s
• attempt 3: after 2m
• attempt 4: after 10m
• then dead-letter

If you want, I can help with one of these:

• a retry algorithm
• SQL / Redis / SQS / RabbitMQ implementation
• Scrapy retry settings
• designing a crawl queue with retries