How do I queue and retry failed crawl requests?

To queue and retry failed crawl requests, the usual pattern is:

1. Persist each crawl request in a durable queue or database with a status field.
2. Process jobs asynchronously with a worker.
3. Mark failures and increment a retry counter.
4. Re-enqueue with backoff until a max retry limit is reached.
5. Move permanently failed jobs to a dead-letter queue or error table for inspection.

Common design

Job record fields

Store something like:

• id
• url
• status = queued | processing | failed | succeeded | dead
• retry_count
• max_retries
• next_attempt_at
• last_error
• created_at
• updated_at

Processing flow

1. Worker pulls a job where:
   • status = queued
   • next_attempt_at <= now()
2. Worker attempts crawl.
3. If success:
   • set status = succeeded
4. If failure:
   • increment retry_count
   • if retry_count < max_retries:
     • set status = queued
     • set next_attempt_at = now() + backoff
   • else:
     • set status = dead

Retry strategy

Use exponential backoff to avoid hammering the target or your own system:

• 1st retry: after 1 minute
• 2nd retry: after 5 minutes
• 3rd retry: after 15 minutes
• 4th retry: after 1 hour

Add jitter so many failed jobs don’t retry at exactly the same time.

Example pseudocode

def process_job(job):
    try:
        crawl(job.url)
        job.status = "succeeded"
        job.last_error = None
    except Exception as e:
        job.retry_count += 1
        job.last_error = str(e)

        if job.retry_count < job.max_retries:
            job.status = "queued"
            job.next_attempt_at = now() + backoff(job.retry_count)
        else:
            job.status = "dead"

    save(job)

If you’re using a queue system

Redis / Celery / Sidekiq / BullMQ / RabbitMQ

Most queue frameworks already support retries:

• Celery: retry, autoretry_for, countdown
• Sidekiq: automatic retries + dead jobs
• BullMQ: attempts, backoff, stalled job handling
• AWS SQS: visibility timeout + redrive policy

For example, in BullMQ you can set:

• attempts: 5
• backoff: { type: 'exponential', delay: 1000 }

Best practices

• Retry only transient failures: timeouts, 429s, 5xx, network errors.
• Don’t retry on permanent failures: invalid URL, 404 if you consider it terminal, auth failure, robots denial.
• Log the failure reason for each retry.
• Use rate limiting to avoid getting blocked.
• Deduplicate requests so the same URL isn’t crawled repeatedly in parallel.
• Add observability: counts of queued, failed, retried, dead jobs.

If you want, I can show this in a specific stack like Python/Celery, Node/BullMQ, Go, or SQL-only.