How do I get my crawler unstuck from blocked requests?

A crawler usually gets “stuck” on blocked requests because it keeps retrying the same failing URLs, slows down too much after throttling, or waits forever on responses it should abandon. The fix is to make blocking detectable, bounded, and recoverable.

What to do

1. Set strict timeouts

Don’t let requests hang indefinitely.

• Connect timeout: e.g. 5–10s
• Read/response timeout: e.g. 10–30s
• Total timeout: e.g. 30–60s

If your crawler framework supports it, enforce:

• request timeout
• retry timeout budget
• per-domain timeout caps

2. Limit retries on blocked responses

If a request returns:

• 403 Forbidden
• 429 Too Many Requests
• CAPTCHA / challenge page
• unusual HTML indicating a block

then do not retry forever.

Use:

• max retries: 1–3
• exponential backoff with jitter
• a “give up” threshold per domain/IP/session

3. Detect blocking explicitly

Add rules to recognize blocked pages and responses:

• HTTP status codes: 401, 403, 407, 429, 503
• “Access denied”, “unusual traffic”, “bot detected”
• CAPTCHA forms, Cloudflare/Akamai challenge pages
• zero-length or suspiciously small responses
• repeated redirects to login/challenge pages

When detected, mark the URL as blocked and move on.

4. Separate failures by category

Treat these differently:

• Transient network failure → retry a few times
• Rate limiting (429) → slow down, backoff, maybe rotate identity
• Permanent block (403) → stop retrying, quarantine domain/session
• Challenge page → classify and skip or escalate to manual handling

5. Use per-domain concurrency and rate limits

A lot of “stuck” behavior comes from overloading one site.

• limit concurrent requests per host
• add random delays
• reduce crawl depth or burst size for blocked hosts
• adaptive throttling: if block rate rises, slow down automatically

6. Rotate identity carefully

If appropriate and allowed for your use case:

• rotate IPs/proxies
• rotate user agents
• maintain cookies/sessions correctly

But don’t rely on this alone; combine it with backoff and stop conditions.

7. Add a dead-letter queue / quarantine

Push repeatedly blocked URLs into a separate queue:

• blocked queue
• retry-later queue
• manual-review queue

That prevents them from clogging the main crawl.

8. Keep crawl state resumable

If the crawler process restarts or hangs:

• persist frontier state
• checkpoint visited URLs
• save in-flight request state with a lease/TTL
• requeue expired leases automatically

9. Use circuit breakers

If a domain starts blocking heavily:

• trip a circuit breaker
• pause that domain for a cooling-off period
• resume later with lower rate

Example rule:

• if 5 of last 10 requests to a host are blocked, pause host for 15 minutes

10. Monitor block rate and latency

Track:

• block rate by domain
• average response time
• retry counts
• timeout counts
• queue backlog

This helps you see whether the crawler is actually blocked, throttled, or just slow.

A practical policy

A good default policy looks like this:

• timeout: 15s connect, 30s total
• retries: 2 for transient errors
• on 429: exponential backoff and lower concurrency
• on 403 or challenge: stop retrying, mark blocked
• per-host circuit breaker: pause after repeated blocks
• queue blocked URLs separately

If you’re using a framework

Most crawler frameworks have built-in support for this:

• Scrapy: DOWNLOAD_TIMEOUT, RETRY_TIMES, AutoThrottle, retry middleware, custom ban detection
• Python requests/httpx/aiohttp: explicit timeouts + retry logic + exception handling
• Playwright/Selenium: page load timeouts, navigation timeouts, block detection, browser context recycling

Short version

To get unstuck:

1. add hard timeouts,
2. stop infinite retries,
3. detect 403/429/challenges,
4. back off per domain,
5. quarantine blocked URLs,
6. persist crawl state.

If you want, I can show:

• a Scrapy-specific setup,
• a Python requests/httpx retry pattern, or
• a generic crawler state machine for handling blocked URLs.