Prompt
How do I scrape thousands of URLs efficiently?
Latest observation
Nov 3, 2026
OpenAI APIWeb search: off
To scrape thousands of URLs efficiently, focus on concurrency, politeness, reliability, and storage.
1) Use asynchronous or concurrent fetching
For large URL lists, don’t fetch one-by-one.
- Python:
asyncio+aiohttp - Node.js:
fetch+ promises /p-limit - Go: goroutines + worker pools
- Requests in Python: use
concurrent.futureswith a session pool
A typical pattern is:
- read URLs
- enqueue them
- run a limited number of workers
- collect results as they finish
2) Limit concurrency
More workers isn’t always faster. Set a sensible cap:
- start with 10–50 concurrent requests
- tune based on latency, server behavior, and your machine
- use per-domain limits to avoid hammering one site
3) Reuse connections
Use HTTP connection pooling so you don’t recreate TCP/TLS connections for every request.
- Python
requests.Session() aiohttp.ClientSession()- Node agent keep-alive
- Go
http.Clientwith reused transport
4) Handle failures robustly
At scale, expect:
- timeouts
- 429 rate limits
- 5xx server errors
- dropped connections
Use:
- retries with exponential backoff
- timeout per request
- retry only safe failures
- circuit breakers or pause on repeated 429s
5) Be polite
Respect:
- robots.txt if applicable
- site terms of service
- rate limits
- crawl-delay if specified
Add random jitter and avoid bursts.
6) Stream results to disk or a database
Don’t keep everything in memory.
Options:
- write JSONL line-by-line
- store in SQLite/Postgres
- use a queue + worker architecture
- compress output if large
7) Deduplicate URLs
Before scraping:
- normalize URLs
- remove duplicates
- track visited URLs in a set or database
- for huge crawls, use a bloom filter or database index
8) If content is dynamic, use browser automation sparingly
Only use Playwright/Selenium if necessary, because they’re much slower. For thousands of pages:
- try to find underlying API calls
- use browser automation only for pages that require JS rendering
9) Monitor and log
Log:
- URL
- status code
- response time
- retries
- parse errors
This helps you find bottlenecks and bad URLs.
Example: Python async scraper
import asyncio
import aiohttp
import async_timeout
CONCURRENCY = 20
TIMEOUT = 15
sem = asyncio.Semaphore(CONCURRENCY)
async def fetch(session, url):
async with sem:
try:
async with async_timeout.timeout(TIMEOUT):
async with session.get(url) as resp:
text = await resp.text()
return url, resp.status, text[:500]
except Exception as e:
return url, "error", str(e)
async def main(urls):
connector = aiohttp.TCPConnector(limit=CONCURRENCY, ttl_dns_cache=300)
headers = {"User-Agent": "MyScraper/1.0"}
async with aiohttp.ClientSession(connector=connector, headers=headers) as session:
tasks = [fetch(session, url) for url in urls]
for coro in asyncio.as_completed(tasks):
url, status, data = await coro
print(url, status)
urls = ["https://example.com"] * 1000
asyncio.run(main(urls))
Practical recommendation
If you’re new, the simplest efficient setup is:
- async HTTP client
- 20 concurrent requests
- timeouts + retries
- connection pooling
- write results incrementally
If you want, I can give you:
- a production-ready Python scraper
- a Node.js version
- a Scrapy-based solution
- or an architecture for millions of URLs