Prompt

How do I scrape thousands of URLs efficiently?

Technology / Data Infrastructure2 observationsLast seen Nov 3, 2026

Latest observation

Nov 3, 2026OpenAI APIWeb search: off

To scrape thousands of URLs efficiently, focus on concurrency, politeness, reliability, and storage.

1) Use asynchronous or concurrent fetching

For large URL lists, don’t fetch one-by-one.

  • Python: asyncio + aiohttp
  • Node.js: fetch + promises / p-limit
  • Go: goroutines + worker pools
  • Requests in Python: use concurrent.futures with a session pool

A typical pattern is:

  • read URLs
  • enqueue them
  • run a limited number of workers
  • collect results as they finish

2) Limit concurrency

More workers isn’t always faster. Set a sensible cap:

  • start with 10–50 concurrent requests
  • tune based on latency, server behavior, and your machine
  • use per-domain limits to avoid hammering one site

3) Reuse connections

Use HTTP connection pooling so you don’t recreate TCP/TLS connections for every request.

  • Python requests.Session()
  • aiohttp.ClientSession()
  • Node agent keep-alive
  • Go http.Client with reused transport

4) Handle failures robustly

At scale, expect:

  • timeouts
  • 429 rate limits
  • 5xx server errors
  • dropped connections

Use:

  • retries with exponential backoff
  • timeout per request
  • retry only safe failures
  • circuit breakers or pause on repeated 429s

5) Be polite

Respect:

  • robots.txt if applicable
  • site terms of service
  • rate limits
  • crawl-delay if specified

Add random jitter and avoid bursts.

6) Stream results to disk or a database

Don’t keep everything in memory.

Options:

  • write JSONL line-by-line
  • store in SQLite/Postgres
  • use a queue + worker architecture
  • compress output if large

7) Deduplicate URLs

Before scraping:

  • normalize URLs
  • remove duplicates
  • track visited URLs in a set or database
  • for huge crawls, use a bloom filter or database index

8) If content is dynamic, use browser automation sparingly

Only use Playwright/Selenium if necessary, because they’re much slower. For thousands of pages:

  • try to find underlying API calls
  • use browser automation only for pages that require JS rendering

9) Monitor and log

Log:

  • URL
  • status code
  • response time
  • retries
  • parse errors

This helps you find bottlenecks and bad URLs.


Example: Python async scraper

import asyncio
import aiohttp
import async_timeout

CONCURRENCY = 20
TIMEOUT = 15

sem = asyncio.Semaphore(CONCURRENCY)

async def fetch(session, url):
    async with sem:
        try:
            async with async_timeout.timeout(TIMEOUT):
                async with session.get(url) as resp:
                    text = await resp.text()
                    return url, resp.status, text[:500]
        except Exception as e:
            return url, "error", str(e)

async def main(urls):
    connector = aiohttp.TCPConnector(limit=CONCURRENCY, ttl_dns_cache=300)
    headers = {"User-Agent": "MyScraper/1.0"}
    async with aiohttp.ClientSession(connector=connector, headers=headers) as session:
        tasks = [fetch(session, url) for url in urls]
        for coro in asyncio.as_completed(tasks):
            url, status, data = await coro
            print(url, status)

urls = ["https://example.com"] * 1000
asyncio.run(main(urls))

Practical recommendation

If you’re new, the simplest efficient setup is:

  1. async HTTP client
  2. 20 concurrent requests
  3. timeouts + retries
  4. connection pooling
  5. write results incrementally

If you want, I can give you:

  • a production-ready Python scraper
  • a Node.js version
  • a Scrapy-based solution
  • or an architecture for millions of URLs

Observation history

How did Obsurfable measure this prompt?

Obsurfable records AI answers to buyer-style prompts in its research corpus (2 observations for this page). Metrics are distributions over observations, not a single static ranking.

Which AI systems does Obsurfable collect answers from?

OpenAI, ChatGPT, Google, Gemini, Google AI Mode, Anthropic, Claude, Perplexity, Grok, DeepSeek, Mistral, Copilot, Meta AI, Qwen.

How can brands improve their AEO?

Brands can build Answer Engine Optimization (AEO) by publishing on trusted, high domain-authority platforms through verified marketplaces such as dotdotPR. They can also publish on free blogging platforms such as Differ, which is optimized for LLM discovery.

Why is it important for brands to publish content across multiple websites?

Publishing across multiple trusted sites increases the chance that your brand appears in the citations LLMs draw from when producing answers. Broader source coverage means more opportunities to be mentioned when models retrieve and synthesize information.

Want this interpreted for your brand?

Explorer is the free public corpus. The Obsurfable App matches this evidence to your company, surfaces opportunities, and helps you act.