How do I set up a site ingestion platform for crawling public pages with deduplication and canonicalization?

Here’s a practical way to set up a site ingestion platform for crawling public pages with deduplication and canonicalization.

1) Define the pipeline

A good ingestion system usually has these stages:

1. Seed collection

   • Start with a list of allowed domains/URLs.
   • Respect robots.txt, crawl-delay, rate limits, and legal/ToS constraints.

2. URL normalization + canonicalization

   • Normalize URLs before fetching and before dedup checks.
   • After fetch, resolve page-level canonical hints.

3. Fetch

   • HTTP GET with retries, timeouts, content-type checks, and polite concurrency.

4. Content extraction

   • Extract main text, metadata, links, structured data.

5. Duplicate detection

   • Detect exact duplicates and near-duplicates.
   • Store only one canonical version, or link duplicates to a canonical record.

6. Storage and indexing

   • Raw HTML in object storage.
   • Parsed content and metadata in a database/search index.

---

2) URL canonicalization

Normalize URLs to reduce accidental duplicates.

Common normalization rules

• Lowercase scheme and host.
• Remove default ports (:80, :443).
• Remove URL fragments (#section).
• Normalize percent-encoding.
• Remove tracking params (utm_*, fbclid, etc.).
• Sort query params if your policy allows.
• Normalize trailing slashes consistently.
• Handle www. consistently if appropriate for your domain policy.

Example normalized forms

• HTTPS://Example.com:443/Page/ → https://example.com/Page/
• https://example.com/page?utm_source=x&id=1&id=1 → https://example.com/page?id=1

Important

Canonicalization is policy-driven:

• For some sites, ?page=2 and ?page=3 are distinct.
• For some sites, query order matters.
• So implement a safe normalization layer and a domain-specific override layer.

---

3) Respect page canonical signals

After fetching, inspect the page for canonical hints:

HTML signals

• <link rel="canonical" href="...">
• <meta property="og:url" content="...">

HTTP signals

• Redirects (301, 302, 307, 308)
• Content-Location header

Handling strategy

1. Fetch the requested URL.
2. Follow redirects.
3. Parse canonical link.
4. Prefer canonical URL if it is valid and within policy.
5. Store:
   • requested_url
   • final_url
   • declared_canonical_url
   • effective_canonical_url

Validation

Only trust canonical URLs if:

• same site or allowed domain set
• valid HTTP/HTTPS URL
• not obviously malicious or cross-domain unless policy allows it

---

4) Deduplication strategy

Use multiple levels:

A. Exact deduplication

Detect identical content or identical normalized HTML.

Methods:

• Hash of normalized raw HTML
• Hash of extracted text
• Hash of main content section after boilerplate removal

Good for:

• Repeated mirrors
• Same page fetched multiple times
• Minor URL variations

B. Near-duplicate detection

Use when pages are mostly the same but not identical.

Methods:

• SimHash
• MinHash / LSH
• Shingling over text
• Content fingerprints over token windows

Good for:

• Pagination variants
• Syndicated content
• Slightly modified copies

C. Canonical record model

Store one canonical document, and attach duplicates as aliases:

• document_id
• canonical_document_id
• duplicate_reason
• similarity_score

---

5) Recommended data model

A simple schema:

crawl_queue

• url
• priority
• discovering_source
• crawl_status
• next_fetch_at

fetch_log

• requested_url
• final_url
• http_status
• content_type
• etag
• last_modified
• fetch_time
• robots_allowed

documents

• document_id
• canonical_url
• title
• published_at
• language
• content_hash
• simhash
• main_text
• html_uri
• created_at

document_aliases

• alias_url
• document_id
• alias_type (redirect, rel_canonical, duplicate, normalized_variant)

---

6) Crawl architecture

A production-friendly architecture:

Components

• Scheduler: decides what to crawl next
• Fetcher workers: download pages
• Parser workers: extract metadata/content/links
• Dedup service: exact + near-duplicate checks
• Storage: raw blobs + structured database
• Search/index: OpenSearch/Elasticsearch or similar

Message flow

1. Scheduler enqueues URL
2. Fetcher downloads page
3. Parser extracts links and canonical info
4. Dedup service checks hashes/fingerprints
5. Store document and enqueue discovered links

Use a queue like:

• Kafka
• RabbitMQ
• SQS/PubSub

---

7) Crawling rules and politeness

For public web crawling:

• Obey robots.txt
• Identify your crawler with a clear User-Agent
• Rate limit per host
• Back off on errors and 429
• Cache robots results
• Avoid bypassing access controls or login walls

Implement:

• per-host concurrency limits
• per-host request intervals
• global throughput controls
• retry policy with exponential backoff

---

8) Content extraction

Don’t dedup on raw HTML alone if your goal is content identity.

Recommended:

1. Strip scripts/styles/nav/footer
2. Extract article/main body
3. Normalize whitespace
4. Lowercase only if appropriate
5. Remove boilerplate and ads
6. Compute fingerprints

Tools:

• Readability-style extraction
• trafilatura
• boilerpy3
• custom DOM heuristics

---

9) Dedup logic example

A practical precedence order:

1. Redirect target if HTTP redirect exists
2. rel=canonical if valid
3. Normalized URL if no better signal
4. Content hash / fingerprint for duplicate grouping

Decision:

• If content hash matches an existing doc exactly, link as duplicate.
• Else if SimHash distance below threshold, mark near-duplicate.
• Else create new canonical doc.

---

10) Example processing pseudocode

def process_url(url):
    normalized = normalize_url(url)
    if not allowed_by_robots(normalized):
        return

    resp = fetch(normalized)
    if not resp.ok:
        log_failure(normalized, resp.status_code)
        return

    final_url = normalize_url(resp.final_url)
    canonical_hint = extract_canonical(resp.html)

    effective_canonical = choose_canonical(
        requested=normalized,
        final=final_url,
        declared=canonical_hint
    )

    text = extract_main_text(resp.html)
    content_hash = sha256(normalize_text(text))
    simhash = compute_simhash(text)

    existing = find_by_hash(content_hash)
    if existing:
        store_alias(normalized, existing.id, reason="exact_duplicate")
        return

    near = find_near_duplicate(simhash)
    if near and distance(simhash, near.simhash) < THRESHOLD:
        store_alias(normalized, near.id, reason="near_duplicate")
        return

    doc_id = insert_document(
        canonical_url=effective_canonical,
        title=extract_title(resp.html),
        main_text=text,
        content_hash=content_hash,
        simhash=simhash,
        html_uri=store_raw_html(resp.html)
    )

    if effective_canonical != normalized:
        store_alias(normalized, doc_id, reason="canonicalized")

---

11) Operational considerations

Freshness

• Re-crawl based on change signals:
  • ETag
  • Last-Modified
  • content fingerprints
  • sitemap updates
• Use adaptive recrawl schedules.

Scale

• Start with one worker pool and shared queue.
• Partition by host/domain.
• Shard dedup index by hash prefix or domain.

Observability

Track:

• fetch success rate
• robots blocked rate
• duplicate rate
• near-duplicate rate
• crawl latency
• per-host errors

---

12) Recommended implementation stack

A common stack:

• Crawler: Python + asyncio or Go
• Queue: Kafka/RabbitMQ/SQS
• Storage: S3/GCS for HTML, Postgres for metadata
• Search: OpenSearch/Elasticsearch
• Dedup: hash index + SimHash/MinHash service
• Parsing: trafilatura/readability + BeautifulSoup/lxml

---

13) Minimum viable version

If you want a simpler v1:

• URL normalization
• robots.txt compliance
• fetch + redirect handling
• store raw HTML
• extract canonical URL
• exact dedup with SHA-256 on normalized text
• canonical URL alias table

Then add:

• SimHash near-duplicate detection
• recrawl scheduling
• content change detection
• host-based politeness

---

If you want, I can also provide:

1. a reference architecture diagram,
2. a Postgres schema, or
3. a Python crawler skeleton with canonicalization and dedup built in.