How do I set up a catalog enrichment tool for SKU matching and near-duplicate detection?

Setting up a catalog enrichment tool for SKU matching and near-duplicate detection usually involves 5 parts:

1. Define the matching goal
2. Normalize your catalog data
3. Build candidate matching logic
4. Add near-duplicate detection
5. Create a review + feedback loop

Here’s a practical way to do it.

---

1) Define what “match” means

Before building anything, decide what you want to match against.

Common matching targets

• Exact SKU matches: same SKU across feeds/suppliers
• Cross-supplier equivalents: same product, different SKU
• Near duplicates: same or very similar item with small differences
• Variant relations: same product family, different size/color/flavor

Decide which fields matter most

Typical fields:

• SKU
• MPN / manufacturer part number
• Brand
• Title
• Description
• Category
• GTIN / UPC / EAN
• Size, color, pack count, unit type
• Image URL or image hash

---

2) Normalize the catalog

You’ll get much better results if you standardize the raw product data first.

Normalize text

• lowercase
• trim whitespace
• remove punctuation where appropriate
• normalize unicode
• standardize abbreviations

Example:

• 12 Oz. → 12 oz
• Pack of 6 → 6 pack

Normalize structured attributes

Create canonical forms for:

• units: oz, lb, ml, kg
• colors: navy blue → blue
• sizes: XL, extra large
• pack counts: 2-pack, pack of 2

Parse key identifiers

Extract:

• GTIN/UPC/EAN
• MPN
• model numbers
• brand tokens

These identifiers are usually the strongest signals for matching.

---

3) SKU matching workflow

A good SKU matching pipeline is usually hierarchical:

Step A: Exact identifier matching

Match on:

• identical SKU
• UPC/EAN/GTIN
• MPN + brand

This gives high precision.

Step B: Fuzzy text matching

If exact matching fails, compare:

• title similarity
• brand similarity
• attribute overlap
• description similarity

Useful methods:

• token overlap
• Jaccard similarity
• cosine similarity with embeddings
• edit distance for short strings

Step C: Rules-based scoring

Build a match score from weighted signals.

Example scoring:

• UPC exact match: +100
• MPN exact match: +80
• Brand exact match: +20
• Title similarity > 0.85: +20
• Same category: +10
• Size mismatch: -30

Then:

• score > threshold → auto-match
• score in middle → human review
• score too low → no match

---

4) Near-duplicate detection

Near duplicates are not always exact matches, so use more flexible methods.

Good approaches

A. Text similarity

Compare normalized titles/descriptions:

• TF-IDF + cosine similarity
• sentence embeddings
• MinHash / SimHash for large-scale dedupe

B. Attribute-based similarity

Compare fields like:

• brand
• size
• pack count
• color
• material
• category

This helps distinguish:

• “Coke Zero 12 oz 12-pack” vs
• “Coke Zero 20 oz 12-pack”

C. Blocking / candidate generation

Don’t compare every item to every other item. First group likely candidates by:

• brand
• category
• first few tokens
• MPN prefix
• GTIN presence

This makes matching much faster.

---

5) Recommended architecture

A practical setup looks like this:

Data pipeline

1. Ingest catalog data from suppliers/internal DB
2. Normalize fields and parse identifiers
3. Generate candidate pairs
4. Score matches
5. Store match results
6. Send uncertain cases to review
7. Use review feedback to improve rules/model

Storage

Keep:

• raw source records
• normalized records
• match candidates
• match scores
• final match decisions
• audit trail

This is important for traceability.

---

6) Tooling options

Lightweight / early-stage

• Python
• pandas
• rapidfuzz
• scikit-learn
• sentence-transformers
• PostgreSQL

More scalable

• Spark / Databricks for large catalogs
• Elasticsearch / OpenSearch for candidate retrieval
• Vector database for embedding-based similarity
• Workflow tool like Airflow or Dagster

Useful libraries

• rapidfuzz for fuzzy string matching
• dedupe for entity resolution
• splink for probabilistic matching at scale
• sentence-transformers for semantic similarity
• recordlinkage for pairwise matching workflows

---

7) Human review interface

You’ll want a reviewer to see:

• source product
• candidate match
• match score
• matching fields
• conflicting fields
• decision buttons: match / no match / variant / duplicate

This improves quality and creates labeled data for tuning.

---

8) A simple implementation strategy

Phase 1: Exact matching

Start with:

• SKU
• UPC/EAN
• MPN + brand

Phase 2: Rules + fuzzy matching

Add:

• normalized title similarity
• category/brand checks
• size/pack parsing

Phase 3: ML or embeddings

Use:

• product title embeddings
• attribute-aware similarity model
• active learning from human reviews

---

9) Example matching logic

A simple rule:

If UPC matches exactly -> match
Else if MPN and brand match -> match
Else if brand matches AND title similarity > 0.90 AND size/pack matches -> likely match
Else if title similarity > 0.95 but size differs -> possible variant
Else no match

---

10) Best practices

• Prefer exact identifiers when available
• Normalize aggressively
• Separate duplicates from variants
• Keep explainable scores
• Use human review for borderline cases
• Measure precision and recall
• Continuously retrain/tune based on review outcomes

---

11) Metrics to track

For enrichment quality:

• precision
• recall
• F1 score
• auto-match rate
• review rate
• false positive rate
• duplicate detection rate

For operations:

• throughput
• processing time
• percentage matched by exact ID vs fuzzy
• reviewer agreement rate

---

12) Suggested first version

If you want the fastest path to a working tool:

1. Normalize SKU/title/brand/MPN/GTIN
2. Exact match on GTIN, SKU, MPN+brand
3. Use fuzzy title matching for fallback
4. Add pack-size and unit parsing
5. Flag uncertain matches for manual review
6. Store decisions and iterate

---

If you want, I can also give you:

• a system design diagram
• a database schema
• a Python starter implementation
• or a matching rules template for SKU enrichment.