How do I dedupe customer profiles coming from multiple systems?

Deduping customer profiles across multiple systems usually works best as a match + merge pipeline:

1) Define what “same customer” means

Start with a business rule. For example:

• Exact same email = same person
• Same phone + name similarity = likely same person
• Same government ID / account number = definite match
• Households may need separate logic from individuals

Without this definition, dedupe becomes inconsistent.

2) Standardize the data first

Normalize fields so comparisons are reliable:

• Emails: lowercase, trim, remove plus-tags if appropriate
• Phones: E.164 format
• Names: trim, lowercase, remove punctuation, handle nicknames
• Addresses: standardize abbreviations and use an address parser
• Dates: normalize formats
• Empty / placeholder values: treat consistently

3) Use a matching strategy with tiers

A common approach:

Deterministic matches

High-confidence exact rules:

• Same email
• Same phone
• Same external ID
• Same verified national ID

Probabilistic / fuzzy matches

For likely duplicates:

• Similar name
• Similar address
• Same birthdate
• Same company/domain
• Behavioral signals, if available

Give each signal a weight and calculate a match score.

4) Create a single customer “golden record”

Once records match, decide how to merge them:

• Keep the most trusted source for each field
• Use newest verified value when conflict exists
• Preserve provenance: which system supplied each field
• Store all source record IDs linked to the master profile

This is important so you can explain why profiles were merged.

5) Build survivorship rules

For each attribute, define which source wins:

• CRM beats marketing tool for account owner
• Billing system beats web form for legal name
• Most recently verified address wins over older address
• Manual review beats automated guess

6) Handle false positives and false negatives

You need a way to manage uncertainty:

• High-confidence auto-merge
• Medium-confidence queue for manual review
• Low-confidence no merge

Track:

• Precision: are merges correct?
• Recall: are you missing duplicates?
• Merge reversal rate: how often humans undo a match?

7) Use an identity graph if the problem is large

If customers appear under many IDs across systems, model them as a graph:

• Nodes = source records / identifiers
• Edges = evidence of sameness
• Connected component = probable customer entity

This works well when identity links are indirect.

8) Prevent future duplicates

Deduping once is not enough. Add controls upstream:

• Real-time duplicate checks on create/update
• Identity resolution service
• Standard validation at intake
• Match against existing master profiles before creating a new one

9) Keep auditability

Store:

• Source record IDs
• Match reason
• Match score
• Timestamp
• Merge history
• Who approved manual merges

This helps with compliance and debugging.

10) Start simple, then improve

A practical rollout:

1. Exact match on strong identifiers
2. Normalize data
3. Add fuzzy matching
4. Introduce survivorship rules
5. Add manual review
6. Move to identity graph / ML if needed

Example rule set

• Auto-merge if:
  • same verified email, or
  • same customer ID across systems
• Review if:
  • same name + same phone, or
  • name similarity > threshold + same address
• Don’t merge if:
  • conflicting DOB and no strong identifier

Common pitfalls

• Merging households and individuals together
• Treating shared emails as unique identity
• Not normalizing phone/address data
• Losing source provenance
• Over-merging based on weak signals
• No manual review for ambiguous matches

If you want, I can also help you design:

• a matching rules engine
• a customer identity graph schema
• or a SQL / Python dedupe implementation.