Prompt
How do I troubleshoot duplicate chunks in vector search?
Latest observation
Duplicate chunks in vector search usually come from one of a few places: ingestion, chunking, embedding, indexing, or retrieval logic. Here’s a practical way to troubleshoot it.
1) First identify what “duplicate” means
There are a few different cases:
- Exact duplicate text chunks: same text appears multiple times in your source or pipeline.
- Near-duplicate chunks: small differences in whitespace, punctuation, headers, or metadata.
- Same content retrieved multiple times: the index contains duplicates, or the retriever is returning multiple chunks from the same source window.
- Repeated results in UI: retrieval is fine, but your display layer is merging or re-splitting results badly.
2) Check your ingestion pipeline
Most duplicates start here.
Common causes
- The source file is ingested more than once
- Re-runs without idempotency.
- Multiple connectors pointing to the same content.
- Document IDs are not stable
- If you generate random IDs on each run, reindexing creates duplicates.
- Incremental sync bug
- Changed files are re-added instead of updated.
- Multiple preprocessing passes
- Same document goes through two loaders or two chunkers.
What to inspect
Log for each chunk:
- source document ID
- chunk text hash
- chunk index / offset
- embedding model version
- index insert/upsert status
If the same (source_id, chunk_hash) appears multiple times, you’ve found the issue.
3) Verify chunking behavior
Chunkers can create accidental overlaps that look like duplicates.
Typical causes
- High overlap
- Example: 500-token chunks with 400-token overlap will repeat a lot.
- Repeated boilerplate
- Headers, footers, navigation text, disclaimers.
- Recursive splitters
- Can produce chunks that differ only slightly.
- Bad dedup after splitting
- Same paragraph gets split, then reassembled incorrectly.
What to check
- chunk size
- overlap size
- whether boilerplate is stripped before chunking
- whether identical chunks are dropped after chunking
A good quick test:
- Print the first 20 chunks and compare them side by side.
- Hash normalized chunk text:
- lowercase
- trim whitespace
- collapse repeated spaces
- remove punctuation if appropriate
4) Check embedding + indexing logic
Even if text is unique, duplicates may appear in the vector store.
Common causes
- Using “insert” instead of “upsert”
- No unique constraint on chunk ID
- Re-embedding the same content with a new ID
- Index recreated partially
- Hybrid systems storing both raw and processed copies
Fixes
- Use a deterministic chunk ID:
doc_id + chunk_start + chunk_end- or a hash of normalized text + source ID
- Make the write operation idempotent:
- upsert by chunk ID
- Store a content hash so you can detect duplicates before insertion
5) Check retrieval logic
Sometimes the index is fine, but retrieval returns duplicates because of your pipeline.
Typical causes
- Top-k results include multiple chunks from the same passage
- Reranker doesn’t deduplicate
- Merging results from multiple queries without dedupe
- Multi-vector retrieval returns overlapping passages
- Parent-child retrieval returns both child chunks and parent docs
What to do
- Deduplicate by:
- exact
chunk_id - then by normalized text hash
- then optionally by parent document ID
- exact
- Add a “max results per document” cap
- If using parent-child retrieval, decide whether to show only child chunks or only parent docs
6) Look for boilerplate and repeated structure
Many document sources repeat the same content across pages or sections:
- page headers/footers
- navigation menus
- legal notices
- “confidential” stamps
- table of contents entries
- repeated templates in tickets or emails
These can dominate retrieval and look like duplicates.
Solutions
- strip page artifacts during ingestion
- remove boilerplate with heuristics
- segment by semantic structure instead of fixed-size windows
- normalize whitespace and line breaks
7) Validate with a simple dedupe audit
A quick audit script can tell you where duplication is happening.
Audit checks
Group chunks by:
chunk_id- normalized text hash
(source_id, normalized text hash)
Then count duplicates:
- If same
chunk_idappears twice → ingestion/index bug - If same normalized text appears across many chunk IDs → chunking/boilerplate issue
- If duplicates only appear in query results → retrieval/merging issue
8) Suggested debugging workflow
- Pick one duplicated result from search.
- Trace it back to:
- source document
- chunk ID
- text hash
- insertion time
- Check whether it exists multiple times in the vector DB.
- If yes, inspect ingestion logs.
- If no, inspect retrieval and result-merging logic.
9) Prevention best practices
- Use deterministic IDs for chunks
- Store content hashes
- Upsert, don’t blindly insert
- Normalize text before hashing
- Remove boilerplate
- Deduplicate retrieval results before display
- Add tests for re-ingestion idempotency
10) If you want a quick “most likely” checklist
In order of likelihood:
- The same document was ingested twice
- Chunk IDs aren’t stable, so reindexing creates duplicates
- Overlapping chunks are too large
- Boilerplate text is repeated across chunks
- Retrieval results aren’t deduplicated before returning
If you want, I can also give you:
- a Python dedupe script for vector chunks,
- a debug checklist for Pinecone/FAISS/Chroma/Weaviate, or
- a schema for making chunk ingestion idempotent.