How do I chunk PDFs and wiki pages for better retrieval?

To get better retrieval, chunk by semantic boundaries rather than fixed character counts alone. PDFs and wiki pages behave differently, so it helps to use slightly different strategies for each.

Core principles

• Keep chunks semantically coherent: one topic per chunk.
• Preserve context: include titles, section headings, and parent section info.
• Avoid overly large chunks: they reduce recall precision.
• Avoid overly small chunks: they lose meaning and context.
• Use overlap sparingly: enough to bridge boundaries, not so much that you duplicate everything.

---

Good chunking strategy for PDFs

PDFs often have messy layout, columns, headers, footers, tables, and page breaks.

1) Extract and clean text first

• Remove repeated headers/footers.
• Fix hyphenation across line breaks.
• Merge wrapped lines into paragraphs.
• Detect and preserve section headings if possible.
• Keep page numbers as metadata, not inline text.

2) Chunk by structure when available

Prefer:

• Title
• Section
• Subsection
• Paragraph

A good unit is often:

• One subsection, or
• A few paragraphs under a heading

3) Use size-based fallback

If structure is poor, chunk by token count:

• 300–800 tokens per chunk is a common starting range
• 10–20% overlap between adjacent chunks

4) Handle tables and figures carefully

• Tables should often be stored as separate chunks.
• Add a short textual summary if the table is important.
• Figure captions can be included with nearby explanatory text.

5) Add metadata

Store:

• Document title
• Page range
• Section title
• Subsection title
• Source file
• Document ID

This helps retrieval and reranking.

---

Good chunking strategy for wiki pages

Wiki pages are usually more structured and cleaner than PDFs.

1) Chunk by headings

Use the page hierarchy:

• Page title
• Heading 1
• Heading 2
• Heading 3

Each chunk should usually contain:

• The heading path
• The paragraph(s) under that heading

2) Keep heading context in every chunk

Example:

• Page: Kubernetes
• Section: Pods > Lifecycle > Restart Policy

That helps a retrieval system understand the chunk even if it’s small.

3) Don’t split lists or procedure steps mid-way

If a page contains:

• bullet lists
• numbered steps
• FAQs
• policy items

Keep each logical list or step group intact.

4) Use moderate chunk sizes

Often:

• 200–600 tokens per chunk
• Slight overlap if a section is long

Wiki pages are usually concise, so smaller chunks often work well.

---

Practical heuristic

A robust default is:

• Chunk at section boundaries first
• If a section is too long, split by paragraph
• If still too long, split by sentence boundaries with overlap
• Keep chunks around ~400 tokens
• Use 50–100 token overlap only when needed

---

Retrieval-specific tips

1) Use hierarchical retrieval if possible

Retrieve:

1. Relevant page/section
2. Then relevant chunk within that section

This often works better than flat chunking alone.

2) Put the title/heading into the chunk text

Not just metadata. Retrieval models often benefit from seeing the heading in the actual text.

Example:

Title: API Authentication
Section: API Keys
API keys are used to authenticate requests...

3) Use reranking

After initial vector retrieval, rerank top results with a cross-encoder or LLM-based reranker.

4) Tune chunk size by document type

• Dense technical docs: smaller chunks
• Narrative docs: larger chunks
• FAQs: one question-answer pair per chunk
• Policies/specs: one clause or subsection per chunk

---

Example chunking patterns

PDF example

Original structure:

• 1. Introduction
• 1.1 Background
• 1.2 Scope
• 2. Methods
• 2.1 Data collection
• 2.2 Analysis

Chunks:

1. Introduction + Background
2. Scope
3. Methods + Data collection
4. Analysis

Wiki example

Original structure:

• Overview
• Installation
• Configuration
• Troubleshooting

Chunks:

1. Overview
2. Installation steps
3. Configuration subsections
4. Troubleshooting entries

---

Recommended starting settings

If you need a default:

• PDFs: 400–800 tokens, 10–15% overlap, structure-aware
• Wiki pages: 200–500 tokens, minimal overlap, heading-aware

---

If you want, I can also give you:

1. a Python chunking pipeline for PDFs and wiki HTML, or
2. a chunking recipe for LangChain/LlamaIndex.