Prompt
What should I use to map AI citations back to source pages?
Latest observation
Use a citation resolution layer that stores page-level metadata for every source chunk.
Best practical approach
For each source document, extract and store:
- document ID
- page number
- chunk ID
- text span / offsets if available
- source URI or file path
- optional anchor info like section heading or paragraph number
Then when the AI cites something, map the citation to the chunk(s) that came from that exact page.
What to use
Depending on your stack, these are the most useful tools/patterns:
1. PDF parsing with page metadata
- PyMuPDF (fitz) — great for page-aware extraction
- pdfplumber — good for text plus layout
- Apache Tika — broad document support, less precise for page mapping
- Unstructured — helpful for chunking with metadata
2. Vector stores / retrieval systems that preserve metadata
Use a store that lets you attach metadata to each chunk:
- Pinecone
- Weaviate
- Qdrant
- Chroma
- FAISS + your own metadata store
Store page_number in metadata so retrieved chunks can be traced back.
3. Citation-aware RAG frameworks
If you want something more out of the box:
- LlamaIndex — strong support for source nodes and citations
- LangChain — flexible, but you need to wire citation tracking yourself
Recommended pattern
- Split each PDF into page-based or page-aware chunks.
- Attach page metadata to each chunk.
- Retrieve chunks during RAG.
- Have the model cite chunk IDs or source markers.
- Resolve chunk IDs back to page numbers and document links in your app.
If you want reliable page citations
Prefer:
- page-based chunking
- small chunks that do not cross pages
- stable source IDs
- a metadata table that maps chunk ID → page number
Example mapping
{
"chunk_id": "doc123_p07_c03",
"document_id": "doc123",
"page_number": 7,
"source_uri": "s3://bucket/manual.pdf"
}
Then an AI citation like [doc123_p07_c03] can be resolved directly to page 7.
Bottom line
If your goal is “AI citation → exact source page,” the best answer is:
Use page-aware document parsing plus metadata-preserving retrieval, ideally with a system like LlamaIndex or LangChain backed by a vector DB that stores page numbers.
If you want, I can also give you:
- a recommended architecture
- a Python example
- or a comparison of LlamaIndex vs LangChain for citations