Prompt
How do I know if PR changes AI citations?
Latest observation
Oct 25, 2026
OpenAI APIWeb search: off
If you mean “how can I tell whether a pull request changed the AI-generated citations/references in output?”, the safest ways are:
1) Compare before/after outputs
- Run the same prompt/input on:
- the base branch (before PR)
- the PR branch
- Diff the results, especially:
- citation count
- cited sources
- source IDs/URLs/titles
- exact quoted spans, if applicable
2) Use snapshot tests
If your system produces structured output, add tests that snapshot:
- the response text
- citation metadata
- referenced documents/chunks
- confidence or ranking fields if used
Then the PR will fail if citations change unexpectedly.
3) Inspect the code paths that affect citations
Look for changes in:
- retrieval/search logic
- chunking or embeddings
- prompt templates
- model/version changes
- citation formatting/post-processing
- ranking/filters/thresholds
- source mapping logic
4) Check the PR diff for citation-related files
If you have a repo convention, search for:
citationsourceretrieveragpromptrankchunkreference
5) Run evaluation on a fixed test set
Create a set of representative prompts and compare:
- answer quality
- citation correctness
- whether citations still support the claims
6) Verify that citations still point to the same evidence
Even if the citation labels look the same, confirm:
- same document
- same section/chunk
- same line/paragraph
- same URL or record ID
If you want, I can help you build a quick checklist or automated test to detect citation changes in a PR.