How do I find out what pages AI is pulling from?

You can figure this out in a few different ways, depending on whether you mean:

1. Which pages AI search tools are citing
2. Which pages an AI bot is crawling from your site
3. Which pages an LLM is using internally for a specific answer

Here’s how to check each.

1) If you mean “what pages are AI tools citing?”

Many AI search/chat products show their sources directly.

Examples

• ChatGPT with browsing / citations: look for linked sources or footnotes in the response.
• Perplexity: shows source cards and citations prominently.
• Google AI Overviews: sometimes shows source links under the summary.
• Bing / Copilot: may show cited websites in the answer panel.

What to do

• Ask the AI the question again and look for:
  • footnotes
  • source links
  • “used sources”
  • cited domains
• Open those links to see exactly what pages it used.

2) If you mean “what pages is an AI bot crawling on my website?”

Check your server logs, analytics, or bot traffic tools.

Ways to inspect

• Server access logs
  Look for user agents like:

  • GPTBot
  • ChatGPT-User
  • ClaudeBot
  • anthropic-ai
  • PerplexityBot
  • Google-Extended
  • Bytespider
  • CCBot

• Cloudflare / CDN logs

  • Often easier to filter by bot user agent and URL.

• Google Analytics / Matomo / Plausible

  • These may show bot-like traffic, though not always reliably.

What to look for

• Requested URL
• Timestamp
• User agent
• IP address
• Status code
• Referrer

That tells you which pages were fetched by which bot.

3) If you mean “what pages are being used to answer a specific AI response?”

This is the hardest one, because many LLMs do not expose their exact internal retrieval path unless they’re using explicit search or citations.

Best approaches

• Use an AI tool with citations enabled
• Ask the system to show sources
• If it’s your own RAG system, check:
  • retrieval logs
  • vector search results
  • reranker logs
  • prompt assembly logs

For your own site: practical steps

A. Check robots.txt

See whether AI crawlers are allowed:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

B. Search logs for AI crawlers

Example Linux command:

grep -i "GPTBot\|ClaudeBot\|PerplexityBot\|Google-Extended" access.log

C. Identify top pages accessed

If logs are large, group by URL:

grep "GPTBot" access.log | awk '{print $7}' | sort | uniq -c | sort -nr

Important note

If you want to know exactly which web pages an AI model trained on, that usually can’t be determined precisely from the model itself, because training data is aggregated and not fully traceable. But if the AI is using live web retrieval, citations or logs usually make it visible.

If you want, I can help you with one of these specifically:

• checking AI crawler logs on your website
• finding citations in ChatGPT/Perplexity/Bing
• setting up a way to track AI bot visits