Prompt
I'm trying to see whether schema changes or new pages changed our citations in AI answers. Is there a good way to measure before and after?
Latest observation
Yes — a solid way to measure this is to treat it like an A/B comparison of AI citation behavior before vs. after your schema or content changes.
What to measure
Track these metrics for a fixed set of queries/pages:
-
Citation presence
- Did the AI answer cite your site at all?
-
Citation share
- Of all citations in the answer, what percentage were yours?
-
Citation rank / position
- Were you cited first, near the top, or only occasionally?
-
Mention rate
- Did your brand/page get mentioned even without a citation?
-
Query coverage
- For how many target queries did you appear?
-
Answer quality context
- Was the citation used as a primary source, supporting source, or peripheral source?
Best measurement setup
1) Build a fixed query set
Use a stable list of prompts relevant to your business, for example:
- informational queries
- comparison queries
- “best X for Y” queries
- product/service queries
- branded queries
Keep this set identical before and after the change.
2) Capture a baseline
Before making changes, run the same prompts against the AI system(s) you care about and record:
- full answer
- all citations/links/sources
- date/time
- model/version if known
3) Re-run after changes
After schema or page updates, rerun the same prompts under similar conditions:
- same prompt wording
- same region/language if possible
- same model/version if possible
- ideally multiple runs, since AI answers can vary
4) Compare at the prompt level
For each prompt, compare:
- whether your domain appeared
- whether specific page URLs appeared
- whether citation order changed
- whether the answer wording shifted toward your content
Recommended output format
A simple spreadsheet works well. Columns like:
- Query
- Baseline cited? (Y/N)
- Baseline citation URLs
- Post-change cited? (Y/N)
- Post-change citation URLs
- Change in citation count
- Change in rank/position
- Notes
Make it more reliable
Because AI answers can be non-deterministic:
- run each query multiple times
- average results across runs
- compare using the same prompt set and timeframe
- segment by query type
If you want a more formal score
You can create a simple citation lift score, like:
- 1 point = your domain cited
- 2 points = cited in top 3
- 3 points = cited as primary source
Then compare the average score before vs. after.
Important caveats
- AI citation behavior is noisy and can change for reasons unrelated to your site.
- Schema changes may help discovery/understanding, but the effect might be indirect and gradual.
- New pages may change which page is cited, not just whether you’re cited.
- Different AI tools cite differently, so measure each separately.
Best practice
If you want something dependable, measure across:
- multiple prompts
- multiple runs
- multiple AI systems
- before/after time windows
If you want, I can help you design a simple measurement template or a more rigorous experiment plan for ChatGPT, Perplexity, Google AI Overviews, or other systems.