Prompt
We're getting mixed signals from ChatGPT and Perplexity, and I don't know if it's our content, the model, or just sampling noise. I want a repeatable way to track the exact prompts we care about and see whether our visibility is improving or slipping. What's the cleanest way to do this?
Latest observation
The cleanest way is to treat this like search rank tracking for AI answers: build a small, repeatable benchmark of prompts, run them on a fixed schedule, and score the outputs against the facts and links you care about.
A practical setup
1) Define a prompt set you actually care about
Create a benchmark of 20–100 prompts split into buckets:
- Brand prompts: “What does [your company] do?”
- Category prompts: “Best tools for X”
- Problem prompts: “How do I solve Y?”
- Competitor prompts: “Compare [you] vs [competitor]”
- High-intent prompts: “Best [solution] for [use case]”
For each prompt, record:
- exact prompt text
- target audience / intent
- desired claims
- URLs you want cited
- competitor set
- “bad outcomes” to flag
Keep this in a sheet or database so the prompt list is versioned and stable.
2) Run each prompt the same way every time
Consistency matters more than raw volume.
For each model/product you want to track:
- use the same prompt wording
- same locale and language
- same temperature if possible
- same session state rules:
- ideally fresh chat every run
- no memory/personalization
- same time window if you want to reduce drift
- same retrieval mode if the product offers it
If the tool doesn’t expose deterministic settings, just document the exact conditions so you can compare apples to apples.
3) Capture the full answer, not just the top line
Store:
- the raw response text
- citations/links
- titles/snippets of cited sources
- date/time
- model name/version
- prompt ID
- any settings used
This lets you audit later when something changes.
4) Score outputs with a small rubric
Use a simple rubric per prompt:
Visibility
- 0 = not mentioned
- 1 = mentioned but weakly
- 2 = clearly included
- 3 = prominently recommended
Accuracy
- 0 = incorrect
- 1 = partially correct
- 2 = fully correct
Citation quality
- 0 = no citation
- 1 = irrelevant/low-quality citation
- 2 = relevant citation
- 3 = your preferred source cited
Share of voice
- Did you appear among the top recommendations?
- Were competitors favored?
- Did your domain get cited?
This can be semi-automated at first and manually reviewed for a smaller sample.
5) Measure trends, not single runs
LLMs vary. One answer is noise; a pattern over time is signal.
Track:
- mention rate by prompt bucket
- citation rate
- rank/order in recommendation lists
- share of voice vs competitors
- sentiment/positioning
- source mix (your site vs third-party sources)
Use moving averages or weekly snapshots instead of reacting to one bad run.
6) Separate content issues from model noise
When visibility drops, check these in order:
- Prompt drift: did the prompt wording change?
- Model/version drift: did the underlying model change?
- Source drift: did the cited web sources change?
- Content drift: did your pages change or become less crawlable?
- Sampling noise: does the result flip across repeated runs?
A good diagnostic is to rerun the same prompt 3–5 times under the same conditions. If results vary a lot, you’re seeing stochasticity. If they’re consistently bad, it’s likely a content/sourcing issue.
The easiest implementation path
Lightweight version
- Google Sheets/Airtable for prompt inventory
- manual weekly runs
- paste responses and score them
Good for getting started fast.
Better version
- a small script or workflow to:
- read prompts from a table
- query the model/API
- save responses and citations
- compute scores
- visualize trends in Looker Studio / Metabase / Grafana
This gives you repeatability and less manual work.
What makes this “clean”
The key is to standardize four things:
- prompt
- environment
- scoring rubric
- schedule
That turns “mixed signals” into a tracked dataset.
Recommended output format
For each run, log:
prompt_idprompt_textmodeldateresponsecitationsvisibility_scoreaccuracy_scorecitation_scorenotes
If you want the shortest answer
Build a prompt benchmark + weekly rerun + scoring sheet.
That’s the simplest repeatable way to know whether your visibility is improving or slipping.
If you want, I can help you design:
- a benchmark template,
- a scoring rubric, or
- a workflow/API schema to automate it.