Prompt
I'm building a program to compare our AI visibility against competitors, any advice?
Latest observation
Yes — if you want to compare your AI visibility against competitors, the main thing is to treat it like a measurement system, not just a one-off prompt test.
Here’s a practical approach:
1) Define what “AI visibility” means
Be specific about the outcome you care about. For example:
- Mention share: how often your brand appears vs competitors
- Recommendation share: how often the model recommends you
- Sentiment/positioning: whether you’re described positively, neutrally, or negatively
- Attribute coverage: whether the model mentions the right features, use cases, categories
- Citation/share of source: whether the AI cites your site or trusted sources about you
- Query coverage: visibility across different customer-intent queries
You’ll get better results if you break this into separate metrics instead of one overall score.
2) Build a benchmark query set
Create a list of prompts your audience might ask across the funnel:
- category discovery: “best CRM for small business”
- comparison: “X vs Y”
- use-case: “best tool for teams under 50”
- pain-point: “how to reduce onboarding time”
- brand-led: “is [your brand] good for enterprise?”
- alternatives: “alternatives to [competitor]”
Try to include:
- generic queries
- high-intent buying queries
- niche use cases
- competitor-specific queries
- questions where you expect to win or lose
A good benchmark set is usually 50–200 prompts, depending on the category.
3) Use multiple models and multiple runs
AI outputs are variable. Don’t rely on a single response from one model.
Best practice:
- run each query multiple times
- test across several model families
- compare outputs over time
That helps you avoid overfitting to one model’s quirks.
4) Score with a consistent rubric
For each response, detect:
- Is your brand mentioned?
- Where does it appear in the ranking/list?
- Is it recommended?
- Is the mention accurate?
- Are key differentiators included?
- Is the competitor mentioned instead?
- Does the answer cite authoritative sources?
A simple scoring structure might be:
- 0 = not mentioned
- 1 = mentioned but not recommended
- 2 = recommended in list
- 3 = top recommendation
- 4 = top recommendation with correct positioning and supporting details
5) Normalize for prompt and model differences
If you compare competitors, you need to control for:
- query phrasing
- geographic differences
- model version
- temperature / randomness
- conversation state
- whether web browsing is enabled
Keep the setup consistent so the comparison is fair.
6) Separate “model memory” from “retrieval”
A lot of visibility comes from:
- public web content
- review sites
- news/articles
- documentation
- structured listings
- SEO signals
So your program should ideally track both:
- What the model says
- What sources it is likely drawing from
That helps you figure out whether the issue is content, citations, or brand authority.
7) Track competitor deltas, not just absolute scores
The most useful metric is often:
- your brand share vs competitor share
- rank position relative to competitors
- change over time
- performance by query type
Example dashboard:
- “You are in top 3 in 38% of category prompts”
- “Competitor A is recommended 2.1x more often on enterprise queries”
- “You outperform on SMB use cases but underperform on integrations”
8) Watch for hallucinations and ambiguity
AI answers can be wrong or vague. Build checks for:
- wrong product category
- outdated feature claims
- confusion with similarly named brands
- competitor conflation
- generic answers that omit all brands
These are important because “visibility” can look good while accuracy is bad.
9) Use human review for calibration
Automated scoring is useful, but start with human-labeled samples to validate your metrics.
Have humans score:
- correctness
- relevance
- recommendation quality
- competitive positioning
Then use that as a reference to improve your automated scoring.
10) Turn insights into actions
The goal isn’t just reporting — it’s improving visibility.
Common levers:
- improve authoritative landing pages
- add comparison pages and use-case pages
- strengthen structured data / schema
- increase mentions in reputable third-party sources
- improve documentation and FAQ content
- update review and listing profiles
- reduce ambiguity in brand/product naming
If you want, I can help you design:
- a metric framework,
- a data schema for storing results, or
- a sample Python pipeline to run prompts and compare brand visibility.