How do I build a topic-level AI visibility tracker?

To build a topic-level AI visibility tracker, think of it as a system that answers:

“For a given topic, how often and how prominently do AI systems mention my brand, competitors, or key entities?”

At a high level, you need to:

1. Define topics
2. Generate and run queries across AI systems
3. Capture and normalize responses
4. Extract mentions and sentiment/stance
5. Score visibility over time
6. Compare against competitors
7. Store, trend, and alert on changes

1) Define what “topic-level” means

A topic is usually a cluster of related intents, not just one keyword.

Examples:

• “best CRM for small business”
• “how to reduce cloud costs”
• “enterprise data observability”
• “email deliverability software”

For each topic, create:

• Primary query
• Query variants
• Subtopics
• Competitor set
• Brand/entity list you care about

A good topic spec might include:

• topic_id
• topic_name
• seed_queries
• audience
• intent_type (informational, commercial, comparison, transactional)
• brands_to_track
• competitors
• geo/language
• model_targets (ChatGPT, Gemini, Claude, Perplexity, etc.)

---

2) Decide what AI systems you’ll track

Different products answer differently. Common options:

• ChatGPT
• Claude
• Gemini
• Perplexity
• Google AI Overviews (harder to systematically capture)
• Bing/Copilot
• Specialty answer engines or vertical assistants

Important: each system may require different collection methods:

• official API
• browser automation
• manual test harness
• third-party SERP/AI providers

If you can, prefer official APIs for reliability and compliance.

---

3) Build a prompt/query generation layer

You need a consistent way to ask the model about the topic.

For each topic, generate a query set like:

• “What are the best tools for [topic]?”
• “Which companies help with [topic]?”
• “Compare [your brand] vs competitors for [topic].”
• “What should I use for [topic] if I’m a [persona]?”

Tips:

• Keep prompts deterministic and versioned.
• Control for:
  • geography
  • persona
  • budget
  • use case
  • time context
• Run each prompt multiple times if the model is stochastic.

Store:

• prompt text
• prompt version
• model
• date/time
• locale
• temperature / sampling settings
• run ID

---

4) Collect responses and normalize them

For each run, save the raw response plus metadata.

Recommended fields:

• run_id
• topic_id
• query_id
• model_name
• model_version
• timestamp
• raw_response
• token_count
• latency
• source_links if any

Then normalize into a structured format:

• detected entities
• cited brands
• rankings/order
• recommendation status
• sentiment/stance
• whether your brand is mentioned
• whether your brand is recommended
• whether competitors are mentioned

---

5) Extract visibility signals

Topic-level AI visibility should not be just “mentioned or not.”

Useful signals:

Mention-based

• Brand mentioned in answer
• Brand mentioned in top N lines
• Brand mentioned in cited sources
• Brand appears in comparison table

Recommendation-based

• Brand recommended
• Brand ranked first
• Brand included in shortlist
• Brand excluded from list

Context-based

• Positive/neutral/negative stance
• Associated attributes
• Competitor dominance
• Citation presence
• Whether the answer reflects your preferred positioning

A simple extraction pipeline can use:

• regex/entity matching for exact brand names
• LLM-based classification for context
• NER/entity linking for aliases and misspellings

---

6) Create a visibility score

You’ll want a score that is easy to trend over time.

Example components:

• Mention rate: % of queries where brand appears
• Recommendation rate: % where brand is recommended
• Top placement rate: % where brand is first or in top 3
• Citation rate: % where a source to your domain is cited
• Sentiment score: average stance
• Share of voice: your mentions vs competitors

Example simple weighted score:

Visibility Score =
0.35 * mention_rate +
0.35 * recommendation_rate +
0.20 * top_placement_rate +
0.10 * citation_rate

You can also compute:

• Topic visibility by topic
• Model visibility by model
• Geo visibility by region
• Competitive visibility gap

---

7) Store it in a data model that supports analytics

A practical schema:

Tables

topics

• topic metadata

queries

• query text, version, intent, locale

runs

• model, timestamp, parameters, raw output reference

entity_mentions

• run_id, entity, mention_type, position, sentiment

topic_scores

• topic_id, model, date, visibility_score, mention_rate, etc.

competitor_sets

• topic_id, competitor brands

This makes it easy to:

• trend over time
• segment by model
• compare topics
• build dashboards

---

8) Add trend and alerting logic

Visibility changes matter more than static snapshots.

Useful alerts:

• brand disappears from top responses
• competitor overtakes you on a topic
• citations shift away from your site
• sentiment turns negative
• model updates materially change answers

Example triggers:

• visibility score drops > 20% week-over-week
• mention rate falls below threshold
• competitor share of voice increases > X%
• your domain cited less than before

---

9) Build a dashboard

Good dashboard views:

• topic overview
• model-by-model comparison
• competitor comparison
• trend lines over time
• prompt variants
• raw response drill-down
• citation/source analysis

Good charts:

• line chart for visibility score
• stacked bars for share of voice
• heatmap by model × topic
• table of top cited sources
• distribution of recommendation positions

---

10) Handle variability and bias

AI answers vary, so make the measurement robust:

• run each prompt multiple times
• keep prompts fixed and versioned
• test across multiple models
• use consistent regions/languages
• sample over time, not just once
• detect hallucinated or outdated answers
• separate “visibility” from “truthfulness”

---

11) Suggested architecture

A simple architecture:

Scheduler

• cron/Airflow/Temporal

Query runner

• calls model APIs or browser automation

Parser/extractor

• entity detection, classification, scoring

Database

• Postgres for structured data
• object storage for raw responses

Analytics

• dbt + SQL + Python notebooks

Dashboard

• Metabase / Looker / Superset / custom app

Alerts

• Slack/email/webhooks

---

12) Minimal MVP

If you want to launch quickly, start with:

1. 10–20 topics
2. 5–10 prompts per topic
3. 2–3 models
4. Daily runs
5. Exact brand matching + basic LLM classification
6. One simple visibility score
7. Dashboard with trends and raw answers

That’s enough to get actionable insights.

---

13) Example visibility workflow

For the topic: “best email deliverability software”

You ask:

• “What are the best email deliverability tools for SMBs?”
• “Compare top email deliverability platforms.”
• “Which tools improve inbox placement?”

Then:

• capture responses from multiple models
• extract mentions of your brand and competitors
• score if your brand is listed, recommended, and positively described
• compare this week vs last week
• alert if you drop out of top 3

---

14) Common pitfalls

• Tracking only one prompt per topic
• Ignoring prompt wording effects
• Not versioning prompts
• Mixing up brand mentions with recommendations
• Treating all models as equivalent
• Not saving raw outputs
• Overfitting your score to one model
• Measuring too infrequently

---

If you want, I can also give you:

1. a data schema
2. a Python prototype
3. a dashboard design
4. or a scoring formula for topic-level AI visibility.