track AI answer citations by topic

To track AI answer citations by topic, use a simple pipeline:

1. Define topics

   • Create a fixed topic taxonomy (e.g., Billing, Security, Product, Legal).
   • Allow one answer to map to multiple topics if needed.

2. Capture each AI response Store:

   • answer_id
   • conversation_id
   • timestamp
   • answer_text
   • citations (URL, document ID, source title, snippet, etc.)

3. Tag citations with topics

   • Manual tagging: reviewers assign a topic to each citation.
   • Rule-based tagging: map source domains/files to topics.
   • ML/LLM tagging: classify citation context into topics automatically.

4. Link citations to topics Example structure:

   {
     "answer_id": "a123",
     "topics": ["Security", "Compliance"],
     "citations": [
       {
         "citation_id": "c1",
         "source": "policy.pdf",
         "topic": "Compliance"
       },
       {
         "citation_id": "c2",
         "source": "kb-ssl-setup",
         "topic": "Security"
       }
     ]
   }
   

5. Track metrics by topic Useful metrics:

   • citation count per topic
   • answers with no citations per topic
   • top sources cited per topic
   • citation freshness by topic
   • citation accuracy / approval rate by topic

6. Build a dashboard Show:

   • topic distribution of citations
   • trend over time
   • source coverage gaps
   • citation quality issues by topic

7. Implementation tip If you want this in SQL, create tables like:

   • answers
   • citations
   • topics
   • citation_topics (many-to-many)

If you want, I can also provide:

• a SQL schema
• a Python example
• or a dashboard design for this.