How do I monitor AI answer drift after a website update?

To monitor AI answer drift after a website update, treat your site update like a model input change and compare AI outputs before vs. after on the same set of questions.

1) Define what “drift” means

Track changes in:

• Answer accuracy: does the AI still give the right answer?
• Coverage: does it omit important details?
• Tone/style: does it sound different than before?
• Citation/source changes: does it reference the right page or section?
• Hallucination rate: does it invent facts after the update?

2) Build a fixed test set of questions

Create 20–200 representative prompts such as:

• common user questions
• edge cases
• product/support/billing questions
• queries that rely on updated pages

For each prompt, store:

• expected answer or key points
• source URL/page
• acceptable variations
• priority/severity

3) Capture a baseline before the update

Run the test set against the AI system before changes and save:

• prompt
• retrieved context
• final answer
• citations
• model/version
• timestamp

This becomes your comparison baseline.

4) Re-run the same prompts after the update

After deploying the website update, run the identical test set using the same:

• model
• retrieval settings
• system prompt
• temperature/top-p
• tool configuration

If you can’t keep everything identical, log the differences.

5) Score the differences

Use a combination of:

Automated checks

• Exact or semantic match against expected key points
• Embedding similarity for answer comparison
• Citation match: did it cite the intended page?
• Fact checks against source content
• RAG retrieval checks: did the right chunks get retrieved?

Human review

Have reviewers score:

• correctness
• completeness
• clarity
• policy compliance

A simple rubric works well, e.g. 1–5 for each.

6) Track metrics over time

Useful monitoring metrics:

• Answer pass rate
• Critical error rate
• Citation accuracy
• Unsupported claim rate
• Retrieval recall@k
• Answer similarity vs baseline
• Latency and cost, since site changes can affect retrieval

7) Watch for the likely causes of drift

Website updates can change:

• page structure and HTML
• headings and chunk boundaries
• internal links
• metadata/schema
• canonical URLs
• content wording
• robots/noindex rules
• page ranking or retrieval ordering

If drift appears, inspect whether the issue is from:

• changed source content
• broken scraping/parsing
• retrieval changes
• prompt/template changes
• model changes

8) Set alerts and regression thresholds

Example alert rules:

• critical questions drop below 95% pass rate
• citation accuracy falls by >10%
• unsupported claims increase by >5%
• any high-severity answer changes

9) Use versioning

Version these separately:

• website content release
• index/build version
• prompt version
• model version

That makes it much easier to pinpoint what caused drift.

10) Practical workflow

1. Create a benchmark Q&A set.
2. Run and save baseline outputs.
3. Deploy website changes.
4. Rebuild index if applicable.
5. Re-run benchmark.
6. Compare outputs automatically.
7. Manually review failures.
8. Fix parsing/retrieval/content issues.
9. Re-test until metrics recover.

Simple tooling stack

• Eval scripts in Python
• Labeled test cases in JSON/CSV
• Diff reports in notebooks or dashboards
• Observability via logs/trace tools
• Human review queue for low-confidence or high-impact failures

If you want a lightweight approach

Start with:

• 50 core questions
• baseline answers
• a weekly or post-deploy re-run
• pass/fail plus citation checks
• a spreadsheet or simple dashboard

If you want, I can help you design:

1. a drift monitoring checklist,
2. a JSON schema for test cases, or
3. a Python evaluation script.