What should I use to detect when AI outputs change?

If you want to detect when AI outputs change, the best choice depends on what kind of change you care about:

1. For exact output changes

Use a diff/compare tool with the model’s responses stored over time.

• Good when you want to know if the wording, structure, or content changed.
• Works best if you save the full prompt + full response.
• Examples:
  • text diff libraries
  • Git-style diffs
  • JSON diff if outputs are structured

2. For semantic changes

Use embeddings + similarity comparison.

• Good when the output is rephrased but the meaning may be the same.
• Compare embedding vectors of old vs. new outputs.
• Useful for “did the answer meaningfully change?” rather than “did the text change?”

3. For regression testing

Use an LLM eval harness or golden test suite.

• Store expected outputs or expected properties.
• Run prompts regularly and compare against baseline.
• Good tools:
  • OpenAI Evals
  • LangSmith
  • promptfoo
  • custom CI tests

4. For production monitoring

Use logging + drift detection.

• Log prompts, model version, parameters, and outputs.
• Alert when:
  • output format changes
  • refusal rates change
  • similarity drops
  • specific fields disappear

Practical recommendation

If you’re just getting started:

• Structured output? Use JSON schema validation + diff
• Free-form text? Use embeddings similarity + human review
• Need automated monitoring? Use promptfoo or LangSmith plus stored baselines

Simple rule of thumb

• Text changed exactly → diff
• Meaning changed → embeddings
• Behavior changed over time → evals / regression tests

If you want, I can recommend a specific setup for:

• a chatbot,
• an API app,
• or monitoring model updates in production.