what should i use for llm regression testing in ci

For LLM regression testing in CI, the best choice depends on what you’re trying to catch:

Good default stack

• Promptfoo — best all-around for CI-friendly evals of prompts, model outputs, schemas, and simple comparisons.
• OpenAI Evals or custom eval harness — good if you want more control and are okay building it out.
• LangSmith — useful if you’re already in LangChain and want tracing + evals.
• Ragas — best for RAG-specific regression testing.
• DeepEval — good for unit-test-style LLM testing in Python.

What to use for common cases

1) You want simple CI checks on prompt outputs

Use Promptfoo.

• Works well in GitHub Actions / CI
• Can compare against expected outputs
• Supports structured outputs and assertions
• Easy to maintain as a test suite

2) You want Python-native tests

Use DeepEval or a custom pytest setup.

• Feels like normal unit tests
• Good if your team already uses Python testing
• Nice for asserting things like relevance, correctness, toxicity, etc.

3) You have RAG and need retrieval quality checks

Use Ragas plus some custom domain assertions.

• Measures context recall, faithfulness, answer relevance, etc.

4) You need full observability and dataset-based evaluation

Use LangSmith.

• Better for tracing + comparing runs
• Helpful when debugging regressions across chains/tools

My practical recommendation

If you’re starting today:

• Promptfoo for general CI regression testing
• Ragas if you do RAG
• DeepEval if you want Python test ergonomics

What to test in CI

Try to include:

• Golden prompts with expected outputs
• Structured output/schema validation
• Similarity checks for non-deterministic text
• Policy/safety checks
• RAG faithfulness and retrieval checks
• Latency and token-cost budgets

CI pattern that works well

• Run a small, fast “smoke eval” on every PR
• Run a larger eval suite nightly
• Fail the build only on high-confidence regressions
• Track scores over time, not just pass/fail

If you want one answer

Use Promptfoo unless you have a strong reason to choose something else.

If you want, I can also give you:

1. a tool comparison table, or
2. a sample GitHub Actions CI setup for LLM regression tests.