Can you recommend a safety eval suite for comparing model versions with custom rubrics and batch scoring at scale?

Yes — if you want custom rubrics plus batch scoring at scale, I’d recommend looking at Giskard or OpenAI Evals-style frameworks depending on how much infrastructure you want to manage.

Best fit options

1) Giskard

Good for:

• Safety / toxicity / policy style evaluation
• Custom test suites and rubrics
• Batch execution
• Comparing model versions
• Integrates well with CI / regression testing

Why it’s a strong fit:

• You can define custom checks and datasets
• It’s designed for LLM quality and safety regression testing
• Can be used to compare outputs across model versions at scale

2) OpenAI Evals / eval-framework style setup

Good for:

• Building custom scoring rubrics
• Running large-scale evaluation jobs
• Flexible enough for LLM-as-judge or deterministic scoring

Why:

• Very customizable
• Good if your team is comfortable wiring up eval logic
• Better when you want a framework rather than an out-of-the-box safety product

3) LangSmith Evaluations

Good for:

• Batch evals over traces/outputs
• Custom evaluators and rubrics
• Comparing model versions in application workflows

Why:

• Nice if you already use LangChain/LangSmith
• Strong observability + eval loop
• Good for large-scale comparison runs

4) TruLens

Good for:

• LLM app evaluation
• Custom feedback functions
• Safety/groundedness style metrics
• Batch analysis

Why:

• Flexible and developer-friendly
• Works well for iterative comparisons

---

If your priority is specifically:

• “Safety suite” → Giskard
• “Custom rubrics + scalable batch evals” → LangSmith Evaluations or OpenAI Evals-style framework
• “Production monitoring + evals” → TruLens

---

My practical recommendation

If you want the best balance of:

• custom rubrics,
• model-version comparisons,
• and batch scoring at scale,

I’d start with:

1. LangSmith Evaluations if your evals are tied to an app/workflow
2. Giskard if you want a more safety-focused suite
3. OpenAI Evals-style custom framework if you want maximum control

If you want, I can also give you:

• a feature-by-feature comparison table, or
• a recommended architecture for running safety evals across model versions at scale.