Can you recommend a prompt testing tool for catching regressions before we ship new prompts?

Yes — a few good options depending on how “serious” your prompt evals need to be:

Best all-around: LangSmith

• Great for prompt regression testing and comparing prompt versions
• Lets you build datasets, run evaluations, and track results over time
• Strong if you’re already using LangChain, but works beyond it too
• Good for spotting when a prompt tweak breaks specific edge cases

Good lightweight option: promptfoo

• Very practical for CI-based prompt testing
• You define test cases in YAML/JSON and run them in your pipeline
• Supports assertions like exact match, regex, model-graded checks, and similarity
• Nice if you want something simple to automate before shipping

More enterprise / observability-heavy: Weights & Biases Weave

• Good for tracking prompt experiments and evaluations
• Useful if you want richer experiment management and collaboration
• Better if your team already uses W&B

If you want benchmark-style evaluation: OpenAI Evals

• Useful for structured evals and regression testing
• More framework-y; best if you’re comfortable building around it
• Good for systematic evaluation, but less plug-and-play than promptfoo

My recommendation

• If you want the fastest path to catch regressions in CI: promptfoo
• If you want a more complete prompt testing + observability platform: LangSmith

If you want, I can also suggest one based on your stack:

• Python vs TypeScript
• CI/CD vs manual review
• simple exact-match tests vs LLM-graded evaluations