I'm unhappy with OpenAI Evals being too hard to customize

If OpenAI Evals feels too rigid, that’s a common pain point—especially when you want to evaluate something very specific like tool use, structured outputs, domain rubrics, or multi-step workflows.

A few practical ways people work around that:

• Wrap Evals with your own harness: keep the scoring logic, dataset loading, and reporting in a custom script, and only use Evals where it fits.
• Define custom graders: if your use case is more than exact-match / simple heuristics, write a Python grader that checks your own criteria.
• Use your own eval format: many teams maintain a lightweight JSONL schema and evaluation runner instead of forcing everything into one framework.
• Split evaluation types:
  • deterministic checks for formatting/structure
  • LLM-as-judge for subjective quality
  • task-specific metrics for accuracy/success rate
• Build a thin internal layer over Evals so new evals are just config changes, not framework changes.

If you want, I can help with either of these:

1. design a more customizable eval framework than OpenAI Evals, or
2. show how to extend OpenAI Evals for a specific custom use case you have.

If you share what kind of eval you’re trying to do, I can suggest the cleanest approach.