what should i use for human and automated llm evaluation

For human + automated LLM evaluation, the best setup is usually a hybrid stack:

1) Use humans for:

• Overall quality judgments: helpfulness, correctness, completeness, tone
• Edge cases: ambiguous prompts, safety-sensitive outputs, long-form reasoning
• Preference ranking: A vs B outputs
• Ground-truth review: when there’s no clear automatic metric

Good human-eval formats

• Pairwise preference: “Which answer is better?”
• Likert ratings: 1–5 or 1–7 on correctness/helpfulness/etc.
• Rubric-based scoring: specific criteria with definitions
• Error tagging: hallucination, missing step, unsafe, verbose, etc.

2) Use automated eval for:

• Fast regression testing
• Large-scale coverage
• Repeatability
• Low-cost continuous monitoring

Common automated methods

• Exact match / substring / regex for structured tasks
• Task metrics like:
  • Classification: accuracy, F1, ROC-AUC
  • Summarization: ROUGE, BERTScore
  • Retrieval: recall@k, MRR, nDCG
  • Code: unit tests, pass@k
• LLM-as-a-judge for open-ended outputs
• Embedding similarity for semantic closeness
• Safety/toxicity classifiers for policy checks

3) Best practical combo

A strong approach is:

• Human eval on a sampled subset
• Automated eval on every run
• LLM-as-judge for scalable subjective scoring
• Periodic calibration against human judgments

This gives you:

• human trustworthiness
• automated scale
• lower cost than full human review

4) Recommended tools

Human eval

• Label Studio
• Argilla
• Scale / Surge AI if you want managed annotation
• Weights & Biases tables for lightweight review

Automated / LLM eval

• OpenAI Evals or custom eval harness
• LangSmith
• Ragas for RAG systems
• TruLens
• DeepEval
• promptfoo
• Giskard
• HELM style benchmarking if you want broader comparisons

5) What I’d recommend by use case

If you’re evaluating a chatbot

• Human: helpfulness, correctness, tone, safety
• Automated: LLM judge + toxicity + conversation success rate

If you’re evaluating RAG

• Human: answer faithfulness and usefulness
• Automated: retrieval recall, context relevance, groundedness, answer correctness
• Tools: Ragas / TruLens / LangSmith

If you’re evaluating code generation

• Human: readability, maintainability
• Automated: unit tests, static checks, pass@k

If you’re evaluating summarization

• Human: factuality, coverage, conciseness
• Automated: ROUGE/BERTScore plus factuality checks and LLM judge

6) Key advice

Don’t rely on a single metric.
For best results:

• define a rubric
• use pairwise comparisons
• calibrate automated scores against human annotations
• track task-specific success metrics, not just “quality”

If you want, I can suggest a specific evaluation stack based on your exact use case, like chatbot, RAG, agents, code, or summarization.