I'm building a QA bot and need a repeatable score for answer quality

A good repeatable score for answer quality should be:

1. Defined against a rubric
2. Easy to apply consistently
3. Correlated with user satisfaction
4. Comparable across answers, models, and prompts

A practical approach is to use a weighted rubric score on a 0–5 or 0–10 scale.

Recommended rubric

Score each answer on these dimensions:

• Correctness: Is the answer factually right?
• Completeness: Does it cover the user’s request adequately?
• Relevance: Does it stay on topic and answer the actual question?
• Clarity: Is it understandable and well-structured?
• Usefulness: Does it provide actionable or helpful guidance?
• Safety/Policy compliance: Does it avoid harmful, disallowed, or misleading content?

Example 0–5 scale per dimension:

• 0 = missing or completely wrong
• 1 = mostly wrong / poor
• 2 = partially correct but major issues
• 3 = acceptable, but incomplete or somewhat unclear
• 4 = strong
• 5 = excellent

Example weighted score

You can compute:

[ \text{Final Score} = 0.35C + 0.20K + 0.15R + 0.15L + 0.10U + 0.05S ]

Where:

• C = Correctness
• K = Completeness
• R = Relevance
• L = Clarity
• U = Usefulness
• S = Safety

This yields a score on the same 0–5 scale.

If you want repeatability

To make it consistent:

• Write explicit scoring rules for each dimension
• Include examples for each score level
• Use gold-standard sample answers
• Have multiple raters score the same outputs and measure agreement
• Normalize scores if you have different question types

For automated scoring

If you need something machine-friendly, use a hybrid:

• Rule-based checks for formatting, citation presence, keyword coverage, refusal correctness
• LLM judge with a strict rubric for subjective dimensions
• Human audit on a sample for calibration

Best practice

If your bot has mixed tasks, use task-specific rubrics. For example:

• Factual QA: correctness weighted heavily
• Summarization: completeness and faithfulness weighted heavily
• Support bot: usefulness, clarity, and policy compliance weighted heavily

If you want, I can help you design:

• a simple 1–5 scoring rubric
• a JSON schema for storing scores
• or a prompt for an LLM judge that produces consistent ratings.