How do I score extraction outputs from an LLM against ground truth?

A good way to score extraction outputs depends on what kind of extraction you’re doing, but the general idea is:

1. Normalize the model output and ground truth
2. Match predicted items to gold items
3. Compute metrics like precision, recall, and F1
4. Optionally add field-level and document-level scores

1) Decide the unit of evaluation

Common extraction tasks:

• Classification / label extraction
  Example: one label per document.
• Span extraction
  Example: extract names, dates, addresses from text.
• Structured extraction
  Example: JSON with fields like {name, date, amount}.
• Set extraction
  Example: list of entities, keywords, or facts.

The scoring method changes a bit depending on the task.

---

2) Normalize outputs first

Before comparing prediction to ground truth, normalize both:

• lowercase
• trim whitespace
• normalize punctuation
• canonicalize dates (2024-01-05 vs Jan 5, 2024)
• normalize numbers ($1,000 vs 1000)
• map synonyms if appropriate
• standardize missing values (null, None, "")

This often matters more than the metric itself.

---

3) Use exact-match metrics for structured fields

For JSON-like outputs, compare each field:

Field-level exact match

For each field:

• 1 if predicted value matches gold value exactly after normalization
• 0 otherwise

Then average over fields/documents.

Example

Gold:

{"name": "Alice Smith", "date": "2024-01-05", "amount": 1000}

Pred:

{"name": "Alice Smith", "date": "2024-01-05", "amount": 100}

Field scores:

• name = 1
• date = 1
• amount = 0

Field accuracy = 2/3

---

4) Use precision / recall / F1 for set-like extraction

If the output is a list of items:

• Precision = correct predicted items / total predicted items
• Recall = correct predicted items / total gold items
• F1 = harmonic mean of precision and recall

Example

Gold entities: {A, B, C} Predicted entities: {A, C, D}

• TP = 2 (A, C)
• FP = 1 (D)
• FN = 1 (B)

So:

• Precision = 2/3
• Recall = 2/3
• F1 = 2/3

---

5) Handle fuzzy matching when exact match is too strict

For many extraction tasks, exact match is too harsh. You can use:

• Token-level overlap
• Edit distance / similarity
• Span overlap for text spans
• Field-specific tolerances
  • dates within 1 day
  • amounts within rounding tolerance
  • names with alias matching

Examples:

• "IBM" vs "International Business Machines" might be a match if aliases are allowed
• "2024-01-05" vs "01/05/2024" should normalize to the same date
• "1000.00" vs "1000" can be equal after numeric normalization

---

6) Match predicted items to gold items carefully

If multiple items are extracted, you need a matching rule.

Common choices:

• Exact set matching if items are unique
• Hungarian matching for best one-to-one assignment
• Greedy matching by similarity score
• IOU/overlap threshold for spans

This prevents double-counting one predicted item against multiple gold items.

---

7) Use document-level scores too

Sometimes a document is only correct if all required fields are correct.

Document accuracy

• 1 if the entire output matches ground truth
• 0 otherwise

This is very strict but useful for production quality checks.

---

8) Common metrics by task

Named entity / span extraction

• Precision / Recall / F1
• Exact span match
• Partial span overlap

Structured JSON extraction

• Field accuracy
• Slot F1
• Document exact match
• Value-level F1

Information extraction with confidence scores

• Precision/Recall/F1 at different thresholds
• ROC-AUC / PR-AUC if it’s framed as ranking/classification

---

9) Practical evaluation recipe

A solid default approach:

1. Parse model output into a schema
2. Normalize values
3. Compare each predicted item to gold
4. Use exact match where possible
5. Use fuzzy matching for dates/numbers/aliases/spans
6. Report:
   • field-level precision/recall/F1
   • document exact match
   • per-field breakdown
   • error examples

---

10) Simple pseudocode

def normalize(x):
    if x is None:
        return None
    return str(x).strip().lower()

def score_field(pred, gold):
    return int(normalize(pred) == normalize(gold))

def precision_recall_f1(tp, fp, fn):
    p = tp / (tp + fp) if tp + fp else 0
    r = tp / (tp + fn) if tp + fn else 0
    f1 = 2*p*r / (p+r) if p+r else 0
    return p, r, f1

---

Rule of thumb

• Single-value fields → exact match / field accuracy
• Lists/sets of items → precision, recall, F1
• Spans → overlap-based matching
• Structured outputs → evaluate per field and per document

If you want, I can also give you:

• a Python evaluation script
• a schema for JSON extraction scoring
• or a method for fuzzy matching spans/fields