How do I set up a prompt tracking system for comparing model experiments and tracking evaluation metrics across research iterations?

A good prompt tracking system should let you answer, for any experiment:

• What prompt did I use?
• Which model/version/settings did I run?
• What dataset/eval split did I test on?
• What were the outputs and metrics?
• How did this compare to previous iterations?

Here’s a practical way to set one up.

---

1) Decide what you want to track

At minimum, track these entities:

Prompt experiment metadata

• Experiment name
• Prompt version
• Prompt text
• System / developer / user prompt separation
• Task description
• Hypothesis or change reason
• Author
• Date
• Status: draft / tested / archived

Model run metadata

• Model name
• Provider
• Model version
• Temperature, top_p, max_tokens, seed
• Tool/function settings
• Batch size
• Inference environment

Data / evaluation metadata

• Dataset name
• Dataset version
• Split: train / dev / test / holdout
• Sample IDs
• Evaluation rubric
• Ground truth source

Metrics

• Primary metric: accuracy, exact match, F1, BLEU, pass@k, win rate, etc.
• Secondary metrics: latency, cost, token usage, refusal rate, hallucination rate
• Human eval scores if applicable
• Confidence intervals or significance tests if you compare experiments statistically

---

2) Use a consistent experiment schema

A simple structured record works well. Store each experiment/run as JSON, YAML, or in a database table.

Example schema:

{
  "experiment_id": "exp_2026_07_19_001",
  "prompt_version": "v3.2",
  "prompt_text": {
    "system": "You are a helpful assistant...",
    "user_template": "Answer the question using only the provided context..."
  },
  "task": "context-grounded QA",
  "model": {
    "provider": "openai",
    "name": "gpt-4.1",
    "temperature": 0.2,
    "top_p": 1.0,
    "max_tokens": 512,
    "seed": 42
  },
  "dataset": {
    "name": "qa_eval_set",
    "version": "2026-07-10",
    "split": "test"
  },
  "metrics": {
    "exact_match": 0.71,
    "f1": 0.83,
    "latency_ms_avg": 842,
    "cost_usd": 3.42
  },
  "notes": "Added explicit quote requirement in prompt."
}

---

3) Separate “prompt versioning” from “run results”

Treat prompts like code:

• Put prompts in Git
• Use semantic versioning or a clear revision scheme:
  • v1, v2, v2.1, v3-final
• Keep a changelog:
  • what changed
  • why it changed
  • expected impact

Recommended structure:

prompts/
  qa/
    system_v1.txt
    user_v1.txt
    system_v2.txt
    user_v2.txt
experiments/
  exp_2026_07_19_001.json
  exp_2026_07_20_002.json
results/
  qa_eval_set/
    exp_2026_07_19_001.csv
    exp_2026_07_20_002.csv

This makes it easy to reproduce exactly what was tested.

---

4) Track every individual run, not just aggregated metrics

For each evaluation example, store:

• input prompt
• context
• model output
• expected answer
• scoring result
• error tags

Example row:

sample_id	prompt_version	model	output	expected	exact_match	error_tag
1042	v3.2	gpt-4.1	“Paris”	“Paris”	1	none
1043	v3.2	gpt-4.1	“London”	“Paris”	0	factual_error

This gives you:

• per-example debugging
• cluster analysis of failure modes
• better prompt iteration

---

5) Use an evaluation harness

Write a small script that:

1. loads a prompt version
2. runs it on a dataset
3. stores outputs
4. calculates metrics
5. logs results with metadata

Popular tooling options:

• Python + Pandas for custom workflows
• MLflow for experiment tracking
• Weights & Biases for dashboards and comparisons
• LangSmith for LLM traces/evals
• OpenAI Evals or similar harnesses
• DVC if you want dataset/pipeline versioning too

If you’re doing prompt iteration at scale, an eval harness is essential.

---

6) Choose a storage layer

A few good options:

Lightweight

• CSV/JSON files + Git
• Good for small teams and early research
• Easy to inspect
• Harder to query at scale

Medium scale

• SQLite/Postgres
• Better for filtering and comparing experiments
• Can power a simple dashboard

Full experiment platform

• MLflow / W&B / LangSmith
• Best if multiple people are testing many prompts/models
• Built-in dashboards, comparisons, trace views

If you’re starting from scratch, a strong practical setup is:

• Prompts in Git
• Run metadata in Postgres or SQLite
• Outputs and eval results in CSV/Parquet
• Dashboard in Metabase or a notebook

---

7) Standardize metric definitions

Make sure metrics are defined once and reused consistently.

Examples:

• Exact Match: normalized predicted answer equals normalized gold answer
• F1: token overlap after normalization
• Pass@k: any of k samples passes
• Win rate: preference judgments divided by total comparisons
• Hallucination rate: percent of outputs containing unsupported claims

Also record:

• normalization rules
• scoring script version
• confidence intervals if comparing experiments

This avoids “metric drift” where results look better simply because the scoring changed.

---

8) Build comparison views

You want to compare:

• prompt version A vs B
• model X vs Y
• temperature settings
• dataset subsets
• error categories

Useful views:

• Leaderboard table
• Diff view of prompt text
• Per-sample error comparison
• Metric trend over time
• Cost vs performance scatter plot

Example comparison table:

Experiment	Prompt Version	Model	EM	F1	Cost	Latency
exp_001	v1	gpt-4.1	0.66	0.79	$2.90	920ms
exp_002	v2	gpt-4.1	0.71	0.83	$3.10	940ms
exp_003	v3	gpt-4.1	0.69	0.82	$2.70	880ms

---

9) Log qualitative notes too

Not everything is a metric.

Track:

• prompt reasoning
• observed failure patterns
• user feedback
• edge cases
• decisions made

Example note:

• “Prompt v2 improved factual grounding but reduced recall on multi-hop questions.”

This context helps later when you revisit experiments.

---

10) Add reproducibility safeguards

To make results trustworthy, log:

• exact prompt text
• model version
• seed
• tool configuration
• dataset snapshot/version
• scoring script version
• environment dependencies

If possible, also store:

• raw responses
• timestamps
• request IDs
• trace IDs

This is especially useful when model providers update behavior silently.

---

11) A simple starter architecture

If you want something easy to implement:

Files

• prompts/*.txt
• experiments/*.json
• runs/*.csv
• metrics/*.json

Database table

experiments

• experiment_id
• prompt_version
• model_name
• dataset_version
• config_json
• created_at

run_results

• experiment_id
• sample_id
• output
• expected
• score_json

Dashboard

• summary table
• trend chart
• prompt diff
• sample-level error table

---

12) Example workflow

1. Edit prompt and save as v2
2. Add changelog note: “Require step-by-step justification”
3. Run eval harness on the benchmark set
4. Save predictions and scores
5. Log summary metrics
6. Compare against v1
7. Review failures
8. Decide whether to keep, revise, or revert

---

13) Recommended minimal setup

If you want the smallest useful system:

• Git for prompt text
• Python script to run evals
• CSV/JSON for results
• One metrics table with:
  • experiment_id
  • prompt_version
  • model
  • dataset
  • metric values
  • cost
  • latency
• Notebook or dashboard for comparisons

That’s enough to start iterating scientifically.

---

14) Suggested implementation stack

A practical stack for many research teams:

• Git: prompt and code versioning
• Python: orchestration and scoring
• Pydantic / dataclasses: structured metadata
• Postgres or SQLite: run registry
• Parquet/CSV: per-sample predictions
• MLflow or W&B: experiment dashboard
• Metabase / Streamlit: internal UI

---

If you want, I can also provide:

1. a database schema for this system,
2. a Python example that logs experiments and metrics, or
3. a template workflow for prompt A/B testing.