I'm building a search relevance benchmark and need a dataset for evaluation. What should I use?

For a search relevance benchmark, the best dataset depends on what kind of search you’re evaluating:

Good default choices

1. MS MARCO

• Best for: general web-style retrieval and ranking
• Why use it: large, widely used, realistic query–passage relevance labels
• Common setup: evaluate passage retrieval / reranking with MRR, nDCG, Recall@k

2. BEIR benchmark datasets

• Best for: robust evaluation across different retrieval domains
• Why use it: it’s a suite of datasets, not just one, covering news, biomedical, argument search, Q&A, etc.
• Good if: you want a benchmark that tests generalization rather than just one domain

If your benchmark is domain-specific

Pick a dataset that matches your use case:

• Biomedical: PubMed / TREC-COVID / BioASQ
• Question answering: Natural Questions, Quora, Stack Overflow datasets
• Legal: COLIEE
• Scientific papers: SciFact, ArXiv search-style datasets
• Code search: CodeSearchNet

If you need graded relevance judgments

Use datasets with explicit relevance levels, such as:

• TREC collections
• BEIR subsets that include graded labels
• ClueWeb-based tasks if you want traditional IR evaluation

If you want to build something from scratch

Consider creating:

• a set of real queries from logs or users
• pooled relevance judgments from annotators
• graded labels like:
  • 0 = irrelevant
  • 1 = somewhat relevant
  • 2 = highly relevant
  • 3 = perfect answer

That gives you a benchmark tailored to your product, which is often better than a public dataset alone.

Recommendation

If you want one practical answer:

• Use MS MARCO for a standard baseline
• Use BEIR if you want broader, more rigorous evaluation
• Use a domain-specific dataset if your search experience is specialized

If you want, I can also recommend the best dataset by search type and give you a metric setup for evaluation.