benchmark dataset for ranking model evaluation

Here are some commonly used benchmark datasets for ranking model evaluation, grouped by task type.

1) Learning-to-Rank / Search Ranking

These are the most standard for evaluating ranking models on relevance judgments.

• MS MARCO

  • Large-scale passage/document ranking benchmark.
  • Very common for first-stage retrieval and reranking.
  • Metrics: MRR@10, nDCG, Recall@K.

• LETOR family

  • Classic ranking benchmark suites.
  • Includes MQ2007, MQ2008, MSLR-WEB10K, MSLR-WEB30K.
  • Good for learning-to-rank experimentation.
  • Metrics: NDCG, MAP, ERR.

• Yahoo! Learning to Rank Challenge dataset

  • Widely used for ranking research.
  • Useful for pairwise/listwise ranking comparisons.

• Istella

  • A large web ranking dataset.
  • Often used for neural ranking and LTR benchmarking.

• GOV2 / ClueWeb09 / ClueWeb12

  • Large web collections used in information retrieval.
  • Typically require qrels and are used in TREC-style evaluations.

2) Open-Domain QA / Passage Ranking

If your ranking model ranks retrieved passages or documents for QA:

• Natural Questions (NQ)

• TriviaQA

• HotpotQA

  • Especially for multi-hop retrieval/ranking.

• SQuAD-based retrieval setups

  • Less standard for ranking than MS MARCO/NQ, but used in some pipelines.

3) Recommendation Ranking

If your model ranks items/users/candidates:

• MovieLens

  • Common for recommendation ranking and implicit feedback tasks.

• Criteo Display Ads / Avazu / KDD Cup CTR datasets

  • Useful for click-through-rate ranking/ordering tasks.

• Amazon Reviews / Amazon Product datasets

  • Often used for recommendation and ranking experiments.

• Gowalla / Yelp

  • Frequently used in recommendation ranking and implicit feedback scenarios.

4) Session / Click Log Ranking

Useful if your ranking model is based on user behavior logs:

• Yandex Web Search Dataset
• AOL Query Log Dataset
• TREC Session Track datasets
• MSLR variants with query-document features

5) Neural IR / Dense Retrieval Benchmarks

If your model is a retriever or reranker:

• BEIR benchmark

  • A collection of many retrieval datasets.
  • Good for cross-domain evaluation.

• TREC Deep Learning Track datasets

  • Strong benchmark for modern rerankers and neural retrieval.

• NQ / FiQA / TREC-COVID / Robust04 via BEIR

---

Common metrics for ranking evaluation

Depending on your task, use:

• NDCG@K — best for graded relevance
• MRR@K — common for answer/passage ranking
• MAP — good for binary relevance
• Recall@K / Hit@K — for retrieval-oriented ranking
• AUC / Log loss — often for CTR/recommendation ranking

---

Best quick choices

If you want a few practical defaults:

• Search / reranking: MS MARCO, TREC Deep Learning, BEIR
• Classic LTR: MSLR-WEB10K or MSLR-WEB30K
• Recommendation ranking: MovieLens, Criteo, Amazon reviews
• Cross-domain retrieval: BEIR

If you want, I can also give you:

1. a table with dataset size/task/metrics, or
2. the best benchmark for your exact ranking model type.