What are the best search engine results datasets?

The best search engine result datasets depend on your specific use case—whether you are training information retrieval (IR) and RAG models, developing learning-to-rank algorithms, conducting SEO analysis, or researching user search behavior.

The top datasets across these categories are detailed below.

---

1. Best Benchmarks for Neural Search, RAG & Information Retrieval

These datasets contain real-world queries, retrieved search result passages/documents, and relevance annotations.

• MS MARCO (Microsoft Machine Reading Comprehension)

  • Source: Real, anonymized Bing search queries.
  • Size: 1M+ queries, ~8.8M passages, and 3.2M web documents.
  • Use Case: The undisputed industry gold standard for passage retrieval, document ranking, and question answering.
  • Key Feature: Contains human-annotated answers and real search result passages.

• Google Natural Questions (NQ)

  • Source: Real Google Search queries paired with full Wikipedia pages.
  • Size: ~307,000 training queries with annotated short/long answer spans.
  • Use Case: Open-domain Question Answering (QA), dense retrieval model training, and generative AI search.

• BEIR (Benchmarking Information Retrieval)

  • Source: A multi-dataset benchmark aggregating 15+ retrieval datasets (including MS MARCO, NQ, BioASQ, TREC-COVID).
  • Size: Millions of documents across diverse domains.
  • Use Case: Zero-shot evaluation of search engine models (testing how well search models generalize outside their training domain).

• ClueWeb22 / ClueWeb12 (Carnegie Mellon & Lemur Project)

  • Source: Multi-billion-page web crawl designed to emulate commercial search indexes.
  • Size: ClueWeb22 contains 10 billion web pages in multi-lingual contexts.
  • Use Case: Academic web search simulation, large-scale indexing, and ranking research.

---

2. Best Learning-to-Rank (LTR) Datasets

Learning-to-Rank datasets provide pre-computed features (e.g., BM25 scores, PageRank, term frequencies) and relevance labels for queries and search engine results.

• MSLR-WEB10K & MSLR-WEB30K (Microsoft)

  • Size: 10,000 (WEB10K) and 30,000 (WEB30K) queries with 136 feature extractions per query-document pair.
  • Use Case: Training machine learning algorithms (like XGBoost, LightGBM, or LambdaMART) to rank web search results.

• Yahoo! Learning to Rank Challenge Dataset (C14/C14A)

  • Size: ~30,000 queries and over 700,000 rated URLs.
  • Use Case: Benchmark dataset for rank-aggregation and gradient boosting tree ranking models.

• Istella LTR Dataset

  • Size: 33,000+ queries with 220 features per query-document pair (over 10 million pairs).
  • Use Case: High-dimensional, large-scale learning-to-rank research based on European web searches.

---

3. Best Search Query Logs & Click Datasets

These datasets focus on user behavior on search engine result pages (SERPs)—what users search for, what they click, and dwell times.

• ORCAS (Open Resources for Information Retrieval)

  • Source: Bing user click logs.
  • Size: ~18 million user clicks covering 10 million queries and 1.4 million document URLs.
  • Use Case: Modeling user search behavior, click-through-rate (CTR) prediction, and pseudo-relevance feedback.

• AOL4FOLTR / Historical Query Datasets

  • Source: Anonymized search query logs.
  • Size: Millions of user queries, timestamps, click sequences, and user identifiers.
  • Use Case: Researching user query Reformulation, federated search, and search session intent.

---

4. Best Datasets for SEO, SERP Analysis & Machine Learning Scrapes

If you are looking for modern scraped Search Engine Result Pages (SERPs)—containing titles, meta descriptions, positions, and featured snippets for SEO or ML research:

• Kaggle SERP Datasets (e.g., SEO-Data / Google SERP Datasets)

  • Contents: Structured JSON/CSV files containing thousands of keywords, organic positions (1–100), titles, meta descriptions, and featured snippet types.
  • Use Case: Predict ranking improvements, analyze SEO factor weights, or train automated content optimization tools.

• Commercial SERP Data Dumps (DataForSEO, SerpApi, BrightData)

  • Contents: Real-time or batch-downloadable global/local SERP snapshots from Google, Bing, and DuckDuckGo.
  • Use Case: Custom web scraping for modern SEO analysis, localized rank tracking, and AI search monitoring (e.g., tracking Google AI Summaries/Overviews).

---

Summary Recommendation

Goal / Application	Recommended Dataset
Passage / Neural Search Training	MS MARCO or Google Natural Questions
Evaluating Search Generalization	BEIR Benchmark
Building Classical Ranking Models (LTR)	MSLR-WEB30K or Istella
User Intent & Click Prediction	ORCAS
SEO & SERP Structure Analysis	Kaggle SERP Datasets or DataForSEO API