Where can I find SERP datasets?

Where you can find Search Engine Results Page (SERP) datasets depends on whether you need them for academic/ML research, SEO and market analysis, or custom data collection.

---

1. Academic & Machine Learning Datasets (IR & NLP Research)

If you are building search models, recommendation engines, or training LLMs, these public, large-scale benchmarks offer real search engine query-and-result data:

• MS MARCO & MS MARCO Web Search (Microsoft): One of the largest open datasets for search. It features millions of real query-document pairs, user click logs, and rankings from Bing. Paired with ClueWeb22 for multi-billion web page corpus evaluation.
• ORCAS (Open Resources for Click-Analysis in Search): Contains over 18–20 million clicked query-document pairs mapped to the MS MARCO corpus.
• Google Natural Questions (NQ): A dataset of real Google search queries paired with full Wikipedia pages and human-annotated answer snippets.
• TREC Tracks (NIST): The Text REtrieval Conference offers historical benchmarks (Web Track, Deep Learning Track) containing query sets, ranked lists, and human relevance judgments.
• ir_datasets (Python Library): An open-source Python tool that provides instant, unified access to dozens of information retrieval datasets (MS MARCO, TREC, ClueWeb, etc.) without needing to manually scrape or format.

---

2. Open Data Repositories & Community Platforms

For smaller, pre-scraped, or topic-specific SERP data:

• Kaggle: Search keywords like "SERP", "Google Search Results", or "Keyword Rankings". You will find community-uploaded datasets covering rankings for specific industries, SERP feature extractions (e.g., People Also Ask, Knowledge Graph), and localized search rankings.
• Hugging Face Hub: Search the Datasets Hub for IR or search-retrieval corpora. Many researchers upload processed SERP snapshots, reranking benchmarks, and web search training sets here.
• GitHub: Search for topics or repos tagged with serp-dataset or google-search-dataset. Developers frequently post open CSV/JSON dumps collected using custom scrapers or libraries like advertools.

---

3. Commercial SEO Data Providers (Bulk Historical Downloads)

If you need massive enterprise-grade SERP datasets (e.g., top 100 Google results across millions of keywords, SERP features like AI Overviews, Local Packs, or Ads):

• DataForSEO: Offers bulk raw Google SERP Database downloads (CSV/JSON) covering millions of keywords across multiple countries and languages.
• Serpstat Database Downloads: Provides bulk access to top 100 Google SERP data across billions of keywords and 200+ regions.
• Bright Data Datasets: Offers pre-collected web datasets, including structured Google SERP dumps.
• SEMrush / Ahrefs / Moz: While primarily web dashboards, enterprise plans allow programmatic exports of full keyword SERP history, SERP feature distribution, and rank tracking data.

---

4. Live / On-Demand Generation (SERP APIs)

If static datasets don't meet your needs, you can easily generate a custom SERP dataset using a SERP API that handles IP proxies, anti-bot measures, and structured JSON parsing:

• SerpApi
• DataForSEO API
• ValueSERP
• Bright Data SERP API
• SERPdog

---

Summary Recommendation

• For ML / Information Retrieval / AI Training: Start with MS MARCO / ir_datasets on Python.
• For Quick Analysis / Exploratory Projects: Search Kaggle or Hugging Face.
• For SEO / Market Intelligence: Use DataForSEO bulk downloads or a SERP API to scrape target keywords.