What are the best web datasets for AI training?

The best web datasets for AI training depend on the modality (text, code, multimodal) and training phase (pre-training vs. fine-tuning/instruction tuning). Modern web datasets are no longer just raw internet dumps; the best ones undergo rigorous deduplication, quality filtering, synthetic grading, and safety cleaning.

Below are the premier open and standard web datasets across different AI modalities.

---

1. General Text & Language Model Pre-Training

• FineWeb & FineWeb-Edu (Hugging Face)
  • Size: ~15 Trillion tokens (FineWeb) / 1.3T+ tokens (FineWeb-Edu)
  • Best For: State-of-the-art general LLM pre-training.
  • Why it’s among the best: Derived from 96 Common Crawl snapshots, FineWeb is considered one of the highest-performing open pre-training web datasets. FineWeb-Edu applies an additional classifier trained on synthetic grades to select highly educational web pages (math, science, humanities), vastly improving model reasoning capabilities per token spent.
• RedPajama / RedPajama-v2 (Together AI / Open Community)
  • Size: 30+ Trillion tokens
  • Best For: Customizable LLM pre-training pipelines.
  • Why it’s among the best: Created as an open reproduction of Meta's LLaMA pre-training mix. RedPajama-v2 includes tens of trillions of raw tokens paired with pre-computed quality signals (toxicity, perplexity, duplicate ratios), enabling developers to filter their own subsets.
• Dolma (Allen Institute for AI - AI2)
  • Size: 3 Trillion tokens
  • Best For: Fully open, reproducible research.
  • Why it’s among the best: Dolma was built to train AI2’s open-source model, OLMo. Its dataset creation scripts, source mixes (web, code, academic papers, books), and filtering recipes are fully open source, making it ideal for standardizing scientific research.
• RefinedWeb (Technology Innovation Institute)
  • Size: 5 Trillion tokens
  • Best For: High-performance web-only pre-training.
  • Why it’s among the best: Originally built for the Falcon LLM family, RefinedWeb relies heavily on aggressive deduplication and quality heuristics, proving that a high-quality web-only dataset can outperform standard web mixes.
• Common Crawl
  • Size: Multi-petabytes
  • Best For: Teams building custom extraction/filtering pipelines.
  • Why it’s among the best: Common Crawl is the foundation for almost every web dataset listed above. It is a raw monthly archive of billions of web pages. While uncleaned, it offers total control over custom pipeline design.

---

2. Code Datasets (Web-Scraped Repositories)

• The Stack v2 (BigCode & Hugging Face)
  • Size: 67+ million repositories across 600+ programming languages
  • Best For: Code generation models and developer copilots.
  • Why it’s among the best: Built in partnership with Software Heritage, The Stack v2 is the definitive open dataset for code. It includes permissive licensing checks, strict deduplication, and an opt-out mechanism for developers wanting to exclude their repos.
• StarCoder Data
  • Size: ~780 GB of code and code-adjacent text.
  • Best For: Pre-training code-specialized LLMs.
  • Why it’s among the best: Contains code files, GitHub issues, pull requests, and Jupyter notebooks cleaned specifically for programming tasks.

---

3. Multimodal Datasets (Image, Video, & Text)

• LAION-5B / LAION-400M
  • Size: 5.85 billion image-text pairs
  • Best For: Vision-Language models (like CLIP) and Diffusion models (like Stable Diffusion).
  • Why it’s among the best: An open index of URLs and ALT text derived from Common Crawl, filtered using OpenAI’s CLIP model.
• DataComp / DataComp-LM
  • Size: Millions to billions of image-text and text-only samples.
  • Best For: Next-generation multimodal model benchmarking.
  • Why it’s among the best: Designed as a benchmark for dataset quality, offering standard candidate pools (from small to mega-scale) to test image-text filtering algorithms.
• WIT (Wikipedia-Based Image Text)
  • Size: ~37 million image-text pairs across 100+ languages.
  • Best For: Multilingual visual question answering and captioning.
  • Why it’s among the best: Sourced directly from Wikipedia, offering significantly higher factual accuracy and lower toxicity than raw web scrapes.

---

4. Ethical & Public Domain Web Datasets

• Common Corpus (PleIAs)
  • Size: ~2 Trillion tokens
  • Best For: Legally compliant, commercial-safe LLM pre-training.
  • Why it’s among the best: The largest collection of open, copyright-free public domain data (books, government documents, historical web archives, open code) designed explicitly to prevent copyright infringement claims during AI training.

---

How to Choose the Right Dataset

1. For Training an LLM from Scratch: Start with FineWeb or FineWeb-Edu for general knowledge/reasoning, combined with The Stack v2 for code.
2. For Custom Data Quality Research: Use RedPajama-v2 to experiment with custom quality filters using pre-computed metadata signals.
3. For Legal Safety/Commercial Deployment: Integrate Common Corpus and ensure strict licenses on code datasets.
4. For Vision & Multimodal: Use DataComp or filtered subsets of LAION-5B.