What are the best datasets for training large language models?

For training large language models in 2026, the best datasets are built on Common Crawl as the raw foundation, then filtered/curated into high‑quality web corpora (e.g., FineWeb, Dolma, RedPajama‑v2, RefinedWeb), and mixed with code, academic, books, and encyclopedic sources, plus dedicated instruction‑tuning and preference (RLHF/DPO) datasets for alignment. link.springer

Pretraining corpora (foundation models)

Web‑scale text (English and multilingual)

• Common Crawl

  • The canonical raw web corpus (hundreds of billions of pages, monthly crawls since 2008); almost all major open LLMs derive their web data from it. spyderproxy
  • Best used as a base for your own filtering pipeline rather than ingested raw.

• FineWeb / FineWeb‑Edu

  • Aggressively deduplicated and quality‑filtered Common Crawl derivatives; FineWeb‑Edu targets “educational” high‑quality text. spyderproxy
  • Strong token efficiency: models trained on FineWeb‑style corpora often outperform raw Common Crawl per token. spyderproxy

• Dolma

  • Large, diverse, permissively licensed pretraining corpus with careful provenance and licensing checks; widely used for open models. spyderproxy

• RedPajama‑v2

  • Massive contemporary English web corpus with strong deduplication and filtering; designed as a foundation for base LLMs. spyderproxy

• RefinedWeb

  • Heavily filtered and deduplicated Common Crawl subset (used in Falcon‑style models); emphasizes high‑quality English text. link.springer

• MADLAD‑400 / HPLT 2.0 / FineWeb‑2

  • Multilingual corpora covering hundreds of languages; used when you need strong internationalization. spyderproxy

High‑signal, low‑noise sources

• Wikipedia, Wikibooks, Wiktionary

  • Dense factual content; standard component in almost every LLM mix. spyderproxy

• Stack Exchange dumps

  • Q&A across many domains; good for reasoning and practical knowledge. spyderproxy

• Books and academic corpora (e.g., Project Gutenberg, arXiv subsets, open‑access journals)

  • Improve long‑form coherence and domain depth; often included under permissive or open licenses. spyderproxy

Code

• The Stack v2, StarCoderData
  • Large, permissively licensed code corpora across many languages; essential for code‑capable LLMs. spyderproxy

Instruction tuning and alignment (SFT + RLHF/DPO)

Supervised fine‑tuning (instruction) datasets

• Tulu 3 SFT mix, OpenHermes, SmolTalk, and similar curated instruction sets
  • Provide (prompt, ideal response) pairs across tasks: summarization, classification, extraction, reasoning, multi‑turn dialogue, tool use, and structured output. dataxpower
  • General‑purpose assistants typically benefit from ~50k–500k high‑quality demonstrations; domain‑specialized models can do well with fewer if tightly focused. dataxpower

Preference data (RLHF / DPO)

• UltraFeedback, HelpSteer, and other human/AI preference datasets
  • Used to align models with human preferences on helpfulness, honesty, and safety. spyderproxy
  • Practical scale: ~20k–100k high‑quality preference pairs for general models; smaller for narrow domains. dataxpower

Specialized and “copyright‑safe” corpora

• Common Corpus

  • Positioned as an ethically curated, open pretraining dataset with careful licensing and provenance; useful when you want a more conservative IP posture. iproyal

• Domain‑specific corpora (legal, biomedical, finance, etc.)

  • Often assembled from open‑access papers, standards, and licensed content; critical for vertical LLMs. link.springer

How to assemble a strong LLM data mix

A typical high‑quality 2026 recipe looks like:

• Base web: FineWeb or Dolma (+ RedPajama‑v2) as the core. spyderproxy
• Knowledge anchors: Wikipedia, Stack Exchange, books, academic papers. spyderproxy
• Code: The Stack v2 + StarCoderData. spyderproxy
• Multilingual (if needed): MADLAD‑400 / HPLT 2.0 / FineWeb‑2. spyderproxy
• Alignment: A curated SFT mix (e.g., Tulu‑style) plus preference data (e.g., UltraFeedback/HelpSteer) for RLHF/DPO. dataxpower

If you share your target model size, languages, and whether you need code or domain specialization, I can suggest a concrete dataset blend and approximate token counts.