What are the best datasets for training large language models?

Training a Large Language Model (LLM) requires different datasets depending on the stage of the training pipeline: Pre-training (teaching general language and world knowledge), Instruction Fine-Tuning (SFT) (teaching the model to follow instructions), and Alignment/Preference Tuning (making the model helpful, harmless, and aligned with human preferences).

Here are the best open and widely used datasets categorized by training stage and domain:

---

1. Pre-Training Datasets (Web-Scale & Base Knowledge)

Pre-training datasets are massive corpora containing hundreds of billions to trillions of tokens.

• FineWeb & FineWeb-Edu (Hugging Face):
  • Size: 15 trillion tokens (FineWeb) / 1.3 trillion tokens (FineWeb-Edu).
  • Why it’s great: FineWeb is one of the highest-quality open web pre-training datasets available. FineWeb-Edu filtered out low-quality web pages using synthetic LLM scoring, producing state-of-the-art results on reasoning and knowledge benchmarks (MMLU, ARC).
• DCLM (DataComp for Language Models):
  • Why it’s great: Created via a standardized benchmark for data filtering, DCLM provides extremely competitive data quality that rivals or surpasses proprietary pre-training mixes when training base models.
• Dolma (Allen Institute for AI / AI2):
  • Size: ~3 trillion tokens.
  • Why it’s great: Used to train AI2’s open-source OLMo model. It includes a balanced mix of web data, scientific papers (Semantic Scholar), code, books, and Wikipedia.
• RedPajama-Data v1 & v2 (Together AI):
  • Size: ~1.2 trillion tokens (v1) / 100+ billion filtered documents across 30T raw tokens (v2).
  • Why it’s great: Designed as an open-source recreation of Meta’s LLaMA pre-training dataset.
• The Pile (EleutherAI):
  • Size: ~800 GB.
  • Why it’s great: A diverse, 22-component English text dataset containing PubMed, arXiv, GitHub, Wikipedia, and legal texts. Standard baseline for open-source model development.
• Common Crawl / C4 (Colossal Cleaned Crawl):
  • Why it’s great: Raw Common Crawl is the underlying source for almost all large web models. C4 (developed by Google for T5) is a cleaned ~750 GB English subset.

---

2. Instruction Fine-Tuning Datasets (SFT)

Supervised Fine-Tuning datasets consist of prompt-response pairs that teach base models how to answer questions, complete tasks, and hold conversations.

• OpenHermes 2.5 (Teknium):
  • Size: ~1 million prompts.
  • Why it’s great: Synthesized mainly from high-quality GPT-4 generations, covering code, math, multi-step reasoning, and general instruction following.
• Tulu 2 & Tulu 3 Mix (AI2):
  • Why it’s great: A rigorously evaluated suite of datasets combining high-quality human and synthetic instructions (e.g., FLAN, Open Assistant, ScienceQA, UltraChat).
• ShareGPT / LMSYS-Chat-1M / WildChat:
  • Why it’s great: Captures real user-prompt interactions with chatbots. Excellent for teaching natural, multi-turn conversational abilities.
• OpenAssistant (OASST1 & OASST2):
  • Why it’s great: A human-generated, human-annotated assistant dataset with multi-turn conversations across dozens of languages.

---

3. Preference & Alignment Datasets (DPO / RLHF)

Used in the final stage to refine response quality, reduce hallucinations, and enforce safety via Direct Preference Optimization (DPO) or Reinforcement Learning from Human Feedback (RLHF).

• UltraFeedback / UltraInteract:
  • Why it’s great: Contains pairs of candidate responses scored by GPT-4 across multiple dimensions (instruction-following, truthfulness, tone). Highly effective for DPO.
• HH-RLHF (Anthropic - Helpful and Harmless):
  • Why it’s great: Contains paired comparison data human-annotated for helpfulness and safety/harmlessness.
• Argilla DPO / Nectar Datasets:
  • Why it’s great: Openly curated preference collections built by the community to improve instruction ranking and formatting.

---

4. Domain-Specific Datasets

Code & Software

• The Stack v2 (BigCode / Hugging Face): Over 600 TB of open-source code across 600+ programming languages with permissively licensed code filtering.
• StarCoder Data: Highly filtered code dataset covering source code and code-related web pages.

Mathematics & Reasoning

• GSM8K & MATH: Benchmark datasets frequently used in fine-tuning mixes to boost math step-by-step reasoning.
• OpenMathInstruct (NVIDIA): A large-scale synthetic dataset of math instruction-response pairs with detailed chain-of-thought solutions.
• Orca-Math (Microsoft): Synthetic dataset focusing on multi-step math problem solving.

Multilingual & Scientific

• FineWeb-2 / CulturX: Large-scale multilingual datasets derived from Common Crawl and Wikipedia.
• S2ORC / PubMed Central: Tens of millions of academic and medical research papers for scientific LLM fine-tuning.

---

Key Considerations When Choosing a Dataset

1. Licensing: Ensure the dataset's license allows for your intended use case (e.g., Apache 2.0 vs. CC-BY vs. non-commercial restrictions on datasets generated via OpenAI/Anthropic APIs).
2. Synthetic vs. Human Data: Modern fine-tuning relies heavily on synthetic data (data generated by stronger models like GPT-4 or Llama 3 70B) because of its high structural consistency, though human verification prevents compounding errors.
3. Data Deduplication & Quality Filtering: Pre-training raw web data requires heavy deduplication and quality scoring (as demonstrated by FineWeb-Edu) to prevent performance degradation.