Prompt
Which dataset should I use instead of Common Crawl for cleaner text data?
Latest observation
If your goal is cleaner text than Common Crawl, the best replacement depends on what you need the data for:
Good cleaner alternatives
-
Wikipedia / Wikidata text dumps
Very clean, well-structured, great for general language modeling and knowledge-heavy tasks. -
The Pile (or filtered subsets of it)
A curated mix of higher-quality sources than raw web crawl data, though not all components are equally clean. -
C4 (Colossal Clean Crawled Corpus)
Derived from Common Crawl but heavily filtered and cleaner than raw CC. A common choice when you still want web-scale data. -
RedPajama / SlimPajama
Open reproductions of LLM pretraining mixtures, including filtered web text plus other high-quality sources. -
Books datasets
- Project Gutenberg
- BookCorpus-like datasets
Generally cleaner prose, but narrower in style and domain.
-
OpenWebText / OpenWebText2 Cleaner web text filtered from high-quality webpages linked on Reddit; often used as a more curated web corpus.
If you want the shortest recommendation
- For general clean pretraining text: C4 or OpenWebText2
- For very clean, structured text: Wikipedia
- For a broader high-quality mixture: The Pile or RedPajama
If you tell me your use case—LLM pretraining, fine-tuning, classification, or retrieval—I can recommend the best dataset more specifically.