Where most of the text actually comes from. Common Crawl is a non-profit that has been copying large parts of the public web for years and giving the results away, and it or something like it underpins nearly every large model. This is why models know about obscure hobbies and forum arguments, and also why the question of consent has become so contentious.
Technical view
Raw crawl data is extremely noisy and needs aggressive filtering, so derived collections such as C4, RefinedWeb and FineWeb apply published pipelines to produce usable subsets. Coverage is skewed towards English, towards pages that were linked to, and towards material that happened to be public at crawl time. Publishers increasingly block crawlers through robots.txt and licensing terms, which is gradually shifting the balance towards licensed and synthetic sources.
Related
Part of AI Terminology Index, section: Data.