The Architecture of Algorithmic Knowledge: An Investigative Report on LLM Training Data Provenance and Source Concentration

Full report

On this page

The foundational capabilities of large language models are fundamentally constrained and directed by the corpora upon which they are pre-trained. Historically, the prevailing assumption within the artificial intelligence community has been that internet-scale scraping inherently captures a democratic cross-section of human knowledge. In this theoretical model, web scraping organically blends the specialized expertise of major corporations, government agencies, scientific publishers, and universities with the colloquial, conversational, and cultural contributions of ordinary people, independent writers, and online communities. However, large-scale empirical audits of major open-source pre-training datasets—including Common Crawl, C4, FineWeb, Dolma, and The Pile—reveal a highly structured, heavily filtered, and systematically skewed reality. The transition from raw internet data to refined pre-training corpora involves a series of technical transformations that actively curate whose knowledge is preserved and whose is erased. This investigative report examines the provenance of AI training datasets over time, analyzing how institutional authority, website popularity, backlink concentration, and automated quality filters disproportionately influence the distribution of knowledge available for model training. The analysis yields five key findings. First, despite the perception of the web as a decentralized forum, measurable dataset composition is overwhelmingly dominated by large institutional sources, with scientific publishers, government patent offices, and encyclopedic databases contributing a profoundly disproportionate share of total tokens compared to independent blogs and personal websites. Second, corpus filtering procedures—particularly domain blocklists and lexical heuristics designed to remove "toxic" or "low-quality" content—disproportionately erase minority dialects and community forums, acting as mechanisms of algorithmic silencing. Third, the graph dynamics that dictate web crawling prioritize harmonic centrality and backlink concentration, meaning independent publications lacking robust backlink profiles are systematically underrepresented before content filters are even applied. Fourth, the recent shift toward automated quality filters, such as utilizing proprietary models to grade the "educational value" of web data, inherently privileges highly structured, academic prose over conversational truths and independent storytelling. Finally, there is a widening proprietary divide; while open-source corpora are increasingly reliant on permissive, academically oriented licenses due to legal risk, proprietary models are suspected of hoarding high-quality, creative, and community-driven dialog data, creating a massive divergence between the measurable concentration of open datasets and the unverified composition of frontier commercial models.

The Evolution and Composition of Major Datasets

To understand how ordinary individuals and massive institutions contribute to AI ecosystems, one must trace the chronological evolution and structural composition of foundational corpora. Modern large language models rely on thousands of disparate datasets aggregated into colossal collections, yet the actual token volume is highly concentrated among a few specific categories and processing methodologies1. Over time, dataset curation has evolved from manually aggregating domain-specific repositories to executing massive, automated extractions over decade-long spans of web history. This temporal expansion has allowed researchers to measure the relationship between compute-optimal scaling laws and dataset size, driving corpus sizes from the gigabyte scale into the multi-terabyte, trillion-token scale.

The Pile, developed by EleutherAI, represents one of the earliest concerted efforts to curate a diverse, high-quality pre-training corpus across multiple domains3. However, an analysis of its sub-components reveals a distinct preference for high-authority, institutional, and academic text over independent web pages. Within its 825 GiB volume, institutional repositories such as PubMed Central (90.27 GiB), ArXiv (56.21 GiB), FreeLaw (51.15 GiB), and USPTO Backgrounds (22.90 GiB) collectively command a massive share of the training weight5. While community-driven platforms like StackExchange and Wikipedia are present, independent blogs or personal websites are relegated entirely to the Pile-CC subset, which itself accounts for only 18.11% of the weighted training distribution5. As the demand for tokens escalated to satisfy scaling laws, researchers pivoted back to massive web scrapes. The Dolma dataset, curated by the Allen Institute for AI, expands the scale to 3 trillion tokens, utilizing 24 Common Crawl snapshots collected between May 2020 and June 20236. The FineWeb dataset further accelerates this temporal and volumetric scale, amassing 15 trillion tokens derived from 96 monthly Common Crawl snapshots spanning from 2013 to 20249. These massive shifts indicate a reliance on algorithmic web scraping over manual curation, fundamentally altering whose data is ingested.

Source-Category Shares: Institutional Dominance Versus Public Participation

While datasets like Dolma and FineWeb are predominantly sourced from the "general web," analyzing the exact domain-level concentrations within these scrapes exposes a profound institutional hegemony. In the Dolma dataset, the vast majority of tokens (2,415 billion) originate from Common Crawl, supplemented by specialized datasets such as peS2o/Semantic Scholar (57 billion tokens), Project Gutenberg (4.8 billion tokens), and Wikipedia (3.6 billion tokens)11. However, the composition of the Common Crawl subset is highly skewed. The most common internet domains in Dolma, measured per token, are patents.google.com, followed closely by major scientific publishers such as www\.nature.com and www\.frontiersin.org12. This institutional dominance is a recurring feature across all major web corpora. Audits of the C4 dataset, which served as the foundation for Google's T5 model, reveal a remarkably similar hierarchy. Out of the 156 billion tokens in C4, the top 25 domains are heavily skewed toward massive institutional aggregators, including patents.google.com, en.wikipedia.org, major news outlets like nytimes.com and latimes.com, open-access scientific publishers like plos.org, and publishing platforms like scribd.com13.

The sheer volume of patent data in these datasets represents a critical architectural insight. Patents contain highly stylized, formalized, and often arcane legal-technical language. The reality that the single most represented website in C4 is a patent aggregator implies that the linguistic patterns of patent lawyers and government bureaucrats exert a stronger gravitational pull on the base model's statistical distribution than the combined creative output of millions of independent bloggers15. Ordinary people and independent writers, whose contributions exist on standalone domains or smaller community platforms, constitute a highly fragmented long tail that is frequently decimated by the data processing pipeline.

The Mechanics of Exclusion: Crawl Graphs and Backlink Concentration

The institutional skew observed in web text corpora does not merely arise from chance; it is a structural byproduct of how web crawling algorithms operate. The assumption that Common Crawl operates as a perfect mirror of human web activity ignores the underlying graph dynamics that dictate domain selection and indexing priority. The Common Crawl foundation utilizes host- and domain-level graphs that rely heavily on algorithms measuring harmonic centrality and backlink rankings16. An authority score is assigned to domains based on these interconnectivity metrics, which fundamentally dictates how thoroughly a given website is crawled. Consequently, massive universities, government agencies, and major corporations—entities that inherently possess immense backlink networks due to their cultural and economic footprint—are crawled exhaustively. Conversely, independent writers, hobbyist forums, and personal publications generally lack the structural graph authority required to be indexed deeply. As a result, the knowledge produced by ordinary people is systematically underrepresented at the point of origin, entirely preempting their inclusion in the AI training ecosystem before any specific content filters are applied.

Corpus Filtering Procedures and Algorithmic Sanitization

Raw web scrapes are universally deemed too noisy for frontier model training, necessitating aggressive automated filtering pipelines. These procedures, however, are not culturally or linguistically neutral. They impose strict, programmatic norms on what constitutes acceptable, "high-quality" human knowledge, often resulting in the disproportionate erasure of marginalized communities and independent voices. Dataset refinement pipelines utilize intense deduplication and heuristic blocklisting. The CCNet pipeline, utilized in the construction of datasets like Dolma's web subset, is famously aggressive, filtering out approximately 84.2% of the total ingested Common Crawl content7. Similarly, the RefinedWeb dataset, utilized for the Falcon LLM, operates on raw HTML and utilizes a severe blocklist containing roughly 4.6 million domains, augmented by strict URL scoring based on violation word-lists18. To prevent exact memorization and remove boilerplate, RefinedWeb utilizes MinHash algorithms (e.g., [Figure omitted from source export], [Figure omitted from source export], [Figure omitted from source export]) to compute document similarity, explicitly excising exact substrings of [Figure omitted from source export] tokens18. While deduplication improves model performance by removing redundant navigation elements, it also heavily penalizes shared colloquialisms, widely circulated cultural commentary, and syndicated independent content, leaving behind unique, long-form institutional texts7. The most concerning implication of automated quality filtering is its demographic impact. An exhaustive audit of the C4 dataset demonstrated that blocklist filtering disproportionately removes minority dialects. Documents containing African American aligned English were removed at a staggering rate of 42%, and Hispanic aligned English at a rate of 32%, compared to a removal rate of just 6.2% for White aligned English13. This algorithmic sanitization effectively erases the cultural knowledge, conversational norms, and lived experiences of marginalized communities from the AI ecosystem, replacing it with standardized "high English." Furthermore, geographical representation remains highly stagnant over time. Over 51.3% of the C4 dataset consists of webpages hosted in the United States, with countries possessing massive English-speaking populations—such as India, Pakistan, Nigeria, and the Philippines—accounting for fractions of a percent of the URL share13. The Data Provenance Initiative has confirmed this trend, noting that geographical and linguistic representation has not meaningfully improved across the data ecosystem in a decade21.

The Shift Toward LLM-Annotated Quality Filters

Recent dataset curation techniques have evolved from heuristic word-list filters to utilizing the models themselves as arbiters of quality, a mechanism often referred to as LLM-as-a-judge. While this approach yields impressive benchmark performance, it profoundly impacts the epistemic diversity of the resulting dataset. The FineWeb-Edu dataset, a highly curated subset of the larger 15 trillion token FineWeb corpus, exemplifies this shift. To create FineWeb-Edu, researchers prompted Llama-3-70B-Instruct to annotate 500,000 web pages, scoring each sample's "educational quality" on a scale from 0 to 522. A linear regressor was subsequently trained on these synthetic annotations to evaluate and filter the entire multi-trillion token corpus24. This process resulted in two optimized subsets: a 1.3 trillion token version representing "very high educational content" and a 5.4 trillion token version for "high educational content"9. While models trained on FineWeb-Edu significantly outperform baselines on academic reasoning benchmarks such as MMLU, ARC, and OpenBookQA26, the rubric utilized to achieve these gains fundamentally redefines valuable knowledge as strictly "educational." The LLM judge naturally favors texts bearing the hallmarks of formal education: structured headings, rhetorical questions answered in subsequent paragraphs, academic citations, and highly formalized prose27. As a direct consequence, the semantic associations within FineWeb-Edu shift drastically. The dataset captures associations that lean heavily toward history and formal health, favoring technical relationships while discarding localized human intimacy26. Personal narratives, emotional support forums, independent artistic critiques, and ordinary human dialogue are systematically down-weighted and excised from the corpus because they do not resemble a textbook. The deployment of LLM-annotated filters enforces an algorithmic monoculture that explicitly elevates the linguistic styles of universities and major publishers over the communication styles of ordinary people.

The Divergence Between Measurable Open Corpora and Proprietary Models

A critical analytical distinction must be drawn between the measurable, documentable realities of open-source datasets and the unverified assumptions surrounding proprietary frontier models (e.g., GPT-4, Claude 3, and Gemini). Open-source initiatives, operating under increasing public and legal scrutiny regarding copyright infringement, are actively gravitating toward explicitly permissive data. The Data Provenance Initiative has documented a sharp and widening divide between commercially open and closed data, warning of a "rapid decline of the AI Data Commons" or a "Consent in Crisis" scenario where high-quality web sources increasingly utilize robots.txt exclusions to block scraping1. Consequently, open corpora like the Common Corpus intentionally over-index on public domain texts, Open Science, and Open Government documents to mitigate legal risk30. This self-imposed restriction inherently drives up the ratio of bureaucratic, institutional data in open-source models2. Conversely, proprietary model developers are heavily suspected of privately ingesting vast amounts of independent community data, recognizing the immense value of ordinary human conversation for conversational AI alignment. A rare historical glimpse into proprietary data mixtures is provided by the documentation for Google's LaMDA model, which was trained on 1.56 trillion words. The precise composition of LaMDA included a massive 50% allocation to dialog data sourced from public forums, alongside 12.5% C4 data, 12.5% Wikipedia, and 12.5% code documents13. This 50% allocation to public dialog highlights a central contradiction in the AI data landscape: while open-source models are increasingly pushed toward a sterile, institutional monoculture through legal risk and automated quality scoring2, proprietary models are likely hoarding the conversational knowledge, troubleshooting expertise, and creative reasoning created by the general public on platforms like Reddit and Stack Overflow.

The Necessity of Widespread Public Participation

The empirical findings of this investigation underscore a critical vulnerability in the current trajectory of artificial intelligence. The definition of "high-quality" training data has been subtly co-opted by algorithmic processes to mean "institutional, highly structured, and aggressively filtered." If the AI ecosystem continues to rely on patent filings, scientific journals, and heuristically sanitized web crawls, the resulting models will suffer from a profound representational deficit. Widespread public participation in knowledge creation—and the deliberate preservation of that knowledge within training corpora—is necessary for a representative AI ecosystem for several fundamental reasons: First, ensuring cultural and linguistic diversity requires unsterilized data. As demonstrated by the erasure of African American English and Hispanic-aligned English in C4, stringent filtering equates formal institutional English with "clean" data20. A representative language model must understand and generate the dialects, slang, syntax, and conversational structures of ordinary people to serve global populations effectively, rather than forcing users to interact through an artificially imposed academic register. Second, the retention of contextual and tacit knowledge relies on independent creators. Universities and government agencies produce theoretical and bureaucratic knowledge. However, tacit knowledge—how to troubleshoot a niche mechanical issue, emotional advice on navigating localized grief, or nuanced cultural critiques—is overwhelmingly produced by independent writers and community forums. When language models are filtered purely for "educational value" via models like Llama-326, this localized, lived expertise is discarded, rendering the model less capable of assisting with practical, everyday human challenges. Third, preventing an algorithmic monoculture necessitates disrupting the backlink hierarchy. If both open and closed models rely exclusively on the same top-heavy backlink graphs and the same major publishers, artificial intelligence will converge on an extremely narrow, homogenized worldview. Expanding the scope of inclusion to independent creators, while navigating proper attribution and consent mechanisms as advocated by the Data Provenance Initiative1, is imperative for building models capable of lateral, diverse, and robust reasoning.

Conclusion

The architecture of algorithmic knowledge is neither random nor entirely democratic. Through the mechanical realities of web-crawl graph centrality, the heavy-handed application of domain blocklists, and the new paradigm of LLM-driven "educational" filtering, modern pre-training datasets systematically over-index on institutional authority at the expense of ordinary people. While open-source corpora like FineWeb and Dolma represent monumental engineering achievements capable of processing trillions of tokens across decades of web history6, their reliance on domains like patents.google.com and open-access scientific journals creates a profound structural imbalance12. Simultaneously, the algorithmic erasure of minority dialects and the proprietary hoarding of community forum data threaten to bifurcate the AI ecosystem into two flawed paradigms: open models that speak with the sanitized voice of bureaucrats, and closed models built entirely upon the uncompensated conversational labor of the public. Correcting this imbalance requires a fundamental reevaluation of how the AI community defines and measures "quality" in human knowledge, ensuring that the diverse, unfiltered voices of independent writers and ordinary people are not permanently lost in the algorithmic filter.

Works cited

1. A Large Scale Audit of Dataset Licensing & Attribution in AI \- arXiv, https\://arxiv.org/html/2310.16787v3

2. A large-scale audit of dataset licensing and attribution in AI, https\://www\.researchgate.net/publication/383566266\_A\_large-scale\_audit\_of\_dataset\_licensing\_and\_attribution\_in\_AI

3. The Pile (dataset) \- Wikipedia, https\://en.wikipedia.org/wiki/The\_Pile\_(dataset)

4. EleutherAI/pile · Datasets at Hugging Face, https\://huggingface.co/datasets/EleutherAI/pile

5. EleutherAI/the-pile \- GitHub, https\://github.com/EleutherAI/the-pile

6. Dolma: an Open Corpus of Three Trillion Tokens for Language, https\://arxiv.org/abs/2402.00159

7. Dolma : an Open Corpus of Three Trillion Tokens for Language, https\://aclanthology.org/2024.acl-long.840.pdf

8. Ai2 Dolma: 3 trillion token open corpus for language model pretraining, https\://allenai.org/blog/dolma-3-trillion-tokens-open-llm-corpus-9a0ff4b8da64

9. FineWeb: decanting the web for the finest text data at scale, https\://huggingface.co/spaces/HuggingFaceFW/blogpost-fineweb-v1

10. FineWeb: HuggingFace's 15-trillion-token web corpus \- ZeroEntropy, https\://zeroentropy.dev/concepts/fineweb/

11. DOLMA, largest curated text dataset for training just dropped \- Reddit, https\://www\.reddit.com/r/LocalLLaMA/comments/15wq66q/dolma\_largest\_curated\_text\_dataset\_for\_training/

12. Papers Explained 97: Dolma \- Ritvik Rastogi \- Medium, https\://ritvik19.medium.com/papers-explained-97-dolma-a656169269cb

13. Google Bard AI \- What Sites Were Used To Train It?, https\://www\.searchenginejournal.com/google-bard-training-data/478941/

14. Documenting Large Webtext Corpora: A Case Study on the Colossal, https\://maartensap.com/pdfs/dodge2021documentingC4.pdf

15. A Case Study on the Colossal Clean Crawled Corpus \- Sites@Rutgers, https\://sites.rutgers.edu/critical-ai/wp-content/uploads/sites/586/2021/09/dodge2021documentingC4.pdf

16. Web Graphs \- Common Crawl, https\://commoncrawl.org/web-graphs

17. CrawlGraph: Backlink and domain-link data from Common, https\://publicapis.dev/resource/crawlgraph/c4ucbfip

18. RefinedWeb Dataset: Scalable Web Data for LMs \- Emergent Mind, https\://www\.emergentmind.com/topics/refinedweb-dataset

19. The RefinedWeb Dataset for Falcon LLM: Outperforming Curated, https\://proceedings.neurips.cc/paper\_files/paper/2023/file/fa3ed726cc5073b9c31e3e49a807789c-Paper-Datasets\_and\_Benchmarks.pdf

20. Documenting Large Webtext Corpora: A Case Study on the Colossal, https\://sh-tsang.medium.com/review-documenting-largewebtext-corpora-a-case-study-on-the-colossal-clean-crawled-corpus-0bcc6554e4b6

21. BRIDGING THE DATA PROVENANCE GAP ACROSS TEXT, https\://www\.dataprovenance.org/Multimodal\_Data\_Provenance.pdf

22. Papers Explained 174: FineWeb \- Ritvik Rastogi, https\://ritvik19.medium.com/papers-explained-174-fineweb-280bbc08068b

23. fineweb-edu \- Dataset details, https\://www\.modelscope.cn/datasets/AI-ModelScope/fineweb-edu

24. FineWeb-Edu: Quality Educational Web Data \- Emergent Mind, https\://www\.emergentmind.com/topics/fineweb-edu-dataset

25. Issue 12: FineWeb and FineWeb-Edu Reports, MAP-Neo Insights, https\://www\.philschmid.de/cloud-attention/issue-12

26. The FineWeb Datasets: Decanting the Web for the Finest Text Data, https\://arxiv.org/html/2406.17557v1

27. Edu-QuRating: Multi-Dimensional Educational Data Curation ... \- arXiv, https\://arxiv.org/html/2609.09425v1

28. FineWeb-Edu is actually nuts : r/LocalLLaMA \- Reddit, https\://www\.reddit.com/r/LocalLLaMA/comments/1dl1k61/finewebedu\_is\_actually\_nuts/

29. Publications \- Shayne Longpre, https\://www\.shaynelongpre.com/publications

30. Common Corpus: The Largest Collection of Ethical Data for LLM Pre, https\://arxiv.org/html/2506.01732v1

31. The Largest Collection of Ethical Data for LLM Pre-Training \- arXiv, https\://arxiv.org/html/2506.01732v3

32. HuggingFaceFW/fineweb-edu-classifier · Hugging Face, https\://huggingface.co/HuggingFaceFW/fineweb-edu-classifier

33. m-a-p/FineFineWeb · Datasets at Hugging Face, https\://huggingface.co/datasets/m-a-p/FineFineWeb