The Architecture of Preservation: Long-Form Human Authorship, Epistemic Diversity, and the Future of the AI Data Commons

Full report

On this page

The transition from a decentralized, human-authored web to an algorithmic ecosystem dominated by large language models (LLMs) represents a profound paradigm shift in how knowledge is preserved, transmitted, and synthesized. Original books, long-form essays, independent journals, personal memoirs, technical manuals, oral histories, and thoughtful community publications have historically served as the bedrock of the epistemic commons. These formats demand sustained cognitive attention, develop complex multi-hop arguments over tens of thousands of words, and preserve idiosyncratic local knowledge that defies broad statistical averaging. However, as the artificial intelligence industry aggressively harvests the internet to build trillion-token pretraining corpora, a critical divergence has emerged between the depth of human-authored long-form text and the homogenized, heavily filtered, and increasingly synthetic outputs of generative AI models. This exhaustive research report investigates the structural, technical, and sociocultural necessity of preserving long-form human writing. It provides a comprehensive literature review comparing human textual complexity against AI-generated approximations, analyzes the architectural composition and heuristic filtering of documented LLM pretraining datasets, and evaluates the severe limitations of current LLM context windows in processing complex narratives. Furthermore, it explores the legal and ethical frameworks necessary to protect human authorship amidst the mass scraping of the internet, culminating in a detailed publishing blueprint for authors and archivists who wish to voluntarily preserve their intellectual contributions for future open-source research while retaining robust legal rights.

Literature Review: The Epistemic Value of Long-Form Authorship and the Risks of Generative Homogenization

The academic literature across computational linguistics, sociology, and information theory provides a robust foundation for understanding why original long-form human writing is irreplaceable. This body of research highlights a growing tension between the nuanced, long-tail diversity of human thought and the center-weighted, popularity-driven mechanics of generative information systems.

Contextual Depth, Argument Complexity, and Viewpoint Diversity

Original books, technical manuals, and independent scholarly journals are characterized by their structural complexity. Human authors utilize recursive logic, extended analogies, and narrative continuity to build arguments that span vast contextual distances. By contrast, the proliferation of short-form digital posts and heavily AI-generated publications tends to favor immediate engagement over deep synthesis. The literature indicates that heavily AI-generated publications operate on probabilistic next-token prediction, which inherently biases the output toward the statistical center of the training data1. This probabilistic nature severely limits argument complexity; the models construct sentences that are statistically likely given the prompt, but they do not engage in the teleological planning required to sustain a novel, multi-layered thesis over a long-form essay. Furthermore, viewpoint diversity is fundamentally tied to the "long tail" of human knowledge. Outlier perspectives, dissenting academic theories, and localized cultural paradigms exist on the margins of statistical frequency. Because LLMs are designed to generate outputs that reflect the highest probability distribution of their training data, they naturally exclude this long-tail information, leading to a homogenization of thought where only the most broadly accepted or frequently repeated viewpoints are represented in the generated text1.

The Mathematics of Model Collapse

The theoretical limits of training generative models on synthetic or short-form data are mathematically formalized in the phenomenon known as "Model Collapse." Model collapse occurs when generative models are trained recursively on data generated by previous generations of models, leading to a rapid degradation in performance and variance3. Research establishes a "Double Scaling Law" for model collapse, demonstrating that the finite size of a training set induces an effective tail-cutting of the probability distribution2. When an LLM generates text, sampling algorithms (such as top\-[Figure omitted from source export] or top\-[Figure omitted from source export] inference) immediately chop the tail of the distribution, ensuring that less frequent outcomes—the specialized vocabulary and unique syntactic structures characteristic of deep human writing—are never generated2. Additionally, temperature scaling during generation causes "tail narrowing," where the distribution of generated concepts becomes progressively steeper and more centralized around common tropes2. In a multi-generational self-consuming loop, the entropy of the training dataset strictly decreases, and the textual distribution progressively collapses into compact clusters5. Consequently, AI-generated text is fundamentally incapable of introducing the novel paradigms, structural complexities, and unique lexical choices that characterize original human authorship. Over successive generations, the models forget the margins of human knowledge entirely, resulting in an epistemic flattening6.

Knowledge Collapse and the Erosion of the Epistemic Commons

Beyond the technical degradation of model weights lies the broader sociological threat of Knowledge Collapse. This is defined as the progressive narrowing over time of the set of information available to humans, alongside a concomitant narrowing in the perceived utility of diverse sets of information1. Human users are often drawn to popular, highly aggregated content to understand trending ideas, but overly favoring popular items leads to the neglect of unique, long-tail interests1. Because AI models are optimized to output the center of their training distribution, reliance on AI for writing, summarization, and search accelerates the degeneration of public knowledge toward an established, homogenized center1. In economic models of information cascades, if the cost of accessing the "center" via AI-generated summaries is significantly lower than the cost of discovering "tail knowledge" via primary source reading, public knowledge inevitably collapses1. Long-form human writing serves as the primary repository for this tail knowledge. These formats allow for the exploration of outlier concepts, the documentation of localized physical phenomena, and the articulation of highly specialized dissenting viewpoints. If these primary sources are overwhelmed by popularity-driven AI systems, the unique knowledge they contain will be permanently lost to future generations who rely exclusively on algorithmic information retrieval.

The Architecture of LLM Pretraining Corpora: Filtration and the Loss of Context

The capabilities of modern foundation models are intrinsically linked to the scale and quality of their pretraining datasets. However, the pursuit of "high-quality" text at a trillion-token scale necessitates automated filtering pipelines that inadvertently marginalize complex, long-form, and culturally specific human knowledge.

Corpus Composition and the Illusion of Comprehensiveness

Open-source and highly documented pretraining datasets, such as RedPajama, Dolma, and FineWeb, provide a transparent window into the curation practices of the AI industry. These corpora rely overwhelmingly on massive web scrapes, particularly Common Crawl, supplemented by academic papers, code repositories, and book collections7. While these datasets boast immense size—RedPajama encompasses over 100 billion text documents and 30 trillion tokens, while FineWeb comprises 15 terabytes of compressed data yielding approximately 3 trillion tokens—their representation of deep human knowledge is heavily skewed7. The "Books" category, historically vital for imparting complex reasoning and narrative continuity to language models, constitutes a surprisingly small fraction of the total token count. Foundational models have heavily utilized datasets like Books1, Books2, and the Toronto BookCorpus11. However, critical analyses of these book corpora reveal profound limitations. BookCorpus, for instance, consists of approximately 11,000 (with some estimates pointing to \~7,200 unique) self-published romance and fantasy novels scraped from platforms like Smashwords11. Consequently, its genre composition is non-representative and heavily biased, representing less than a fraction of one percent of all books ever published11. These corpora overwhelmingly favor Western, English-language texts while neglecting the vast majority of historical, technical, and localized literary works11. By contrast, initiatives like the Harvard Library Public Domain Corpus are attempting to rectify this by releasing nearly 1 million digitized books, including historical legal texts, casebooks, statutes, and treatises in diverse languages such as Czech, Icelandic, and Welsh, providing a much deeper repository of structured knowledge15.

Heuristic Filtering and the Eradication of the Long Tail

To refine raw web dumps into usable training data, dataset curators employ aggressive filtering, deduplication, and quality classification pipelines. While necessary to remove spam, boilerplate, and toxic content, these techniques systematically penalize specific styles of human writing. The FineWeb dataset utilizes a "FineScore" system based on internally developed heuristic filters to rank documents7. Similarly, the Dolma dataset, a 3-trillion token corpus used to train the OLMo models, utilizes a high-performance toolkit for unified filtering and mixing across domains8. The following table details common filtering mechanisms applied in state-of-the-art dataset curation and their corresponding negative impacts on original long-form human text:

As these filters select text according to criteria that are inherently ideological and standardized, the resulting corpora reflect a homogenized, "center-weighted" view of human knowledge9. Complex, idiosyncratic, and highly specialized original texts are frequently discarded as statistical anomalies, ensuring that models trained on these datasets lack the foundational context required to generate or understand deep, localized human reasoning.

Evaluating Contextual Depth: The Illusion of Infinite Context

A common technological counterargument to the necessity of pretraining on massive book datasets posits that as LLMs achieve massive context windows (ranging from 128,000 to over 1 million tokens), they will simply ingest entire books, manuals, and archives at runtime via Retrieval-Augmented Generation (RAG) or zero-shot prompting. However, rigorous empirical evaluations of long-context understanding reveal critical architectural failures in how current attention mechanisms process extensive human writing.

The Superficiality of "Needle In A Haystack" Evaluations

Initial claims of long-context mastery were heavily based on the "Needle-In-A-Haystack" (NIAH) test, which examines a model's ability to retrieve a single, explicitly stated fact (the "needle") hidden within a large block of unrelated distractor text (the "haystack")16. While top-tier models achieve near-perfect accuracy on vanilla single-needle tests, researchers have identified this task as a highly flawed proxy for true reading comprehension16. Standard NIAH tests evaluate only superficial associative recall16. Furthermore, inserting highly anomalous facts—such as a random UUID or a magic number within a book about San Francisco—creates a deep semantic heterogeneity16. This stark contrast makes the needle artificially easy for the model's attention mechanisms to identify, failing to replicate the nuanced difficulty of extracting relevant information from a cohesive, long-form narrative where all facts are topically related19.

RULER and BABILong: Exposing the Limits of Multi-Hop Reasoning

To address the severe shortcomings of the standard NIAH evaluation, the research community introduced more rigorous, multifaceted benchmarks such as RULER and BABILong. These benchmarks are designed to test behaviors beyond simple retrieval, focusing on the complex cognitive tasks required to process human-authored books and manuals16. The RULER benchmark significantly expands the evaluation typology, demonstrating that LLMs fail to maintain performance as task complexity and input length increase16. The benchmark evaluates models across several critical categories:

Similarly, the BABILong benchmark embeds complex reasoning tasks within natural, long-form documents sourced from books (specifically the PG19 corpus), allowing for the construction of tasks of almost arbitrary length20. The findings are stark: current LLMs do not efficiently utilize their full context windows20. When facts are embedded naturally within literary texts rather than stark noise, model performance falls significantly20. While single-fact retrieval might exceed the 85% satisfactory threshold, tasks requiring the synthesis of three or more supporting facts across a long document prove to be "extremely challenging to current LLMs, with the best scores falling below 80%" and frequently resulting in complete failure (accuracy below 30%)20. These evaluations underscore a critical reality regarding AI and human knowledge: Large Language Models are not "reading" long-form texts in the human sense. They cannot hold a complex, book-length thesis in working memory, track the nuanced evolution of a character across chapters, or synthesize a dense technical manual to troubleshoot a novel, multi-step mechanical failure. True long-form human writing possesses a hierarchical, recursive logic that current attention mechanisms simply fail to map accurately over long distances. Thus, reducing long-form works to short-form AI summaries inherently destroys the structural integrity of the knowledge itself.

Archival Preservation and the Necessity of Local Expertise

The algorithmic suppression of long-tail knowledge through heuristic filtering and context-window limitations is most acutely felt at the community level. Local expertise, municipal histories, and community archives contain high-fidelity data regarding human infrastructure, migration patterns, and cultural evolution that cannot be generalized from macro-level statistical data. Consider the historical preservation efforts in Cicero, Illinois, particularly concerning the Hawthorne Works plant. The Western Electric Hawthorne Works was a massive manufacturing facility that profoundly shaped the socioeconomic landscape of Cicero and greater Chicago, famously serving as the site of the foundational Hawthorne studies in industrial psychology and worker productivity21. The preservation of this rich industrial history relies heavily on localized, human-curated archives, such as the Local History Room at the Cicero Public Library, which maintains physical photo collections of women workers, specific plant histories, and community newspapers22. Furthermore, the Illinois Regional Archives Depository (IRAD) System systematically collects and preserves local government records of profound genealogical and historical value23. When global technology companies deploy crawlers to scrape the web for AI training data, they prioritize high-traffic, highly centralized domains (e.g., Wikipedia, Reddit, major news outlets). Micro-histories hosted on local historical society websites, scanned community newspapers, and independent blogs documenting specific industrial labor disputes or the stained glass history at Navy Pier are frequently overlooked21. Even when scraped, these specialized texts are often malformed during HTML parsing or outright discarded as "low quality" or low-entropy by algorithmic filters designed to isolate modern, conversational web text7. Oral histories and personal memoirs are equally vulnerable. These documents are characterized by non-standard grammar, localized idioms, and conversational pacing—traits that routinely trigger entropy and readability filters in LLM data pipelines7. By relying on LLMs trained predominantly on standardized, homogenized web text, society risks effectively deleting the digital memory of marginalized communities, indigenous populations, and hyper-local industrial histories. Preserving original long-form publications and thoughtfully supporting decentralized archival initiatives (such as those utilizing LOCKSS, IPFS, or WARC files for long-tail open knowledge preservation24) is the only robust defense against this form of algorithmic cultural amnesia.

The Crisis of Consent and the Tragedy of the Data Commons

The realization that generative AI models are heavily reliant on the uncompensated, uncredited, and unconsented ingestion of human labor has precipitated a profound legal and ethical crisis. As technology companies scrape the web indiscriminately, content creators, publishers, and independent authors have retaliated, leading to what AI researchers now term the "Tragedy of the Data Commons"25.

The Foreclosure of the Open Web

The Data Provenance Initiative (DPI), an exhaustive audit of 1,800 curated datasets widely used in AI training, has documented a rapid and systemic proliferation of restrictions on web crawlers26. In a single year (from April 2023 to April 2024), the percentage of restricted tokens from the most critical, high-quality domains jumped from less than 3% to between 20% and 33%26. Across the entire corpora of major datasets like C4, RefinedWeb, and Dolma, the absolute number of restricted tokens surged, with restrictions rising over 500% year-over-year26. Websites are increasingly modifying their robots.txt files and Terms of Service (ToS) to explicitly forbid data mining, crawling, and commercial AI training26. While this response is a natural and necessary defense mechanism for authors protecting their copyright, it carries severe collateral damage. The rapid foreclosure of the open web impacts not only commercial AI developers but also non-commercial academic researchers, linguists, and open-source data initiatives that rely on public datasets for scientific progress26. The internet is rapidly fracturing into walled gardens, making it exceedingly difficult to curate diverse, high-quality, legally compliant datasets that adequately represent the breadth of human knowledge.

The Fair Use Dispute and the Authors Guild

At the core of this data crisis is the high-stakes legal dispute over whether training AI on copyrighted works constitutes "fair use" under U.S. copyright law. AI developers and certain legal scholars argue that using copyrighted text to train a machine learning model is a highly transformative, non-expressive use that serves the progress of science29. They liken the process to a student reading a book to learn facts, styles, and syntactical rules without distributing the original book itself, arguing that AI's generative capability serves a fundamentally fair use purpose29. Conversely, rights holders, authors, and organizations like the Authors Guild strenuously reject the fair use defense31. When evaluating the four factors of fair use—specifically the purpose and character of the use, the nature of the copyrighted work, the amount and substantiality used, and the effect on the market—authors argue that AI developers fail on all counts32. Generative models require the wholesale copying of highly creative, expressive works (novels, memoirs, non-fiction) to function32. More critically, LLMs serve as direct market substitutes for original authors. Generative models frequently output regurgitated snippets, closely mimic an author's distinct voice, and allow users to generate competing works in the exact style of the original creator, causing immediate and severe market harm30. The Authors Guild emphasizes that AI-generated text is inherently not copyrightable because it lacks human authorship, and that injecting millions of synthetic books into the literary market fundamentally devalues the human profession33. To regain control over their intellectual property, the Authors Guild strongly advocates for a robust, consent-based licensing framework31. They propose that replacing the tech industry's unilateral reliance on fair use with a consensual licensing system gives authors the sovereign power to opt-out entirely or to demand fair compensation for the use of their data31. The Guild provides specific model contract clauses for trade publishing agreements to help authors protect their rights. These clauses include:

• Prohibiting the uploading of the author's personal information or manuscript to commercial AI without guardrails and explicit permission34.
• Prohibiting publishers from using AI to substantially edit a manuscript or generate audiobook narrations without consent34.
• Mandating specific subsidiary rights splits if the author agrees to license the work for AI training, Retrieval-Augmented Generation (RAG) (e.g., demanding 80-90% of the license revenue), AI-generated summaries, or "Chat with a Character" features34.

A Publishing Blueprint: Preserving Intellectual Contributions for the Open Future

For authors, academics, and archivists who wish to contribute their original, long-form human writing to the preservation of global knowledge—and who are willing to allow ethical, open-source AI researchers to utilize their work under highly specific, legally binding terms—traditional web publishing is insufficient. Text published on a standard blog without machine-readable metadata and explicit licensing parameters is either scraped indiscriminately by commercial giants or blocked entirely from academic view via blunt robots.txt directives. The following publishing blueprint outlines a comprehensive, technical methodology for publishing long-form texts such that they are permanently preserved, semantically discoverable by AI systems, and robustly protected through machine-readable consent protocols.

Phase 1: Semantic Structuring and Machine-Readable Metadata (Schema.org)

To ensure that the nuanced context of long-form writing is not lost in standard HTML boilerplate removal pipelines (such as trafilatura7), authors must embed structured data directly into their web hosting architecture. Schema.org, implemented via JSON-LD (JavaScript Object Notation for Linked Data), is the industry standard for translating human-readable web content into machine-readable knowledge graphs35. Search engines and AI crawlers heavily rely on JSON-LD because it decouples the data structure from the visual HTML content, making it syntactically clean, hierarchical, and immune to parsing errors that strip context37. Authors should implement the following specific Schema types:

1. CreativeWork / Article / Book Schema: Use these overarching schemas to define the core properties of the text. Critical fields for AI citation and proper provenance attribution include name, author, datePublished, publisher, and usageInfo (which can link to specific usage rights)38.

2. Organization / Person Schema: Establish the definitive entity graph of the author or publishing body. Utilizing the sameAs array within the Person schema allows AI systems to mathematically cross-reference the author's identity across platforms (e.g., ORCID, LinkedIn, academic repositories), building a stronger entity graph and increasing the confidence of AI citations38.

3. BreadcrumbList Schema: This schema reinforces the topical clusters and hierarchical structure of the website, providing the AI crawler with the exact contextual architecture of a complex essay or multi-chapter manual, ensuring the machine understands the relationship between disparate sections of the text38.

Phase 2: Asserting Rights via the TDM Reservation Protocol (TDMRep)

To prevent unauthorized scraping by commercial entities while simultaneously signaling a willingness to negotiate or grant licenses to open research initiatives, authors must implement the W3C Text and Data Mining Reservation Protocol (TDMRep)40. Developed by a W3C Community Group in response to Article 4 of the European Union's Directive on Copyright in the Digital Single Market (CDSM), the TDMRep provides a standardized, machine-readable method for rights holders to express their exact TDM permissions41. The protocol relies on two core properties:

• tdm-reservation: A boolean value where 1 indicates TDM rights are reserved (mining is prohibited without a specific license) and 0 indicates TDM rights are not reserved (mining is permitted)40.
• tdm-policy: An optional URL pointing to an ODRL (Open Digital Rights Language) policy file detailing the exact licensing terms and contact information for acquiring rights40.

Implementation Hierarchy: The TDMRep specification defines a clear priority order for discovering declarations44:

1. The .well-known File: Deploy a tdmrep.json file in the .well-known directory of the domain. This file operates similarly to robots.txt but provides granular, path-specific control. It declares a site-wide baseline reservation44.

2. HTTP Response Headers: For dynamic content or individual files, servers can return tdm-reservation: 1 and tdm-policy: \<url\> directly in the HTTP header, overriding the well-known file for that specific resource40.

3. HTML Meta Tags: Tags embedded in the document head provide page-level declarations for simple blogs or essays41.

4. Embedded PDF/EPUB Metadata: For long-form works distributed as downloadable files (technical manuals, memoirs), authors must embed the tdm:reservation and tdm:policy properties directly into the XMP metadata of the PDF or the package metadata of the EPUB. This guarantees that the rights declaration travels permanently with the document, even if it is uploaded to a third-party repository or shared via email44.

Phase 3: Archival Preservation and DOI Registration (Zenodo)

Because the open web is highly volatile and subject to domain expiration and link rot, ensuring the permanent historical preservation of original works requires depositing them in trusted, persistent digital archives. Zenodo, an open-access repository developed and maintained by CERN, is the premier platform for the long-tail preservation of scientific, technical, and intellectual output46. When an author publishes a record on Zenodo, the platform assigns a globally unique, persistent Digital Object Identifier (DOI)48. The Zenodo Archival Strategy:

• Cryptographic Permanence: Zenodo operates strictly according to archival best practices. Once a record is published, the files and the DOI cannot be modified48. If a manual or essay is updated, a new version is created with a new DOI, preserving the precise historical lineage of the text48.
• Machine-Readable Licensing: Authors can select exact Creative Commons licenses (e.g., CC BY 4.0, CC BY-NC 4.0) during deposit. Zenodo embeds the open-access status and the license directly into all exported metadata formats, ensuring that AI compliance auditing tools (such as the Data Provenance Explorer) can easily verify the legal status of the text50.
• Integration with Publishing Workflows: For independent journals and community publishers, plugins like the Zenodo DOI Sync for Open Journal Systems (OJS) allow for the automated, API-driven synchronization of metadata (authors, ORCIDs, formatted HTML abstracts) directly from the journal's backend to the CERN servers, eliminating human error in the archival process46.

Phase 4: Formatting for Ethical Machine Learning (Croissant & Fairly Trained)

If an author, independent journal, or community archivist actively desires their collection to be ingested by ethical, open-source AI projects (while excluding commercial exploitation), they must package the text in a standardized format that machine learning engineers can seamlessly integrate into data loaders. Croissant is a high-level metadata format for machine learning datasets, developed by the MLCommons Datasets Working Group53. Built as an extension of the schema.org/Dataset vocabulary, Croissant provides a unified, standardized way to describe a dataset's contents, structural arrangements, responsible AI (RAI) attributes, and legal provenance54. Architecting a Croissant Dataset:

1. Metadata Layer: Define the dataset's name, description, and strict licensing parameters. Crucially, Croissant integrates W3C PROV-O standards to capture machine-readable data provenance, allowing the dataset to carry an indelible, verifiable record of its human origins and copyright status55.

2. Resource Layer: Describe the individual files containing the raw text (e.g., Parquet, CSV, or JSON files) using FileObject or FileSet definitions, providing exact URLs and SHA-256 checksums to guarantee data integrity56.

3. Semantic Layer: Map the structural columns of the text data directly to ML semantics, ensuring that automated AI data loaders (like those utilized in PyTorch, TensorFlow, or Hugging Face) inherently understand which fields contain the core text, which contain the authorial metadata, and which delineate the publication date54.

By publishing a croissant.json file alongside a Zenodo DOI, authors make their work instantly discoverable via global registries like Google Dataset Search and the Hugging Face Hub58. The baked-in metadata ensures that the text can only be legally utilized by developers building "Fairly Trained" certified models—AI systems that have achieved L-Certification by proving they are trained entirely on licensed, public domain, or affirmatively consented data60.

Conclusion

The preservation of the depth, nuance, and long-tail diversity of human knowledge depends inextricably on the survival and archiving of original long-form human authorship. Books, deeply researched scholarly essays, independent technical manuals, and localized oral histories provide a level of structural complexity, contextual depth, and epistemic diversity that short-form social media posts and probabilistically generated AI summaries cannot replicate. The inherent limitations of current LLM architectures—demonstrated empirically by catastrophic multi-hop benchmarking failures on tests like RULER and BABILong, and theoretically by the mathematical inevitability of model and knowledge collapse—prove that generative AI cannot substitute for primary human reasoning; it can only average it. As the artificial intelligence industry scales its insatiable demand for data, the mass, unconsented scraping of the internet has triggered a defensive foreclosure of the digital commons, threatening the very foundations of open scientific research. To break this destructive cycle, the digital publishing and archiving ecosystem must evolve. By combining robust legal advocacy, such as the contract models championed by the Authors Guild, with precise technical protocols—including JSON-LD semantic structuring, W3C TDMRep rights reservations, persistent Zenodo DOI archiving, and Croissant ML packaging—authors can unequivocally protect their intellectual property. Concurrently, these frameworks empower creators to selectively and voluntarily contribute their original insights to the next generation of ethical, consensual artificial intelligence. Preserving the epistemic commons is no longer merely a matter of writing profoundly; it requires the deliberate, machine-readable architecting of the provenance of human thought.

Works cited

1. AI and the Problem of Knowledge Collapse \- arXiv, https\://arxiv.org/pdf/2404.03502

2. A Tale of Tails: Model Collapse as a Change of Scaling Laws \- arXiv, https\://arxiv.org/html/2402.07043v2

3. 1Introduction \- arXiv, https\://arxiv.org/html/2503.03150v2

4. Position: Model Collapse Does Not Mean What You Think \- arXiv, https\://arxiv.org/pdf/2503.03150

5. A Closer Look at Model Collapse: From a Generalization-to ... \- arXiv, https\://arxiv.org/html/2509.16499v2

6. A note on “AI models collapse when trained on recursively ... \- Medium, https\://medium.com/@aliborji/a-note-on-ai-models-collapse-when-trained-on-recursively-generated-data-5bee1de1ab87

7. FineWeb: High-Quality Web Text Dataset | PDF | Html \- Scribd, https\://www\.scribd.com/document/967506995/Fine-Web

8. dolma \- ModelScope, https\://modelscope.cn/datasets/modelscope/dolma

9. Dolma: an Open Corpus of Three Trillion Tokens for Language, https\://arxiv.org/html/2402.00159v2

10. RedPajama-Data-v2: an Open Dataset with 30 Trillion ... \- GitHub, https\://github.com/togethercomputer/RedPajama-Data

11. AI Training Datasets: the Books1+Books2 that Big AI eats for, https\://gregoreite.com/drilling-down-details-on-the-ai-training-datasets/

12. CS294-288 Pre-Training Data Curation \- Sewon Min, https\://www\.sewonmin.com/courses/cs294\_288/assets/slides/cs294-288-pre-training-data-curation.pdf

13. Overview of data used to train language models | Our Hometown, https\://our-hometown.com/overview-of-data-used-to-train-language-models/

14. A closer look at BookCorpus, a key dataset in machine learning, https\://news.ycombinator.com/item?id=37568847

15. Harvard Unveils a Million-Book Corpus for AI Training, https\://www\.deeplearning.ai/the-batch/harvard-unveils-a-million-book-corpus-for-ai-training

16. RULER: Benchmark to evaluate long-context modeling capabilities, https\://medium.com/@techsachin/ruler-benchmark-to-evaluate-long-context-modeling-capabilities-of-language-models-7eb13a269e36

17. What's the Real Context Size of YourLong-Context Language Models?, https\://arxiv.org/html/2404.06654v3

18. Evaluating Long-Context Question & Answer Systems \- Eugene Yan, https\://eugeneyan.com/writing/qa-evals/

19. Systematic Evaluation of Long-Context LLMs on Financial Concepts, https\://aclanthology.org/2024.emnlp-industry.88.pdf

20. BABILong: Testing the Limits of LLMs with Long Context Reasoning, https\://proceedings.neurips.cc/paper\_files/paper/2024/file/c0d62e70dbc659cc9bd44cbcf1cb652f-Paper-Datasets\_and\_Benchmarks\_Track.pdf

21. HOME\~INDEX, https\://drloihjournal.blogspot.com/p/index.html

22. Explore Cicero History, https\://cicerolibrary.org/explore-cicero-history/

23. Illinois \- BYU Library, http\://files.lib.byu.edu/family-history-library/research-outlines/US/Illinois.pdf

24. Fatcat Guide | Fatcat Bibliographic Database \- Internet Archive Scholar, https\://scholar.archive.org/fatcat/guide

25. Tragedy of the (data) commons \- The Stack Overflow Blog, https\://stackoverflow.blog/2024/10/25/tragedy-of-the-data-commons/

26. Consent in Crisis: The Rapid Decline of the AI Data Commons, https\://www\.dataprovenance.org/Consent\_in\_Crisis.pdf

27. The Data Provenance Initiative: A Large Scale Audit of Dataset, https\://www\.cs.cmu.edu/\~sherryw/assets/pubs/2023-data-provenance.pdf

28. Robert Mahari and Shayne Longpre: “Discit ergo est: Training Data, https\://www\.networklawreview.org/mahari-longpre-generative-ai/

29. Fair Use and the Origin of AI Training \- Houston Law Review, https\://houstonlawreview.org/article/147422-fair-use-and-the-origin-of-ai-training

30. Artificial Intelligence Impacts on Copyright Law \- RAND Corporation, https\://www\.rand.org/pubs/perspectives/PEA3243-1.html

31. AI Licensing: What Authors Should Know, https\://authorsguild.org/advocacy/artificial-intelligence/ai-licensing-what-authors-should-know/

32. AI Training and Copyright: The Fight for Creative Consent \- Identity.org, https\://www\.identity.org/ai-training-and-copyright-is-it-fair-use-or-infringement/

33. AI Best Practices for Authors, https\://authorsguild.org/resource/ai-best-practices-for-authors/

34. Authors Guild AI-Related Model Publishing Contract Clauses, https\://authorsguild.org/advocacy/artificial-intelligence/ai-model-clauses/

35. Schema.org \- Schema.org, https\://schema.org/

36. Organization of Schemas, https\://schema.org/docs/schemas.html

37. Top JSON-LD Schema for SEO Patterns Driving AI Search Visibility, https\://growthnatives.com/blogs/seo/top-json-ld-schema-patterns-for-ai-search-success/

38. Schema Markup for AI: The 5 Types ChatGPT Actually Crawls, https\://www\.answermaniac.ai/blog/schema-markup-for-ai-chatgpt-citations

39. Article \- Schema.org Type, https\://schema.org/Article

40. TDM Reservation Protocol (TDMRep) \- W3C, https\://www\.w3.org/community/reports/tdmrep/CG-FINAL-tdmrep-20240510/

41. TDM Reservation Protocol Ready for AI \- W3C, https\://www\.w3.org/2023/05/pbg-first-day-slides/TDM.pdf

42. Text and Data Mining Reservation Protocol Community Group | tdm, https\://w3c-cg.github.io/tdm-reservation-protocol/

43. Parsing TDMRep and AI.txt: Purpose-Based Scraping Controls, https\://www\.scrapingbee.com/blog/tdmrep-ai-txt-scraping-controls/

44. TDM Rights in PDF Documents & the TDMRep Protocol \- Mapsoft, https\://mapsoft.com/posts/tdmrep-pdf-mining-rights.html

45. What Is TDMRep and How Do You Implement It in 2026? \- Glippy, https\://www\.glippy.dev/glossary/tdmrep/

46. Why Zenodo DOI for Academic Journals? \- OJS-Services.com, https\://ojs-services.com/journals-publication-indexing/why-zenodo-doi-for-academic-journals/

47. Zenodo, a free and open platform for preserving and sharing, https\://zenodo.org/records/51902

48. About records \- Zenodo Help, https\://help.zenodo.org/docs/deposit/about-records/

49. Digital Object Identifier (DOI) \- Zenodo Help, https\://help.zenodo.org/docs/deposit/describe-records/reserve-doi/

50. Principles \- Zenodo, https\://about.zenodo.org/principles/

51. Datasets for Large Language Models: A Comprehensive Survey, https\://arxiv.org/html/2402.18041v1

52. Datasets for Large Language Models: A Comprehensive Survey, https\://assets-eu.researchsquare.com/files/rs-3996137/v1\_covered\_568fda26-897b-4241-8e70-5bf660c61345.pdf?c=1762186156

53. New Croissant Metadata Format helps Standardize ML Datasets, https\://mlcommons.org/2024/03/croissant\_metadata\_announce/

54. Croissant is a high-level format for machine learning ... \- GitHub, https\://github.com/mlcommons/croissant

55. Croissant \- MLCommons, https\://mlcommons.org/working-groups/data/croissant/

56. Croissant Format Specification \- MLCommons, https\://docs.mlcommons.org/croissant/docs/croissant-spec-1.1.html

57. metadata: add Croissant 1.0 ML dataset descriptor \- Hugging Face, https\://huggingface.co/datasets/SmartQHSE/hse-benchmarks-2026/commit/cf519695e1fdc18e93e85b739f001fd14352a182

58. Get Croissant metadata \- Hugging Face, https\://huggingface.co/docs/dataset-viewer/en/croissant

59. Croissant: A Metadata Format for ML-Ready Datasets \- arXiv, https\://arxiv.org/pdf/2403.19546

60. Mitsua/mitsua-likes \- Hugging Face, https\://huggingface.co/Mitsua/mitsua-likes

61. Towards Best Practices for Open Datasets for LLM Training \- arXiv, https\://arxiv.org/pdf/2501.08365

62. RaagaPay Launches Ethical AI Dataset for Hindustani Music, https\://www\.aimusicpreneur.com/ai-music-news/raagapay-hindustani-classical-music-ai-dataset/