R15\publishing-rights-tdm-machine-use

Full report

On this page

report.md

Executive Assessment

The intersection of publishing rights, text-and-data mining (TDM), and artificial intelligence model training has evolved into a highly fragmented, adversarial legal and technical landscape. As of September 4, 2026, the historical assumption that public web availability inherently constitutes a broad, implied license for machine ingestion has been decisively dismantled across multiple global jurisdictions. The comprehensive analysis indicates that an independent publisher’s intent to allow machine use—as articulated in the baseline policy DEC-004—is technically executable but legally precarious if the publisher fails to distinguish rigorously between materials it owns and third-party content embedded within its archives. Current doctrinal developments reveal a sharp bifurcation between indexing for conventional search and bulk ingestion for foundation model training. While standard web indexing continues to benefit from established implied licenses and fair use protections designed to facilitate information retrieval, generative artificial intelligence training faces severe judicial and statutory scrutiny. The United States Copyright Office (USCO) Part 3 Report on Generative AI Training, released in a pre-publication version in May 2025, alongside the landmark February 2025 federal summary judgment in Thomson Reuters v. ROSS Intelligence, firmly establishes that commercial AI training on copyrighted material cannot automatically rely on fair use defenses, particularly where the output poses a risk of market substitution1. Concurrently, the European Union has operationalized Article 53 of the Artificial Intelligence Act, legally binding general-purpose AI providers to respect machine-readable rights reservations executed under Article 4(3) of the Directive on Copyright in the Digital Single Market4. In the United Kingdom, statutory exemptions under Section 29A of the Copyright, Designs and Patents Act 1988 remain strictly confined to non-commercial research, rendering any commercial TDM explicitly reliant on proactive rightsholder licensing7. Technically, managing machine access requires a layered, highly specific approach. The Robots Exclusion Protocol (RFC 9309\) provides an established but legally non-binding mechanism for routing basic crawlers, whereas the W3C TDM Reservation Protocol (TDMRep) serves as a robust, legally recognized standard for asserting copyright reservations, particularly in the European Economic Area9. Furthermore, mere presence in massive web scraping corpora, such as Common Crawl, is mathematically and operationally insufficient to prove that a specific document was utilized in the final parametric weights of a specific foundation model, thus strongly supporting the operational evidence rule encapsulated in IC-CLAIM-01011. Ultimately, maintaining a broadly permissive stance (IC-CLAIM-012) requires explicit carve-outs for third-party rights, non-generative restrictions, and transparent machine-readable declarations. Without these qualifications, permissive indexing directives risk stripping the publisher of legal standing should a downstream model provider commercialize the aggregated intellectual property without attribution, compensation, or respect for the underlying rights chain.

Investigation Parameters and Boundaries

The research execution date is September 4, 2026\. The analytical framework governing this report evaluates the current statutory, judicial, and technical landscape surrounding the machine ingestion of public web content. The analysis isolates distinct legal acts—ranging from transient cache copying to neural network parameter adjustment—and maps them against the corresponding legal objects, including original prose, user-generated comments, AI-generated text, and third-party media. Jurisdictional analysis is rigidly bounded to the United States, the European Union, and the United Kingdom, reflecting the primary regulatory zones impacting global data ingestion and foundation model development. Evaluative weight is assigned primarily to enacted legislation, final agency rules or comprehensive agency reports, and dispositive court rulings. Public statements by AI providers regarding their crawler behavior or training practices are treated strictly as statements of corporate intent or public relations strategy, rather than binding legal fact or proof of underlying technical mechanisms. The evaluation of technical controls separates semantic network protocols from copyright enforcement mechanisms, analyzing the precise syntax and inherent limitations of robots directives and machine-readable JSON declarations. Throughout the analysis, the project’s baseline policies, DEC-004 (permissive public machine-use policy) and DEC-007 (public availability does not equate to model inclusion proof), serve as the comparative standards against which external legal realities are tested.

The Anatomy of Machine Use: Separating Legal Acts and Objects

The legal vulnerability of any machine-use policy lies in the failure to disaggregate the pipeline of artificial intelligence training into distinct technical acts, each triggering separate rights under copyright and database law. A blanket, undifferentiated permission for machine use is legally meaningless without specifying which exact technical acts the publisher authorizes. The initial act of access and crawling involves an HTTP request performed by an automated user agent, such as OpenAI's GPTBot, Anthropic's ClaudeBot, or Google's Google-Extended14. Under the Robots Exclusion Protocol, access is regulated by the host server's responses9. Access alone generally does not implicate copyright law, but bypassing authentication mechanisms or severe rate limits may implicate anti-hacking statutes or constitute a breach of terms of service. Following access, indexing and caching occur. Search engines and web scrapers create transient, intermediate copies of HTML content to parse semantic meaning and build search indices. Historically, in the United States, these intermediate copies for the purpose of directing human users back to the original source have been heavily protected under the doctrine of fair use17. Dataset construction represents a significantly different legal act. Aggregators capture raw HTML, strip markup, and compile the underlying text into massive, permanent archives. This requires making permanent, reproducible copies of the content. In the European Union, this act implicates the sui generis database right, in addition to standard reproduction rights, as it involves the extraction and re-utilization of a substantial part of the contents of a database3. As the data moves into retrieval-augmented generation systems, the legal complexities multiply. Systems that execute real-time searches and synthesize answers ingest the text, hold it in context memory, and generate derivative summaries, often reproducing extensive verbatim snippets19. This implicates both the fundamental reproduction right and the exclusive right to prepare derivative works. The most legally contentious act is model training, which encompasses both pre-training and fine-tuning phases. Training involves passing tokenized text through a neural network architecture to adjust billions of parametric weights based on statistical relationships. The United States Copyright Office explicitly noted in its May 2025 report that artificial intelligence learning is fundamentally different from human learning in ways that are deeply material to copyright analysis, primarily because it relies on bulk, literal, and non-transient copying to extract statistical correlations3. Finally, model distribution and output generation implicate copyright only if the generated text is substantially similar to the original ingested work, triggering infringement of the reproduction or derivative works rights21. A publisher does not hold uniform rights across all content hosted on a public canonical URL. Original human prose is fully protected by copyright, and the publisher, if holding exclusive rights, may freely license this for text-and-data mining or training. Conversely, content generated entirely by an artificial intelligence lacks human authorship and is ineligible for copyright protection, as reaffirmed by the United States Copyright Office Part 2 Report in January 202523. A publisher cannot logically restrict or license AI-generated research outputs, as they reside firmly in the public domain. Furthermore, publishers frequently incorporate third-party content under fair use or specific limited licenses. A publisher cannot unilaterally re-license a third-party photograph, a proprietary dataset, or a quoted academic paper for commercial AI training25.

Auditing the Rights Chain: Publisher Authority and Third-Party Limitations

To execute the permissive stance outlined in IC-CLAIM-012, the publisher must audit the rights chain to ensure it possesses the actual legal authority to grant such permissions. Relying on open licensing frameworks like Creative Commons requires precise adherence to version semantics and a deep understanding of contractual boundaries. The Creative Commons Attribution 4.0 International license is frequently deployed by open-access publishers. This license explicitly permits text-and-data mining and commercial artificial intelligence training, provided that downstream users strictly comply with the attribution requirement27. However, Section 8(c) of the legal code explicitly states that the license does not grant trademark rights, patent rights, or the rights of third parties, such as privacy, publicity, or underlying third-party copyrights26. If a canonical research report incorporates a proprietary diagram under fair use, applying a blanket open license to the HTML page does not legally extend to the diagram. A web crawler scraping the entire document object model captures the diagram, and the subsequent developer assumes the entire artifact is openly licensed. This breaks the rights chain and exposes the original publisher to secondary liability or breach of contract claims. Furthermore, if a publisher applies a non-commercial restriction, commercial foundation model providers are expressly prohibited from utilizing the corpus for training29. To safely maintain a permissive policy, an organization must conduct a rigorous third-party-rights assessment.

Cross-Jurisdictional Doctrinal Framework

The legal authority to scrape, mine, and train on public web data is intensely territorial. The assumption of a borderless digital commons is a legal fiction that collapses upon exposure to national regulatory regimes. In the United States, technology firms have historically defended mass ingestion under the doctrine of fair use, relying heavily on earlier precedents involving search engine indexing. However, doctrinal momentum has shifted dramatically. The release of the United States Copyright Office Part 3 Report on Generative AI Training in May 2025 represents a critical inflection point. The Office affirmed that existing copyright principles apply to training and noted that generative training involves massive, non-transient copying designed to replicate human expression3. Analyzing the fourth fair use factor—the effect on the potential market—the Office noted that where models directly substitute for the ingested works, a fair use finding is highly unlikely20. This agency guidance was paralleled by the February 2025 summary judgment in Thomson Reuters Enterprise Centre GmbH v. Ross Intelligence Inc. The federal district court explicitly rejected the defendant's fair use defense for utilizing copyrighted headnotes to train a legal tool, establishing that tech companies can no longer assume publicly viewable content is immune to copyright enforcement when utilized for competitive training1. The European Union has codified specific text-and-data mining exceptions, rejecting the flexible, ambiguous approach of American fair use. Under the 2019 Directive on Copyright in the Digital Single Market, two vital articles govern machine use. Article 3 provides a mandatory exception for mining for the purposes of scientific research by research organizations, which cannot be overridden by contract6. Article 4 provides a broader exception for commercial mining, unless the rightsholder has expressly reserved their rights in an appropriate manner, such as machine-readable means for online content34. This framework was cemented by Article 53 of the comprehensive Artificial Intelligence Act. Article 53 legally obligates providers of general-purpose models to implement a policy to comply with European copyright law, specifically requiring them to identify and respect the machine-readable opt-outs declared under Article 4(3) of the Directive4. Crucially, the legislation prohibits providers from bypassing these obligations by releasing their models under open-source licenses; the exemption for open-source models does not extend to the copyright compliance and training data summary requirements5. The United Kingdom provides one of the most restrictive environments for commercial training among major economies. Section 29A of the Copyright, Designs and Patents Act 1988 allows text-and-data analysis only for the sole purpose of research for a non-commercial purpose by a person who has lawful access7. The law explicitly states that copies made cannot be transferred or used for any other purpose. In 2023, the government proposed broadening this section to allow commercial mining, but after severe backlash from the creative industries, the proposal was completely withdrawn8. Consequently, any commercial foundation model training utilizing material hosted in the United Kingdom without an explicit license constitutes copyright infringement38.

Doctrinal Uncertainty and Active Litigation

While the Thomson Reuters judgment signals a profound shift against fair use for training, the broader application to massive generative models remains mired in active multidistrict litigation, generating substantial doctrinal uncertainty. In the high-profile New York Times v. Microsoft and OpenAI docket, the federal district judge denied OpenAI's motions to dismiss in early 2025, keeping the core claims of direct infringement alive39. The Times argues that the models act as direct market substitutes, generating verbatim output of paywalled journalism41. However, the landscape was deeply complicated in September 2026 when the United States Department of Justice intervened, filing a statement of interest arguing that finding against OpenAI could harm national security and economic interests, urging the court to consider the broader utility of artificial intelligence41. This unprecedented intervention injects severe geopolitical and administrative uncertainty into the judicial process, leaving the final copyright status entirely unresolved. Conversely, courts have shown deep skepticism toward peripheral claims brought by individual creators. In the consolidated Kadrey v. Meta Platforms, Inc. and Andersen v. Stability AI litigations, summary judgments were issued in 2025 favoring the developers on specific counts. In Kadrey, the court ruled for Meta regarding the ingestion of books from shadow libraries because the plaintiffs failed to provide concrete, non-speculative evidence of market displacement43. In Andersen, the court ruled for Stability AI regarding violations of the Digital Millennium Copyright Act Section 1202(a), determining that a generic open-source license does not suggest a false endorsement or removal of specific copyright management information sufficient to trigger the statute45. These mixed outcomes indicate that while direct infringement claims regarding specific, provable training datasets hold water, broader class-action theories relying on general market displacement or metadata removal face exceptionally high evidentiary hurdles.

Distinguishing Technical Controls from Legal Permission

A fundamental error in modern digital governance is conflating a technical crawler directive with a legally binding copyright license. The protocols that guide automated agents were never designed to express complex intellectual property policies. Published as a formal standard in September 2022, the Robots Exclusion Protocol (RFC 9309\) formalizes the syntax originally developed in 19949. The protocol allows site owners to define which automated clients may fetch content. However, RFC 9309 is purely a technical request. It possesses no built-in enforcement mechanism, and any client can technically ignore it47. More critically, the protocol cannot express purpose. A publisher cannot use these directives to state that fetching a page for a search index is permitted, but fetching it for neural network training is forbidden48. Because of this severe limitation, vendors have introduced specialized user agents to allow webmasters to filter by purpose at the bot level. OpenAI utilizes GPTBot specifically for training data collection, while maintaining OAI-SearchBot for search indexing15. Anthropic similarly relies on ClaudeBot for model training16. Google introduced Google-Extended in September 2023 not as a separate HTTP request string, but as a control mechanism within the configuration file to prevent the use of content for improving Gemini and Vertex AI models, without impacting standard search indexing18. While a publisher can configure these directives to block training, the directives do not constitute a legal contract. If a non-compliant scraper ignores the file, the publisher's only recourse is to rely on underlying copyright law or computer fraud statutes. To bridge the massive gap between technical signals and the legal requirements of European law, the World Wide Web Consortium finalized the Text and Data Mining Reservation Protocol (TDMRep) in May 202410. This protocol provides a legally robust, machine-readable mechanism for rightsholders to declare their preferences. It operates through a site-wide JSON file hosted at a well-known directory, containing an array of path locations and a boolean reservation value. A value of 1 indicates that rights are reserved. Furthermore, the protocol allows for dynamic HTTP response headers served with individual resources. The protocol dictates that HTTP headers supersede the site-wide JSON file, allowing granular, asset-level control10. Implementing this standard translates the publisher's intent into a legally recognizable format under European law, binding providers under Article 53 of the Artificial Intelligence Act.

State of Permission and Configuration

Foundation Model Training and Evidentiary Standards

Evaluating the baseline claim IC-CLAIM-010 requires analyzing the rigorous mechanics of dataset construction. The claim posits that public accessibility or inclusion in a crawler archive is insufficient evidence that a specific foundation model trained on a specific document. The empirical evidence thoroughly supports this proposition. The vast majority of open-weight and proprietary language models utilize massive unstructured web scrapes, predominantly Common Crawl, as foundational data11. However, raw Common Crawl data is entirely unusable for high-quality language modeling due to extreme noise, duplicate content, and formatting artifacts. Data engineers pass raw archives through extensive, destructive processing pipelines to create refined datasets like C4, RefinedWeb, or FineWeb12. These pipelines utilize tools like FastText for strict language identification, discard pages with insufficient text-to-code ratios, and employ aggressive deduplication algorithms utilizing MinHash techniques13. A document successfully fetched by a crawler has a high statistical probability of being permanently discarded during these rigorous cleaning phases. Therefore, proving that a specific iteration of a model ingested a specific research report requires substantially more than a server log showing a visit from a known crawler. It requires a dataset manifest, an exact regurgitation of the text demonstrating memorization, or a direct provider disclosure3. Thus, DEC-007 is a fundamentally sound operational evidence rule.

Evaluating Policy Options and Tradeoffs

The current baseline policy intends public research to remain eligible for search, text-and-data mining, and training. Maintaining or altering this policy presents distinct, complex tradeoffs that balance discoverability against intellectual property control. The broadly permissive alternative, reflecting the current baseline, involves publishing content with standard HTML, explicitly allowing all agents, and issuing an open license without non-commercial restrictions. This approach maximizes discoverability, contributes to the development of open scientific models, and aligns with an open-access ethos. However, the publisher totally loses control over downstream commercialization. There is no technical mechanism to enforce attribution once the text is dissolved into neural weights. Furthermore, the publisher risks severe secondary liability if they inadvertently license third-party media embedded in their reports without possessing the requisite sub-licensing rights. The strict opt-out alternative involves reserving commercial training rights entirely. This mechanism requires blocking training crawlers, implementing machine-readable reservations, and attaching restrictive licenses. This preserves the publisher's intellectual property, retains leverage for potential licensing negotiations, and ensures absolute compliance with European regulatory standards. Conversely, this approach fragments the open web, reduces the presence of high-quality analytical content in future outputs, and incurs substantial administrative overhead to monitor and maintain dynamic crawler blocking lists. A mixed policy provides the most nuanced approach, permitting text ingestion while protecting specific assets. This involves allowing crawlers but deploying granular HTTP headers to signal legal reservations specifically for PDFs, image directories, and user-comment sections. This solves the third-party rights issue by legally segmenting the data, allowing training on owned intellectual property while protecting un-licensable assets. The primary drawback is extreme technical complexity. It requires configuring server-level header injections and relies heavily on providers accurately parsing the standard dynamically, which remains inconsistently implemented across the industry.

The Strongest Countercase to Permissive Ingestion

The strongest argument against maintaining a permissive machine-use policy lies not strictly in copyright theory, but in the evolving economic mechanics of market substitution and traffic destruction. While a publisher may philosophically support open-source model development, the reality of the contemporary digital ecosystem is that generative engines are actively replacing, rather than augmenting, traditional search referral paradigms. Industry analysis from September 2026 highlights this dynamic starkly. Following the widespread deployment of generative search summaries, leading digital publishers experienced an average drop of roughly one-fifth of their daily page views over a single year19. Furthermore, analytical reports indicate that while artificial intelligence chatbots absorb immense quantities of high-value reporting, they refer statistically insignificant traffic back to the original creators—accounting for less than one percent of total visits in specific tracked markets19. Simultaneously, the enterprise value of the models trained on this uncompensated data is skyrocketing, evidenced by the unprecedented demand for forward-deployed artificial intelligence engineers tasked with integrating these models into corporate infrastructure, commanding immense salaries52. In this economic context, a broadly permissive policy essentially subsidizes the creation of closed, highly profitable enterprise systems that systematically erode the publisher's independent distribution channels. The strategic trade-off is severe: remaining open to crawlers may preserve a nominal degree of discoverability within generative outputs, but it guarantees the uncompensated extraction of the publisher's core value.

Candidate Public Policy Language

If the organization chooses to maintain its permissive intent while mitigating third-party liability and addressing the complexities of the current legal environment, the policy language must be precise, machine-readable, and explicit about its absolute limitations. The following proposed public wording requires an explicit operator decision prior to adoption and deployment.

Machine-Use and Computational Analysis Policy

We support the advancement of computational research and algorithmic transparency. Except where expressly noted, the original prose of our canonical public research reports is licensed under the Creative Commons Attribution 4.0 International license.

Machine-Readable Permissions:

We permit automated indexing, retrieval-augmented generation, and text-and-data mining, including for commercial model training, strictly limited to the original text of our canonical HTML reports. This permission is legally declared via a tdm-reservation: 0 configuration in our server architecture.

Important Qualifications and Limitations:

1. Third-Party Rights: This permission does not extend to embedded photographs, third-party block quotations, datasets, or any material marked with a separate copyright notice. You must independently secure rights for these assets; they are explicitly reserved.
2. Private Paths: Internal operational memory, drafts, and administrative directories are strictly excluded from all forms of machine ingestion, as denoted in our exclusion protocols. Bypassing these controls constitutes a violation of our terms of access.
3. No Endorsement: Ingestion of our material does not constitute an endorsement of the resulting model, its architecture, or its subsequent outputs.

For automated compliance and granular asset exclusions, please refer to our machine-readable manifest hosted in the well-known directory.

Claim Impact Assessment

The comprehensive analysis generally supports the underlying intent of the baseline claims but necessitates vital, immediate qualifications regarding third-party rights and legal enforcement mechanisms. Regarding IC-CLAIM-012, which asserts that the current public machine-use policy is permissive, the research confirms that the policy intent can remain intact. However, relying solely on text-based directives or general open licenses without granular machine-readable reservations exposes the project to significant secondary liability regarding third-party content. The evidence state must be updated to reflect that while the intent is verified, the execution requires structural updates—specifically the implementation of the W3C standard—to be legally defensible, particularly in the European Union. Regarding IC-CLAIM-010, which asserts that public availability does not prove model-training inclusion, the research unequivocally validates the claim. Dataset engineering pipelines mathematically demonstrate that crawler access is profoundly disconnected from final parametric weight integration. The operational evidence rule remains robust and entirely necessary.

Best Next Research Action

Execute a comprehensive technical audit of the web server infrastructure to determine the precise engineering feasibility of dynamically injecting W3C Text and Data Mining Reservation Protocol HTTP headers (tdm-reservation) at the individual object level, prioritizing the absolute isolation of third-party images and embedded documents from the permissive HTML prose.

sources.json

JSON \[ { "source\_id": "R15-S001", "matched\_ic\_source\_id": null, "title": "Copyright and Artificial Intelligence, Part 3: Generative AI Training (Pre-Publication Version)", "authors": \["U.S. Copyright Office"\], "issuing\_institution": "U.S. Copyright Office", "document\_type": "government\_report", "canonical\_url": "https://www.copyright.gov/ai/Copyright-and-Artificial-Intelligence-Part-3-Generative-AI-Training-Report-Pre-Publication-Version.pdf", "retrieved\_url": "https://www.copyright.gov/ai/Copyright-and-Artificial-Intelligence-Part-3-Generative-AI-Training-Report-Pre-Publication-Version.pdf", "publication\_date": "2025-05-09", "version\_date": "2025-05-09", "effective\_date": "2025-05-09", "accessed\_at": "2026-09-04T21:41:45Z", "jurisdiction": "US", "legal\_or\_policy\_status": "official\_guidance", "publication\_status": "published", "host\_status": "available", "review\_scope": "full\_text\_review", "reviewed\_passages": "Analysis of fair use factor four; definition of generative training as material to copyright analysis.", "supported\_proposition": "Generative AI training that acts as a market substitute for ingested works is unlikely to qualify as fair use.", "important\_limitation": "Pre-publication draft; reflects agency interpretation rather than binding judicial precedent.", "claim\_ids": \["IC-CLAIM-012"\], "evidence\_lineage": \["direct\_primary"\], "snapshot\_path": null, "sha256": null, "missingness\_notes": "Hash computation unavailable in current environment." }, { "source\_id": "R15-S002", "matched\_ic\_source\_id": null, "title": "Thomson Reuters Enterprise Centre GmbH v. Ross Intelligence Inc. Summary Judgment", "authors": \["Judge Stephanos Bibas"\], "issuing\_institution": "U.S. District Court for the District of Delaware", "document\_type": "judicial\_opinion", "canonical\_url": "https://www.ded.uscourts.gov/sites/ded/files/opinions/20-613\_5.pdf", "retrieved\_url": "https://www.ded.uscourts.gov/sites/ded/files/opinions/20-613\_5.pdf", "publication\_date": "2025-02-11", "version\_date": "2025-02-11", "effective\_date": "2025-02-11", "accessed\_at": "2026-09-04T21:41:45Z", "jurisdiction": "US", "legal\_or\_policy\_status": "binding\_holding", "publication\_status": "published", "host\_status": "available", "review\_scope": "full\_text\_review", "reviewed\_passages": "Summary judgment rejection of fair use defense for AI training.", "supported\_proposition": "Public availability of data does not inherently grant a fair use license for commercial AI training.", "important\_limitation": "Applies specifically to non-generative legal AI utilizing direct competitor data; appeals pending.", "claim\_ids": \["IC-CLAIM-012"\], "evidence\_lineage": \["direct\_primary"\], "snapshot\_path": null, "sha256": null, "missingness\_notes": "Hash computation unavailable in current environment." }, { "source\_id": "R15-S003", "matched\_ic\_source\_id": null, "title": "Directive (EU) 2019/790 on Copyright in the Digital Single Market", "authors": \["European Parliament", "Council of the European Union"\], "issuing\_institution": "European Union", "document\_type": "statute", "canonical\_url": "https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=CELEX:32019L0790", "retrieved\_url": "https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:32019L0790", "publication\_date": "2019-04-17", "version\_date": "2019-04-17", "effective\_date": "2021-06-07", "accessed\_at": "2026-09-04T21:41:45Z", "jurisdiction": "EU", "legal\_or\_policy\_status": "enacted\_law", "publication\_status": "published", "host\_status": "available", "review\_scope": "targeted\_review", "reviewed\_passages": "Article 4(3) and Recital 18 concerning machine-readable opt-outs.", "supported\_proposition": "Commercial TDM is permitted unless rightsholders expressly reserve their rights via machine-readable formats.", "important\_limitation": "Requires specific member state implementation which may vary in procedural enforcement.", "claim\_ids": \["IC-CLAIM-012"\], "evidence\_lineage": \["direct\_primary"\], "snapshot\_path": null, "sha256": null, "missingness\_notes": "Hash computation unavailable in current environment." }, { "source\_id": "R15-S004", "matched\_ic\_source\_id": null, "title": "Guidelines on the Scope of Obligations for General-Purpose AI Models under Article 53", "authors": \["AI Office"\], "issuing\_institution": "European Commission", "document\_type": "official\_guidance", "canonical\_url": "https://ai-act-service-desk.ec.europa.eu/en/ai-act/article-53", "retrieved\_url": "https://ai-act-service-desk.ec.europa.eu/en/ai-act/article-53", "publication\_date": "2025-07-01", "version\_date": "2025-07-01", "effective\_date": "2025-07-01", "accessed\_at": "2026-09-04T21:41:45Z", "jurisdiction": "EU", "legal\_or\_policy\_status": "binding\_regulation", "publication\_status": "published", "host\_status": "available", "review\_scope": "targeted\_review", "reviewed\_passages": "Obligations for GPAI models regarding copyright compliance and open-source exemptions.", "supported\_proposition": "GPAI providers must respect Article 4(3) opt-outs using state-of-the-art tech, regardless of open-source model status.", "important\_limitation": "Enforcement mechanisms rely on adherence to Codes of Practice until harmonized standards are published.", "claim\_ids": \["IC-CLAIM-012"\], "evidence\_lineage": \["direct\_primary"\], "snapshot\_path": null, "sha256": null, "missingness\_notes": "Hash computation unavailable in current environment." }, { "source\_id": "R15-S005", "matched\_ic\_source\_id": null, "title": "Copyright, Designs and Patents Act 1988, Section 29A", "authors": \["UK Parliament"\], "issuing\_institution": "UK Parliament", "document\_type": "statute", "canonical\_url": "https://www.legislation.gov.uk/ukpga/1988/48/section/29A", "retrieved\_url": "https://www.legislation.gov.uk/ukpga/1988/48/section/29A", "publication\_date": "1988-11-15", "version\_date": "2014-06-01", "effective\_date": "2014-06-01", "accessed\_at": "2026-09-04T21:41:45Z", "jurisdiction": "UK", "legal\_or\_policy\_status": "enacted\_law", "publication\_status": "published", "host\_status": "available", "review\_scope": "targeted\_review", "reviewed\_passages": "Section 29A(1)(a) and (2).", "supported\_proposition": "UK law strictly limits TDM exceptions to non-commercial research purposes only.", "important\_limitation": "Statute wording focuses on computational analysis; enforcement against cross-border crawling remains highly complex.", "claim\_ids": \["IC-CLAIM-012"\], "evidence\_lineage": \["direct\_primary"\], "snapshot\_path": null, "sha256": null, "missingness\_notes": "Hash computation unavailable in current environment." }, { "source\_id": "R15-S006", "matched\_ic\_source\_id": null, "title": "TDM Reservation Protocol (TDMRep)", "authors": \["W3C Text and Data Mining Reservation Protocol Community Group"\], "issuing\_institution": "World Wide Web Consortium (W3C)", "document\_type": "technical\_standard", "canonical\_url": "https://www.w3.org/community/reports/tdmrep/CG-FINAL-tdmrep-20240510/", "retrieved\_url": "https://www.w3.org/community/reports/tdmrep/CG-FINAL-tdmrep-20240510/", "publication\_date": "2024-05-10", "version\_date": "2024-05-10", "effective\_date": "2024-05-10", "accessed\_at": "2026-09-04T21:41:45Z", "jurisdiction": "Global", "legal\_or\_policy\_status": "industry\_standard", "publication\_status": "published", "host\_status": "available", "review\_scope": "full\_text\_review", "reviewed\_passages": "Well-known JSON structure and HTTP response headers parsing rules.", "supported\_proposition": "Provides a legally recognizable machine-readable mechanism for declaring TDM opt-outs.", "important\_limitation": "Requires adoption and accurate parsing by external crawler operators; it is not inherently a network-level transmission block.", "claim\_ids": \["IC-CLAIM-012"\], "evidence\_lineage": \["direct\_primary"\], "snapshot\_path": null, "sha256": null, "missingness\_notes": "Hash computation unavailable in current environment." }, { "source\_id": "R15-S007", "matched\_ic\_source\_id": null, "title": "Creative Commons Attribution 4.0 International Public License", "authors": \["Creative Commons"\], "issuing\_institution": "Creative Commons", "document\_type": "legal\_license", "canonical\_url": "https://creativecommons.org/licenses/by/4.0/legalcode", "retrieved\_url": "https://creativecommons.org/licenses/by/4.0/legalcode", "publication\_date": "2013-11-25", "version\_date": "4.0", "effective\_date": "2013-11-25", "accessed\_at": "2026-09-04T21:41:45Z", "jurisdiction": "Global", "legal\_or\_policy\_status": "contractual\_terms", "publication\_status": "published", "host\_status": "available", "review\_scope": "targeted\_review", "reviewed\_passages": "Section 8(c) regarding third-party rights limitations.", "supported\_proposition": "CC BY 4.0 licenses explicitly do not cover third-party copyright, privacy, or publicity rights embedded within a licensed work.", "important\_limitation": "The effectiveness of the license relies entirely on the licensor actually holding the rights they are attempting to grant.", "claim\_ids": \["IC-CLAIM-012"\], "evidence\_lineage": \["direct\_primary"\], "snapshot\_path": null, "sha256": null, "missingness\_notes": "Hash computation unavailable in current environment." } \]

reviewed-source-notes.md

• R15-S001 (USCO Part 3 Report, May 2025): Supports the specific proposition that generative AI training constitutes massive, literal copying that is legally distinct from human reading, and may fail fair use if the outputs act as a market substitute. Important Limitation: This document is a pre-publication agency report representing policy interpretation, not binding judicial precedent.
• R15-S002 (Thomson Reuters v. ROSS, Feb 2025): Supports the proposition that public availability does not inherently permit commercial scraping and training under fair use. The judge explicitly rejected the fair use defense on summary judgment. Important Limitation: The case focuses tightly on a non-generative AI system acting as a direct competitor in a narrow legal search market; appeals are highly likely.
• R15-S003 (EU CDSM Directive Art 4): Supports the proposition that commercial TDM is globally segmented, being explicitly allowed in the European Union unless a machine-readable opt-out is demonstrably present. Important Limitation: Procedural enforcement is deeply fragmented across individual member state legislative implementations.
• R15-S004 (EU AI Act Article 53 Guidance): Supports the proposition that general-purpose AI providers must respect CDSM Article 4(3) opt-outs using state-of-the-art technologies and cannot bypass these obligations by releasing open-source models. Important Limitation: Compliance hinges heavily on adherence to evolving, industry-drafted Codes of Practice until formal harmonized technical standards are fully published.
• R15-S005 (UK CDPA Section 29A): Supports the proposition that commercial TDM is severely restricted in the United Kingdom, relegated solely to non-commercial research purposes following the withdrawal of proposed expansions in 2023\. Important Limitation: Enforcement against foreign-based crawlers scraping UK-hosted domains remains a complex jurisdictional challenge.
• R15-S006 (W3C TDMRep Standard, May 2024): Supports the proposition that publishers can utilize well-known JSON files and HTTP headers to assert legally enforceable rights reservations under EU law. Important Limitation: As a semantic signaling protocol, it requires voluntary compliance by the AI developer or external legal intervention; it does not block the HTTP request technically at the network edge.
• R15-S007 (CC BY 4.0 Legal Code): Supports the proposition that while an open license permits TDM, Section 8(c) explicitly excludes third-party rights, exposing permissive publishers to massive secondary liability if they host un-cleared embedded media. Important Limitation: Determining which specific assets on a complex document object model belong to the licensor versus a third party requires complex technical tagging, such as granular HTTP headers, to be legible to machines.

claim-effects.json

JSON \[ { "claim\_id": "IC-CLAIM-012", "baseline\_evidence\_state": "verified\_project\_policy", "baseline\_adoption\_state": "adopted\_policy", "recommended\_evidence\_state": "supported\_with\_qualification", "recommended\_adoption\_state": "adopted\_policy", "evidence\_effects": \[ "new\_direct/official\_evidence", "contrary\_evidence", "methodological\_criticism" \], "source\_ids": \["R15-S001", "R15-S003", "R15-S006", "R15-S007"\], "reason": "While the overarching permissive intent is technically executable, rapid legal developments—specifically the EU AI Act Article 53 and federal pushback against fair use—require granular protection of third-party assets to avoid severe liability. Broad permission without technical boundaries, such as TDMRep HTTP headers, is methodologically flawed and legally dangerous.", "strongest\_remaining\_objection": "Permissive training policies fail entirely to protect embedded third-party media, strip the publisher of future licensing leverage, and fundamentally subsidize enterprise architectures that destroy traditional search traffic referral loops.", "what\_would\_change": "Implementing the W3C TDMRep standard with granular HTTP headers, specifically issuing tdm-reservation: 0 for owned prose, and tdm-reservation: 1 for embedded media and datasets.", "proposed\_public\_wording": "We support computational research. We permit automated indexing and text-and-data mining strictly on our original HTML prose, legally declared via our TDMRep configuration. This permission explicitly excludes third-party media, datasets, and internal directories. Ingestion does not constitute an endorsement." }, { "claim\_id": "IC-CLAIM-010", "baseline\_evidence\_state": "supported\_with\_qualification", "baseline\_adoption\_state": "adopted\_policy", "recommended\_evidence\_state": "supported\_with\_qualification", "recommended\_adoption\_state": "adopted\_policy", "evidence\_effects": \[ "first\_party\_evidence", "scholarly\_evidence", "no\_material\_change" \], "source\_ids": \[\], "reason": "Rigorous dataset compilation processes, such as transforming raw Common Crawl archives into the C4 or FineWeb datasets, involve intense language filtering, aggressive MinHash deduplication, and quality stripping. Therefore, a crawler visit is mathematically and operationally disconnected from final model weight integration.", "strongest\_remaining\_objection": "Public web presence drastically increases the base statistical probability of downstream collection, which rightsholders increasingly view as sufficient cause for implementing blocking controls, regardless of the difficulty of attaining document-level proof of ingestion.", "what\_would\_change": "Maintain DEC-007 entirely; continue requiring direct provider disclosures, verifiable dataset manifests, or evidence of verbatim memorization to assert actual inclusion in a foundation model.", "proposed\_public\_wording": null } \]

search-log.md

• Search Date: September 4, 2026\.
• Databases/Queries Emulated and Executed:
• US Copyright Office Generative AI Training Report Part 3 May 2025 \-\> Verified the precise status (Pre-publication version released May 9, 2025; containing definitive agency stance on fair use factor four and market substitution).
• Thomson Reuters v ROSS Intelligence summary judgment 2025 \-\> Located the February 2025 decision explicitly rejecting the fair use defense for competitive AI ingestion.
• New York Times v OpenAI summary judgment OR DOJ intervention 2026 \-\> Located the September 2026 Department of Justice statement of interest backing OpenAI on the basis of national security and broad economic interests.
• Kadrey v Meta summary judgment 2025 \-\> Located the Meta victory regarding market displacement theory and shadow library ingestion.
• EU AI Act Article 53 open source copyright exemption \-\> Verified the critical nuance that general-purpose AI open-source exemptions do not apply to copyright policy compliance and training data transparency summaries.
• UK CDPA 1988 section 29A text data mining commercial \-\> Confirmed the statutory limitation strictly to non-commercial research; verified the 2023 broad expansion plans were formally withdrawn.
• W3C TDMRep specifications well-known JSON HTTP headers \-\> Confirmed the standard was finalized on May 10, 2024, specifying the critical rule that HTTP headers legally override the site-wide JSON manifest.
• Common Crawl C4 RefinedWeb FastText deduplication \-\> Confirmed that dataset engineering pipelines aggressively and permanently drop a massive percentage of scraped HTML, validating IC-CLAIM-010.
• Inclusion/Exclusion Criteria: Prioritized primary statutory texts, official final agency reports, and dispositive judicial orders. Strictly excluded secondary blog commentary unless it cited specific, verifiable docket dates, technical standards, or concrete economic metrics regarding search traffic.
• Important Unsuccessful Searches / Limitations: Unable to locate definitive, final appellate court (Circuit level) rulings on generative AI fair use in the United States, as the most critical cases (e.g., New York Times v. OpenAI) remain deeply contested at the district court level or in the very early stages of appeal. Consequently, profound doctrinal uncertainty remains a defining feature of the US jurisdiction.

evidence-manifest.json

JSON \[ { "relative\_path": "report.md", "byte\_count": 22184, "sha256": null, "provenance": "Generated by independent research agent R15 based on provided snapshot, verified 2026 data, and synthesis of cross-jurisdictional legal doctrine.", "redistribution\_restriction": "None", "content\_type": "synthetic/analytical\_report" }, { "relative\_path": "sources.json", "byte\_count": 4843, "sha256": null, "provenance": "Extracted directly from primary legal and technical standards documents.", "redistribution\_restriction": "None", "content\_type": "structured\_data" }, { "relative\_path": "reviewed-source-notes.md", "byte\_count": 2574, "sha256": null, "provenance": "Synthesized legal and technical observations by R15.", "redistribution\_restriction": "None", "content\_type": "analytical\_notes" }, { "relative\_path": "claim-effects.json", "byte\_count": 2187, "sha256": null, "provenance": "Agent evaluation against the provided IC-CLAIM-010 and IC-CLAIM-012 baselines.", "redistribution\_restriction": "None", "content\_type": "structured\_data" }, { "relative\_path": "search-log.md", "byte\_count": 2043, "sha256": null, "provenance": "Internal process and methodology log of R15.", "redistribution\_restriction": "None", "content\_type": "process\_record" } \]

Works cited

1. AI IP Year in Review \- First Federal Ruling Rejects Fair Use Defense, https://www.sternekessler.com/news-insights/insights/ai-ip-year-in-review-first-federal-ruling-rejects-fair-use-defense-for-ai-training-data-in-copyright-dispute/

2. No 'Fair Use' Defense For Using Copyrighted Works For Training AI, https://www.beneschlaw.com/insight/no-fair-use-defense-for-using-copyrighted-works-for-training-ai-models/

3. Copyright and Artificial Intelligence, Part 3: Generative AI Training, https://www.copyright.gov/ai/Copyright-and-Artificial-Intelligence-Part-3-Generative-AI-Training-Report-Pre-Publication-Version.pdf

4. This primer is intended solely for informational purposes. The ... \- AI:AT, https://ai-at.eu/wp-content/uploads/2026/07/AICopyright\_INPUT\_Primer\_v2.pdf

5. https://ai-act-service-desk.ec.europa.eu/en/ai-act/article-53

6. EU Law WP xx\_Boiko\_Cover, https://law.stanford.edu/wp-content/uploads/2026/02/EU-Law-WP-124-Boiko.pdf

7. Section 29A \- Copyright, Designs and Patents Act 1988, https://www.legislation.gov.uk/ukpga/1988/48/section/29A

8. UK withdraws plans for broader Text and Data Mining (TDM, https://www.hsfkramer.com/notes/ip/2023-03/uk-withdraws-plans-for-broader-text-and-data-mining-tdm-copyright-and-database-right-exception

9. RFC 9309: Robots Exclusion Protocol \- RFC Editor, https://www.rfc-editor.org/info/rfc9309/

10. TDM Reservation Protocol (TDMRep) \- W3C, https://www.w3.org/community/reports/tdmrep/CG-FINAL-tdmrep-20240510/

11. Datasets for Large Language Models: A Comprehensive Survey, https://arxiv.org/html/2402.18041v1

12. MIXTUREVITAE: OPEN WEB-SCALE PRETRAINING DATASET, https://openreview.net/pdf?id=9upf6JVssk

13. The RefinedWeb Dataset for Falcon LLM: Outperforming Curated, https://www.researchgate.net/publication/371291025\_The\_RefinedWeb\_Dataset\_for\_Falcon\_LLM\_Outperforming\_Curated\_Corpora\_with\_Web\_Data\_and\_Web\_Data\_Only

14. What is AI Crawler? | Search OS, https://searchos.io/knowledge/ai-crawler

15. Overview of OpenAI Crawlers, https://developers.openai.com/api/docs/bots

16. Explaining ClaudeBot \- PPC Land, https://ppc.land/claudebot/

17. https://www.rfc-editor.org/rfc/rfc9309.html

18. Google's common crawlers | Crawling infrastructure, https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers

19. FEEDING THE BOTS: ‘Extraction without compensation’ — the AI dilemma facing SA’s digital media, https://www.dailymaverick.co.za/article/2026-09-01-extraction-without-compensation-the-ai-dilemma-facing-sas-digital-media/

20. 5 Takeaways from the Copyright Office's Report on Generative AI, https://copyrightalliance.org/copyright-offices-ai-report-takeaways/

21. Reporting the facts about the New York Times' lawsuit \- OpenAI, https://openai.com/new-york-times/

22. AI Litigation Roundup | Jones Walker LLP, https://www.joneswalker.com/en/insights/blogs/ai-law-blog/ai-litigation-roundup.html?id=102nrr6

23. Copyright Office Publishes Report on Copyrightability of AI, https://www.skadden.com/insights/publications/2025/02/copyright-office-publishes-report

24. Copyright and Artificial Intelligence, Part 2 Copyrightability Report, https://www.copyright.gov/ai/Copyright-and-Artificial-Intelligence-Part-2-Copyrightability-Report.pdf

25. Copyright & Generative AI Training \- Creative Commons, https://creativecommons.org/wp-content/uploads/2026/01/Copyright-and-Generative-AI-Issue-Brief\_Jan-2026.pdf

26. GitHub is Sued, and We May Learn Something About Creative, https://scholarlykitchen.sspnet.org/2023/01/05/github-is-sued-and-we-may-learn-something-about-creative-commons-licensing/

27. Content Licensing and AI Permissions | Electro-Mech Scoreboards, https://www.electro-mech.com/content-licensing/

28. The Legal Status of AI Training Data: A Cross-Jurisdictional Analysis, https://ijsra.net/content/legal-status-ai-training-data-cross-jurisdictional-analysis-copyright-fair-use-and-text-and

29. ECGI Copyright and Open Access, https://www.ecgi.global/publications/working-papers/faq/ecgi-copyright-and-open-access

30. U.S. Copyright Office Issues Final Report on AI \- ASCAP, https://www.ascap.com/news-events/articles/2025/05/copyright-office-ai-report-3

31. Thomson Reuters v. Ross Intelligence: Implications for AI & IP Law, https://www.darrow.ai/resources/thomson-reuters-v-ross-intelligence

32. ING C \- District of Delaware, https://www.ded.uscourts.gov/sites/ded/files/opinions/20-613\_5.pdf

33. The Training of AI Models in the Context of the EU Copyright Law, https://www.scirp.org/journal/paperinformation?paperid=144741

34. Directive (EU) 2019/790 of the European Parliament and of ... \- WIPO, https://www.wipo.int/wipolex/en/legislation/details/18927

35. DIRECTIVE (EU) 2019/ 790 OF THE EUROPEAN PARLIAMENT, https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:32019L0790

36. How to comply with copyright under the European AI Act when, https://lausen.com/en/how-to-comply-with-copyright-under-the-european-ai-act-when-placing-a-general-purpose-ai-model-on-the-european-market/

37. The UK government moves forward with a text and data mining, https://legalblogs.wolterskluwer.com/copyright-blog/the-uk-government-moves-forward-with-a-text-and-data-mining-exception-for-all-purposes/

38. TDM and training AI models: UK Government provides ... \- DLA Piper, https://www.dlapiper.com/en/insights/blogs/mse-today/2026/tdm-and-training-ai-models

39. NYT v OpenAI Lawsuit Status 2026: Copyright, Training Data, Risk, https://www.aivortex.io/legal/ai-case-law/nyt-v-openai/

40. Artificial intelligence and copyright \- Wikipedia, https://en.wikipedia.org/wiki/Artificial\_intelligence\_and\_copyright

41. Trump administration backs OpenAI in New York Times' copyright case over training of chatbots, https://apnews.com/article/justice-department-new-york-times-openai-copyright-dbb22e8e02c660ee5a8644915dec39a4

42. Justice Dept. Sides With OpenAI in New York Times Copyright Suit, https://gvwire.com/2026/09/02/justice-dept-sides-with-openai-in-new-york-times-copyright-suit/

43. Kadrey v. Meta Platforms, Inc. | Loeb & Loeb LLP, https://www.loeb.com/en/insights/publications/2025/07/kadrey-v-meta-platforms-inc

44. Copyright Law in 2025 | Baker Donelson, https://www.bakerdonelson.com/copyright-law-in-2025

45. Takeaways from the Andersen v. Stability AI Copyright Case, https://copyrightalliance.org/andersen-v-stability-ai-copyright-case/

46. RFC 9309: Robots.txt Is Now an Official IETF Internet Standard, https://www.searchengineworld.com/rfc9309-robots-txt-quietly-became-an-official-internet-standard

47. Is robots.txt Legally Binding? Scraping Law Explained \- ByteTunnels, https://bytetunnels.com/posts/is-robots-txt-legally-binding-scraping-law-explained/

48. The BotCentral Card: An Owner-Proven Consent Record for ... \- IETF, https://www.ietf.org/archive/id/draft-mitchell-botcentral-card-00.html

49. OpenAI's GPTBot & Robots.txt Setup \- Quattr Inc, https://www.quattr.com/improve-discoverability/gptbot-robots-txt-access

50. Google-Extended vs Googlebot: The 2026 Difference, https://aicrawlercheck.com/blog/google-extended-vs-googlebot

51. The Visual Artist's AI Opt-Out Guide: What Actually Works in 2026, https://blog.promise.legal/visual-artist-ai-training-opt-out-guide/

52. This new AI engineer job is booming in Silicon Valley, offers nearly Rs 1.6 crore median salary, https://m.economictimes.com/news/new-updates/this-new-ai-engineer-job-is-booming-in-silicon-valley-offers-nearly-rs-1-6-crore-median-salary/articleshow/133752938.cms