External Distribution Observatory: Narrow Verification of Crawling, Indexing, Citation, Archival, and Training Evidence

Full report

On this page

Executive Assessment

The verification of external distribution—spanning web crawling, indexing, on-demand retrieval, citation, archival, and ultimate foundation model training—demands a fundamental epistemological shift from inferring intent to measuring concrete, downstream cryptographic and network artifacts. An exhaustive assessment of current technical standards, network protocols, and crawler documentation reveals that local publishing configurations provide zero proof of external consumption. The baseline operational evidence rules under review for IntelligenceCompact.com, specifically IC-CLAIM-009 and IC-CLAIM-010, correctly assert that local permissions do not equate to external execution, nor does broad public availability or network transit prove inclusion in a specific model’s training corpus. This investigation demonstrates that observability requires strictly bounded evidence thresholds for each distinct stage of the data distribution funnel. Crawling events can only be verified through rigorous network-layer identity checks, encompassing Forward-Confirmed Reverse DNS (FCrDNS) for legacy search indexers or precise, temporally bound matching against provider-published Classless Inter-Domain Routing (CIDR) blocks for modern foundation model crawlers. However, a verified crawl proves nothing beyond payload transit. Archival and indexing must be independently verified by interrogating external endpoints, such as the Crawl Index JSON (CDXJ) APIs provided by the Common Crawl Foundation, which yield exact cryptographic hashes and byte-offset coordinates confirming persistent data storage. Ultimately, the boundary between indexing and model training remains an opaque epistemic gap. Proving that a specific document was integrated into a specific model run cannot rely on probabilistic inferences, synthetic benchmarking, or the mere presence of a vendor’s training crawler in edge server logs. Because raw crawled data undergoes massive heuristic filtration, deduplication, and toxicity scrubbing prior to training, specific training inclusion remains an unsupported inference absent direct provider disclosure or a reproducible dataset manifest. The resulting recommendation is to maintain the strict separation between local configuration rules and external observation schemas, deploying a centralized, privacy-preserving intake framework to systematically categorize verifiable distribution events while deliberately redacting unconfirmed inferences to prevent the manufacturing of proof.

Research Boundaries, Methodology, and Execution Context

This research was executed on Friday, September 4, 2026, originating from Cicero, Illinois, United States. The investigation utilizes ordinary authorized public access to technical documentation, standards drafts, and indexing specifications to establish an exhaustive evidentiary standard for external distribution claims independent of local system configurations. The analysis strictly isolates spontaneous external observations from induced operator tests. Induced tests—such as triggering a retrieval-augmented generation (RAG) agent to fetch a specific Uniform Resource Locator (URL) during a live chat session—prove only that an on-demand retrieval pipeline is functional. They contaminate the observation environment by manufacturing a synthetic request, which cannot be conflated with autonomous algorithmic ingestion or independent index discovery. Consequently, all observational schemas designed herein require logging mechanisms that can differentiate between authenticated, user-directed fetches and autonomous algorithmic crawling. Where explicit execution of a live external API request was technically inaccessible or unauthorized within the read-only parameters of this environment, the report provides a bounded, explicit null-observation report while outlining the exact reproducible protocol required for future deployment. No synthetic network responses are presented as live empirical data, ensuring the integrity of the evidence baseline.

The Epistemological Chasm: Intent Versus Execution (IC-CLAIM-009)

The foundation of web access control relies heavily on the Robots Exclusion Protocol (REP), standardized by the Internet Engineering Task Force (IETF) in RFC 93091. Initially introduced by Martijn Koster in 1994 and formalized in 2022, this protocol provides a mechanism for site operators to declare which automated clients may access specific URI paths2. However, RFC 9309 is fundamentally advisory; it represents a cooperative discovery layer rather than an enforced application-layer access control system3. The presence of an Allow directive or the absence of a Disallow directive solely dictates the publisher's local intent. It generates no cryptographic artifact, network receipt, or observable proof that any external system successfully requested, parsed, or stored the content. The semantic limitations of RFC 9309 have become glaringly apparent with the proliferation of generative artificial intelligence. Fetching a page to answer a human user via live search is fundamentally different from copying that page into a model training corpus5. To address this, the industry is currently fracturing into multiple proposed extensions, all of which remain local policy declarations rather than evidence of external action.

Despite the sophisticated granularity these proposals introduce, they share a unified epistemic limitation: they are signs on an unlocked door3. A vendor’s documentation may guarantee compliance with a BotCentral train: false directive, but the existence of the policy file on the origin server constitutes zero empirical evidence of downstream compliance, ingestion, or exclusion. Therefore, IC-CLAIM-009 is strongly supported and verified: a permissive robots policy, sitemap, feed, or machine-use statement proves local publication intent exclusively. Transitioning a document's status in an external distribution observatory to "crawled" or "indexed" requires independent network receipts or external index verification.

Validating Network Transit: The Mechanics of Crawler Identity

To prove a crawling event, the identity of the requesting agent must be definitively established at the network layer. User-agent strings are trivially spoofed by malicious actors, unauthorized scrapers, and generic vulnerability scanners4. Validating an authorized crawler requires mapping the user-agent claim against verifiable network infrastructure. The methodologies for this validation bifurcate into DNS-based authentication for legacy search engines and CIDR-based authentication for modern AI foundation model developers.

Forward-Confirmed Reverse DNS (FCrDNS) Verification

Google relies on a robust FCrDNS protocol for verifying the identity of its indexing fleet, including Googlebot8. When an edge server receives a request claiming to be a Google crawler, the observatory must execute a multi-step domain name system cryptographic loop:

1. Extract the originating Internet Protocol (IP) address from the raw server logs.

2. Execute a reverse DNS (PTR) lookup on the IP address. The resulting domain must terminate in a recognized Google suffix, specifically googlebot.com, google.com, or googleusercontent.com8.

3. Execute a forward DNS (A or AAAA) lookup on the exact hostname returned in the second step.

4. The IP address returned by the forward lookup must perfectly match the original IP address recorded in the server logs8.

Only when this loop successfully closes can an observation legitimately claim that Google crawled the document. A failure at any step invalidates the event, relegating it to the category of unverified spoofing. This standard applies equally to other search providers that document their FCrDNS suffixes.

Dynamic CIDR-Based Verification and the Shared Prefix Problem

Conversely, providers such as OpenAI and Anthropic do not support reverse DNS validation for their crawling fleets10. Instead, they publish dynamic JSON endpoints containing authorized Classless Inter-Domain Routing (CIDR) ranges. Anthropic publishes a unified bots.json feed containing prefixes for ClaudeBot, Claude-User, and Claude-SearchBot11. OpenAI publishes multiple, distinct lists corresponding to highly specialized bot behaviors:

This architecture introduces a critical epistemic ambiguity known as the shared prefix problem. Because OpenAI's autonomous crawlers operate on shared infrastructure, several CIDR prefixes appear in both gptbot.json and searchbot.json simultaneously10. Consequently, verifying the IP address against the JSON file is insufficient to determine the purpose of the crawl. A rigorous observation schema must first confirm the IP address falls within the published CIDR block at the exact timestamp of the request, and subsequently read the exact user-agent string from the identical log line to determine if the event was an indexing sweep (OAI-SearchBot) or a training sweep (GPTBot)10. Failing to record both variables synchronously results in an ambiguous evidence artifact that cannot justify a specific external distribution claim. Furthermore, measurement contamination frequently occurs at the Web Application Firewall (WAF) layer. Edge protections often rate-limit AI crawlers, serving HTTP 429 (Too Many Requests) or HTTP 403 (Forbidden) responses7. If a server log records a verified GPTBot request that resulted in a 429 response, the crawler was successfully identified, but the data payload was never transmitted. An observatory must explicitly filter out non-200 HTTP status codes before logging a successful network transit event7.

Archival and Indexing Observability: Cryptographic Proof via CDXJ

Proving that a crawled document was successfully archived and indexed requires moving beyond the origin server's ingress logs and directly interrogating external repository infrastructure. The Common Crawl Foundation provides an open-source repository of web crawl data, representing one of the largest upstream datasets utilized for training foundation models15. Validating inclusion in Common Crawl requires navigating the Crawl Index JSON (CDXJ) protocol, a format originally derived from the Internet Archive's Wayback Machine15. The CDXJ index is a compressed, sorted plain-text file where each line represents a discrete historical capture event15. The files are gzip-compressed in blocks to allow rapid random access and binary searching across billions of records15. Querying this index requires standardizing the target URL into a Sort-friendly URI Reordering Transform (SURT). The SURT algorithm converts the URL to lowercase, strips the protocol, reverses the host name components into a comma-separated format, appends a closing parenthesis, and preserves the path15. For example, the URL https://www.example.org/index.html is mathematically transformed into the primary index key org,example,www)/index.html15. When the CDX Server API (index.commoncrawl.org) is queried with a valid SURT, it returns a JSON object containing vital metadata that constitutes absolute cryptographic proof of indexing15.

This JSON response constitutes definitive proof of indexing. To prove archival retrieval and verify the exact textual contents, the observer must use the filename, offset, and length variables to construct an HTTP Range request directed at the primary data bucket (data.commoncrawl.org)16. A successful HTTP 206 Partial Content response containing the raw HTML payload confirms the archival claim18. Unless this sequence is mathematically reproduced and logged, any claim of "Common Crawl inclusion" is merely theoretical and cannot be admitted as observational evidence.

The Epistemic Gap Between Availability and Model Training (IC-CLAIM-010)

The most rigorous analytical boundary in this domain exists between public indexing and foundation model training. IC-CLAIM-010 stipulates that public accessibility, explicit crawler permission, Common Crawl eligibility, or even a verified GPTBot log request is wholly insufficient evidence that a specific model trained on a specific Intelligence Compact document. This limitation is structurally inherent to modern AI dataset curation pipelines. When an AI vendor ingests an upstream corpus like Common Crawl or conducts a proprietary massive-scale scrape via GPTBot, the raw HTML undergoes severe downstream heuristic processing16. Datasets are subjected to strict deduplication algorithms (minhash/SimHash) to remove boilerplate and mirror sites, toxicity classifiers to purge unsafe content, and perplexity scoring to eliminate low-quality or machine-generated text16. A document may be successfully crawled by a verified training bot, perfectly indexed via CDXJ, and successfully archived in a WARC file, only to be completely discarded during the data preparation phase preceding the actual GPU training run. Consequently, observing a verified GPTBot IP traversing a domain merely proves that the document entered the candidate pool for future training. It is statistically and epistemologically impossible to assert document-level training inclusion based solely on upstream network transit. Proof of training inclusion requires direct downstream evidence: an explicit disclosure from the model provider, a verifiable dataset manifest (such as cryptographic hashes included in a documented training run), or reproducible extraction attacks. While extraction attacks—prompting a model to recite a specific document verbatim—can occasionally prove memorization, they are methodologically fraught, highly prone to hallucination false positives, and fail to detect data that was trained upon but not memorized. Thus, IC-CLAIM-010 is fully supported by the technical realities of machine learning pipelines: the evidence threshold for specific model-training inclusion must remain strictly gated behind direct provider artifacts, entirely rejecting public network availability or upstream crawler transit as sufficient proof.

Methodological Frameworks and Topic-Specific Requirements

To operationalize these boundaries, the observatory requires specific protocols for categorizing evidence, measuring citations without contamination, and logging events while preserving privacy.

Event-by-Evidence Threshold Matrix

Reproducible Citation-Benchmark Protocol

To measure Retrieval-Augmented Generation (RAG) citations without manufacturing proof through induced queries, researchers must deploy a rigorous, blinded query protocol:

1. Isolation: Utilize a clean browser instance or headless automated browser with no authentication, history, or personalized session cookies. Route traffic through a neutral VPN endpoint to eliminate geographic personalization and IP-based cache bias.

2. Query Design: Formulate a zero-shot prompt that interrogates a highly specific, novel factual assertion unique to the target site. Crucially, the prompt must not contain the target domain name, author name, or exact proprietary phrasing. Providing the domain name contaminates the test by forcing the model's routing algorithm to target the site.

3. Execution: Submit the query to the target model interface (e.g., ChatGPT with Search, Perplexity).

4. Logging: Record the exact prompt, the model string identifier exposed in the UI or API, the exact UTC time, the browsing settings, and the full raw textual output payload.

5. Verification: Parse the output payload for citation links. If the model spontaneously discovers and cites the target URL to support its assertion, the independent citation event is validated.

6. Negative Control: Submit a query about a known fabricated entity or synthetic claim that does not exist on the target site or the broader web. If the model cites the target site for the synthetic claim, the citation algorithm is hallucinating, invalidating the benchmark reliability for that specific model version.

Redacted Observation Schema (Privacy-Preserving JSON Design)

To maintain privacy while preserving verifiable evidence of network transit, raw server logs must be mathematically transformed before public storage. Unrelated IP addresses, session cookies, and local file paths must be scrubbed, leaving only the verified network artifact.

JSON { "schema\_version": "1.0", "observation\_id": "OBS-20260904-001", "event\_type": "verified\_crawl\_cidr", "timestamp\_utc": "2026-09-04T14:22:11Z", "redacted\_evidence": { "agent\_claimed": "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; \+https://openai.com/gptbot", "ip\_verification\_status": "MATCH", "matched\_cidr\_block": "141.136.192.0/20", "source\_registry": "https://openai.com/gptbot.json", "target\_path\_hash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "http\_status\_delivered": 200, "byte\_count": 14520 }, "privacy\_transformations": \[ "origin\_ip\_redacted", "target\_path\_hashed\_sha256", "cookies\_and\_parameters\_dropped" \], "unsupported\_inferences": \[ "This record does not prove the payload was successfully retained by the crawler.", "This record does not prove the document was included in a foundation model training run." \], "permitted\_public\_statement": "A verified OpenAI training crawler requested a document on this domain and received a successful HTTP 200 response." }

Positive and Negative Synthetic Fixtures

To ensure the observatory software correctly processes network logs, the following synthetic fixtures demonstrate the required logic for validation and invalidation. Positive Synthetic Fixture (Verified Googlebot via FCrDNS): \[RAW\_LOG\]: 66.249.66.1 \- \- \[04/Sep/2026:21:00:00 \+0000\] "GET /research-paper.html HTTP/1.1" 200 4521 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; \+http://www.google.com/bot.html)" \[DNS\_REVERSE\_CHECK\]: 66.249.66.1 \-\> crawl-66-249-66-1.googlebot.com (PASS: Domain ends in googlebot.com) \[DNS\_FORWARD\_CHECK\]: crawl-66-249-66-1.googlebot.com \-\> 66.249.66.1 (PASS: IP matches origin) \[CONCLUSION\]: Verified spontaneous crawling event. \[PERMITTED\_PUBLIC\_STATEMENT\]: "A verified Googlebot indexer fetched the document." \[UNSUPPORTED\_INFERENCES\]: "Does not prove the document is ranking in search results." Negative Synthetic Fixture (Spoofed GPTBot via CIDR Mismatch): \[RAW\_LOG\]: 192.168.1.105 \- \- \[04/Sep/2026:21:05:00 \+0000\] "GET /research-paper.html HTTP/1.1" 200 4521 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; \+https://openai.com/gptbot" \[CIDR\_CHECK\]: 192.168.1.105 not present in current https://openai.com/gptbot.json arrays (FAIL) \[CONCLUSION\]: Unverified scraping event. User-agent spoofing detected. Event discarded from evidence ledger. \[PERMITTED\_PUBLIC\_STATEMENT\]: "An unverified agent spoofing OpenAI's crawler was detected and logged locally." \[UNSUPPORTED\_INFERENCES\]: "Cannot claim OpenAI crawled the site based on this request."

Bounded Observation Report

Target Objective: Verify indexing capabilities via the Common Crawl CDX API using strictly documented SURT parameters and HTTP Range requests. Execution Constraint: Active, dynamic API execution for novel payload retrieval was unauthorized within the read-only parameters of the current local execution environment. Bounded Execution Protocol: An analysis of the officially documented index structure for index.commoncrawl.org was executed against the exact schema defined by the Common Crawl Foundation15. Protocol Step 1: Standardize URL to SURT format: commoncrawl.org/get-started transforms to org,commoncrawl)/get-started15. Protocol Step 2: Query parameter formulation: ?url=commoncrawl.org/get-started\&output=json18. Observed Artifact (from Official Source Documentation):

JSON { "urlkey": "org,commoncrawl)/get-started", "timestamp": "20251014220259", "url": "https://www.commoncrawl.org/get-started", "mime": "text/html", "status": "200", "digest": "D4IRZZ6NS7QW37BB2ODQPCUGG7ISRFGV", "length": "12675", "offset": "686242195", "filename": "crawl-data/CC-MAIN-2025-43/segments/1759648664410.92/warc/CC-MAIN-20251016175923-20251016205923-00549.warc.gz" }

Conclusion from Bounded Test: The CDXJ protocol is highly robust and mathematically capable of returning exact cryptographic digests and storage coordinates, enabling an observatory to definitively prove archival15. However, no live, novel site observation was generated during this specific run due to environmental execution limits. This constitutes an explicit gap in the live data feed, successfully validating the protocol while strictly preventing the manufacturing of synthetic proof. Permitted Public Statement: "The CDXJ API protocol is structurally verified for archival confirmation, but no live documents have been scanned in this session." Unsupported Inferences: "Cannot claim any specific project document is currently archived in Common Crawl based on this structural test."

Claim-Impact Assessment and Strategic Implications

The baseline operational rules established in DEC-007 are fundamentally sound and align precisely with the technical realities of web architecture. IC-CLAIM-009 is validated: local configuration proves intent only. IC-CLAIM-010 is validated: public availability and upstream network transit do not equal specific model ingestion.

Strongest Countercase and Resolution

Countercase: A statistical argument suggests that if a site operator allows GPTBot, observes tens of thousands of verified GPTBot hits on high-value canonical URLs, and later observes a major ChatGPT model release exhibiting specific, novel knowledge directly mirroring those URLs, the inference of model-training inclusion is overwhelming, even without a vendor manifest. From a pragmatic intelligence standpoint, demanding a cryptographic manifest sets an impossible evidentiary burden that prevents reasonable risk modeling and data governance. Resolution of Countercase: While the statistical correlation is highly suggestive and useful for internal threat modeling, it fails the standard of independent public verification. Because foundation models frequently hallucinate, or arrive at similar conclusions via disparate upstream data lineages (e.g., secondary reporting of the site's original data), output mimicking the input is not definitive proof of direct extraction. Retaining the strict requirement of direct disclosure or manifest validation prevents the project from publishing confident public assertions that could later be mathematically disproven by external dataset audits.

Best Next Research Action

The immediate next operational step is to engineer an automated, read-only log-parsing utility that continuously cross-references ingested HTTP server logs against the daily-fetched JSON CIDR ranges of major AI providers (OpenAI, Anthropic, ByteDance). This utility must synchronously validate the IP address against the CIDR block and the exact user-agent string against the required token (e.g., separating GPTBot from OAI-AdsBot), tagging matching events with their respective purpose classifications, and ruthlessly discarding all unverified spoofing attempts into a separate anomaly ledger.

Deliverable 2: sources.json

JSON { "agent\_id": "R16", "research\_date": "2026-09-04T21:34:53-05:00", "sources": \[ { "source\_id": "R16-S001", "matched\_ic\_source\_id": null, "title": "The BotCentral Card: An Owner-Proven Consent Record for Automated Web Clients (draft-mitchell-botcentral-card-00)", "authors": "Mitchell", "issuing\_institution": "IETF (Internet-Draft)", "document\_type": "technical\_standard\_draft", "canonical\_url": "https://www.ietf.org/archive/id/draft-mitchell-botcentral-card-00.html", "retrieved\_url": "https://www.ietf.org/archive/id/draft-mitchell-botcentral-card-00.html", "publication\_date": "2026-09-01", "version\_date": "2026-09-01", "effective\_date": null, "accessed\_at": "2026-09-04T21:34:53-05:00", "jurisdiction": "Global", "legal\_or\_policy\_status": "Informational Draft", "publication\_status": "Published", "host\_status": "Available", "review\_scope": "Full Text Snippet", "reviewed\_passages": "Section 1, 3.1. Consent separation into retrieve, train, act, cite.", "supported\_proposition": "Robots Exclusion Protocol (RFC 9309\) cannot express purpose; new protocols are required to distinguish retrieval from training consent.", "important\_limitation": "This is an informational IETF draft, not an enforceable global standard, and relies on voluntary client compliance at the application layer.", "claim\_ids": \["IC-CLAIM-009"\], "evidence\_lineage": "Primary Source", "snapshot\_path": null, "sha256": null, "missingness\_notes": "SHA-256 and snapshot path null due to read-only constraint preventing local file creation." }, { "source\_id": "R16-S002", "matched\_ic\_source\_id": null, "title": "Verify Googlebot and other Google crawlers", "authors": "Google Search Central", "issuing\_institution": "Google", "document\_type": "official\_documentation", "canonical\_url": "https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests", "retrieved\_url": "https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests", "publication\_date": "2026-03-20", "version\_date": "2026-03-20", "effective\_date": null, "accessed\_at": "2026-09-04T21:34:53-05:00", "jurisdiction": "Global", "legal\_or\_policy\_status": "Corporate Policy/Technical Standard", "publication\_status": "Published", "host\_status": "Available", "review\_scope": "Full Text Snippet", "reviewed\_passages": "Manual verification steps using host command for reverse and forward DNS.", "supported\_proposition": "Cryptographic proof of a Googlebot crawl requires a successful forward-confirmed reverse DNS (FCrDNS) lookup matching the original IP.", "important\_limitation": "Only verifies the identity of the network transit request, does not indicate if the fetched content was successfully indexed or retained.", "claim\_ids": \["IC-CLAIM-009"\], "evidence\_lineage": "Primary Source", "snapshot\_path": null, "sha256": null, "missingness\_notes": "SHA-256 and snapshot path null due to read-only constraint preventing local file creation." }, { "source\_id": "R16-S003", "matched\_ic\_source\_id": null, "title": "OpenAI Crawler User Agents and Verification", "authors": "OpenAI Documentation", "issuing\_institution": "OpenAI", "document\_type": "official\_documentation", "canonical\_url": "https://developers.openai.com/api/docs/bots", "retrieved\_url": "https://developers.openai.com/api/docs/bots", "publication\_date": null, "version\_date": null, "effective\_date": null, "accessed\_at": "2026-09-04T21:34:53-05:00", "jurisdiction": "Global", "legal\_or\_policy\_status": "Corporate Policy/Technical Standard", "publication\_status": "Published", "host\_status": "Available", "review\_scope": "Full Text Snippet", "reviewed\_passages": "Descriptions of OAI-SearchBot, OAI-AdsBot, GPTBot, and ChatGPT-User JSON endpoints.", "supported\_proposition": "OpenAI utilizes distinct JSON-published CIDR ranges to identify crawlers designated for search indexing versus foundation model training.", "important\_limitation": "Because OpenAI's ranges can overlap, IP verification alone cannot determine intent without synchronously reading the specific user-agent string.", "claim\_ids": \["IC-CLAIM-009", "IC-CLAIM-010"\], "evidence\_lineage": "Primary Source", "snapshot\_path": null, "sha256": null, "missingness\_notes": "SHA-256 and snapshot path null due to read-only constraint preventing local file creation." }, { "source\_id": "R16-S004", "matched\_ic\_source\_id": null, "title": "Common Crawl CDXJ Index API Documentation", "authors": "Common Crawl Foundation", "issuing\_institution": "Common Crawl", "document\_type": "official\_documentation", "canonical\_url": "https://commoncrawl.org/cdxj-index", "retrieved\_url": "https://commoncrawl.org/cdxj-index", "publication\_date": null, "version\_date": null, "effective\_date": null, "accessed\_at": "2026-09-04T21:34:53-05:00", "jurisdiction": "Global", "legal\_or\_policy\_status": "Technical Standard", "publication\_status": "Published", "host\_status": "Available", "review\_scope": "Full Text Snippet", "reviewed\_passages": "CDXJ index fields, SURT formatting, offset and length range requests.", "supported\_proposition": "Archival indexing is independently verifiable by querying the CDX server for a SURT formatted URL to retrieve JSON byte-offset coordinates.", "important\_limitation": "Verification of indexing in Common Crawl proves data payload existence, but does not prove downstream ingestion by any specific AI provider.", "claim\_ids": \["IC-CLAIM-009", "IC-CLAIM-010"\], "evidence\_lineage": "Primary Source", "snapshot\_path": null, "sha256": null, "missingness\_notes": "SHA-256 and snapshot path null due to read-only constraint preventing local file creation." } \] }

Deliverable 3: reviewed-source-notes.md

R16-S001 (draft-mitchell-botcentral-card-00)

• Narrow proposition supported: Traditional robots.txt implementations lack the semantic vocabulary to express the critical distinction between indexing a document for real-time retrieval and copying a document for offline foundation model training.
• Important limitation: The BotCentral protocol is merely a declarative governance draft. A client's failure to adhere to the requested policy cannot be prevented by the presence of the card itself at the network edge.
• Exact passage: "It cannot express purpose: fetching a page to answer a person is not the same act as copying it into a model training set... 'retrieve' true does not imply 'train' true."
• Reviewer attribution: Independent research agent R16.

R16-S002 (Google Search Central Crawler Verification)

• Narrow proposition supported: Definitive proof of a Googlebot crawl requires closing a cryptographic loop by executing a reverse DNS lookup to a Google domain, followed by a forward DNS lookup that perfectly matches the originating IP address.
• Important limitation: The DNS methodology strictly verifies the identity of the network transit request, but does not guarantee the HTML payload was successfully parsed, avoided 429 WAF blocking, or retained in the index.
• Exact passage: "Run a reverse DNS lookup on the accessing IP address from your logs... Verify that the domain name is either googlebot.com, google.com, or googleusercontent.com. Run a forward DNS lookup... Verify that it's the same as the original accessing IP address."
• Reviewer attribution: Independent research agent R16.

R16-S003 (OpenAI Bot Documentation)

• Narrow proposition supported: OpenAI validates its crawlers via dynamically published JSON endpoints (gptbot.json, searchbot.json) rather than reverse DNS, distinguishing its training fleet from its search indexing fleet.
• Important limitation: The CIDR blocks across OpenAI's discrete crawler services can overlap, meaning an IP match without a concurrent user-agent header validation is fundamentally insufficient to classify the nature of the crawl.
• Exact passage: "Published IP addresses: https://openai.com/gptbot.json ... Each setting is independent of the others – for example, a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot."
• Reviewer attribution: Independent research agent R16.

R16-S004 (Common Crawl CDXJ Index)

• Narrow proposition supported: A successful archival indexing event can be proven by querying a CDX API with a SURT-formatted URL to retrieve a filename, offset, and length, allowing direct HTTP byte-range retrieval of the stored WARC payload.
• Important limitation: Common Crawl is an upstream dataset; proof of storage in Common Crawl does not constitute proof that the document survived subsequent vendor heuristic filtration pipelines to be included in a model.
• Exact passage: "The raw index format... consists of a 'urltimestamp' header followed by a json dictionary... containing the urlkey... filename, offset and length, as well as a checksum (digest) of the content."
• Reviewer attribution: Independent research agent R16.

Deliverable 4: claim-effects.json

JSON \[ { "claim\_id": "IC-CLAIM-009", "baseline\_evidence\_state": "verified\_local\_rule", "baseline\_adoption\_state": "adopted\_policy", "recommended\_evidence\_state": "verified\_local\_rule", "recommended\_adoption\_state": "adopted\_policy", "evidence\_effects": \[ "new direct/official evidence", "first-party evidence", "no material change" \], "source\_ids": \["R16-S001", "R16-S002"\], "reason": "First-party documentation from standard bodies and major crawler operators definitively confirms that robots.txt, ai.txt, and consent cards only dictate local policy and provide zero technical guarantee or evidence of external distribution.", "strongest\_remaining\_objection": "None regarding the logic; practically, tracking compliance requires robust infrastructure that many operators lack, creating an illusion of compliance.", "what\_would\_change": "Evidence demonstrating that a local policy file directly triggers an enforced cryptographic lock on external systems (which is technically impossible under current HTTP semantics).", "proposed\_public\_wording": "A permissive robots policy, sitemap, or machine-use statement establishes local publication intent only. It does not prove that an external system crawled, indexed, retrieved, cited, archived, or trained on the content. Verified logging is required to prove downstream events." }, { "claim\_id": "IC-CLAIM-010", "baseline\_evidence\_state": "supported\_with\_qualification", "baseline\_adoption\_state": "adopted\_policy", "recommended\_evidence\_state": "supported\_with\_qualification", "recommended\_adoption\_state": "adopted\_policy", "evidence\_effects": \[ "methodological criticism", "no material change" \], "source\_ids": \["R16-S003", "R16-S004"\], "reason": "The existence of extensive data curation pipelines (deduplication, toxicity filtering, perplexity scoring) between raw crawls (GPTBot or Common Crawl) and actual model training mathematically invalidates using upstream network transit as proof of downstream model inclusion.", "strongest\_remaining\_objection": "While absolute cryptographic proof is lacking, highly specialized or highly traversed content has a statistically overwhelming probability of inclusion, which may be sufficient for risk-modeling purposes even if it fails the verification standard.", "what\_would\_change": "The introduction of a standardized 'training receipt' protocol by major AI vendors that cryptographically logs exact URL hashes included in final training runs.", "proposed\_public\_wording": "Public availability, explicit crawler permission, or verified crawling logs cannot prove that a specific foundation model trained on a specific document. Claims of specific training inclusion require a direct vendor manifest, reproducible artifact, or equivalent direct disclosure." } \]

Deliverable 5: search-log.md

• Actual Search Date: September 4, 2026\.
• Databases/Sources Consulted: Provided prompt discovery leads and localized snippet repository containing primary specifications (IETF standards track, Google Developers documentation, OpenAI API Documentation, Common Crawl Foundation Technical Docs).
• Selection Criteria: Exclusively primary-source technical documentation, established network protocols (RFCs), proposed architectural standards (draft-mitchell), and official provider instructions for crawler validation.
• Included Evidence: RFC 9309 documentation, draft-mitchell-botcentral-card-00 semantics, Googlebot FCrDNS validation schema, OpenAI JSON CIDR validation schema (including the distinction between GPTBot, SearchBot, and AdsBot), Common Crawl CDXJ index parameters and SURT transformations.
• Excluded Evidence: Third-party Search Engine Optimization (SEO) blogs providing unverified secondary commentary on AI crawlers. Anecdotal social media reports of site scraping.
• Inaccessible Material: Live querying of index.commoncrawl.org or live DNS lookups via command line, explicitly constrained by the read-only limitations of the local intelligence environment.
• Scope Limits: The analysis relies strictly on documented architectures and published standards rather than real-time network probing, ensuring compliance with the prohibition on evading restrictions or executing scale crawls.
• Important Unsuccessful Searches: Not applicable; primary documentation targets validating the impossibility of intent-as-proof and the mechanics of CDXJ indexing were successfully located and extracted from the provided baseline context.

Deliverable 6: evidence-manifest.json

JSON \[ { "relative\_path": "report.md", "byte\_count": null, "sha256": null, "provenance": "Agent R16 original synthesis and epistemological framing", "redistribution\_restriction": "None", "content\_type": "transformed\_analytical\_report" }, { "relative\_path": "sources.json", "byte\_count": null, "sha256": null, "provenance": "Extracted and structured from primary technical documentation", "redistribution\_restriction": "None", "content\_type": "raw\_structured\_data" }, { "relative\_path": "reviewed-source-notes.md", "byte\_count": null, "sha256": null, "provenance": "Agent R16 critical review of limitations within primary texts", "redistribution\_restriction": "None", "content\_type": "transformed\_analytical\_notes" }, { "relative\_path": "claim-effects.json", "byte\_count": null, "sha256": null, "provenance": "Agent R16 logic and evidence application against baseline DEC-007", "redistribution\_restriction": "None", "content\_type": "transformed\_analytical\_data" }, { "relative\_path": "search-log.md", "byte\_count": null, "sha256": null, "provenance": "Agent R16 methodology and exclusion record", "redistribution\_restriction": "None", "content\_type": "raw\_log" } \]

Works cited

1. The BotCentral Card: An Owner-Proven Consent Record for ... \- IETF, https://www.ietf.org/archive/id/draft-mitchell-botcentral-card-00.html

2. ai.txt: A Domain-Specific Language for Guiding AI Interactions with, https://arxiv.org/html/2505.07834v1

3. Designing Websites for AI Agents \- llms.txt, Structured Data, and, https://hidekazu-konishi.com/entry/designing\_websites\_for\_ai\_agents.html

4. Walsh-Research Bot Compliance Specification, https://wal.sh/research/bots/compliance-spec

5. The BotCentral Card: An Owner-Proven Consent Record for, https://datatracker.ietf.org/doc/draft-mitchell-botcentral-card/

6. Really Simple Licensing (RSL) 1.0 Specification, https://rslstandard.org/rsl

7. GPTBot: OpenAI's Crawler \- robots.txt & WAF \- Murat Ulusoy, https://www.muratulusoy.de/en/glossary/gptbot.html

8. Verify Requests from Google Crawlers and Fetchers, https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests

9. Verified Bots: How Googlebot Proves It's Googlebot | Evomi, https://evomi.com/blog/how-googlebot-proves-its-googlebot

10. Verify GPTBot IP: Which OpenAI Bot? \- TrustMyIP.com, https://trustmyip.com/gptbot-verifier

11. ClaudeBot IP addresses · Source CIDR, https://sourcecidr.com/anthropic/

12. OpenAI user agents — xSeek Docs, https://www.xseek.io/docs/openai-crawlers-and-user-agents

13. https://developers.openai.com/api/docs/bots

14. IP Range for Bot Detection Allow list \- OpenAI Developer Community, https://community.openai.com/t/ip-range-for-bot-detection-allow-list/1287217

15. CDXJ Index \- Common Crawl, https://commoncrawl.org/cdxj-index

16. Inside Common Crawl: The Dataset Behind AI Models (and Its Real, https://dev.to/extractdata/inside-common-crawl-the-dataset-behind-ai-models-and-its-real-world-limits-2eo2

17. Announcing the Common Crawl Index\!, https://commoncrawl.org/blog/announcing-the-common-crawl-index

18. Get Started \- Common Crawl, https://commoncrawl.org/get-started

19. Anthropic ClaudeBot | AI Crawler Directory \- DataFast, https://datafa.st/crawlers/anthropic-claudebot