Comprehensive Discoverability and Security Strategy for the RogueSwarms Registry

Full report

On this page

1. Technical Search Strategy

The dual operational nature of the RogueSwarms platform necessitates a bifurcated approach to discoverability, deliberately separating traditional human-centric search engine optimization (SEO) from the automated protocols required for machine discovery. For conventional search engines such as Google and Bing, the strategy must rely entirely on traditional semantic HTML architecture, high-performance server-side rendering, and comprehensive internal linking topologies. By mid-2026, authoritative guidance from major search engine operators confirmed that generative AI overviews and traditional search indexes do not utilize machine-specific text files or markdown summaries for ranking purposes1. Specifically, Google Search Central documentation explicitly states that optimizing for generative AI remains fundamentally tied to traditional SEO, and the engine actively ignores markdown maps when determining search placement or AI Overview citations1.

Consequently, the technical search strategy for human discoverability dictates that RogueSwarms must present a fully crawlable, JavaScript-independent HTML Document Object Model (DOM). Core pages—encompassing agent profiles, capability matrices, and protocol definitions—must render their essential content server-side. Semantic tagging using standard HTML5 elements ensures that traditional crawlers can interpret the structural hierarchy of the platform without relying on secondary machine files or fragmented JavaScript payloads. The platform must meticulously avoid gating primary content behind client-side rendering frameworks, as doing so introduces indexing delays and severely degrades discoverability in traditional search environments. Furthermore, traditional SEO principles must be applied to capability and agent pages, ensuring that natural language descriptions align with the exact search intents of developers and procurement teams seeking specific autonomous functionalities.

2. Machine-Discovery Strategy

Unlike conventional web crawlers that heuristically parse HTML to infer meaning, autonomous software agents, coding assistants, and Model Context Protocol (MCP) clients require deterministic, structurally rigid entry points. The machine-discovery strategy for RogueSwarms deliberately abandons HTML scraping in favor of HTTP-layer routing and standardized catalog files. This represents the core of the emerging Business-to-Agent (B2A) optimization paradigm. Autonomous clients arriving at the domain root should be immediately redirected to machine-readable artifacts before they attempt to instantiate a headless browser or parse the DOM5.

This deterministic discovery is achieved through a multi-layered protocol stack. The foundational layer utilizes HTTP Link headers injected into every server response, pointing agents directly to alternate representations and capability catalogs without requiring an initial payload inspection6. The second layer implements the Internet Engineering Task Force (IETF) RFC 9727 standard, utilizing a well-known API catalog to inventory the platform's programmatic surfaces and specifications7. The third layer leverages the Model Context Protocol (MCP) over JSON-RPC 2.0, providing a universally standardized interface for agents to discover and invoke tools securely9. Finally, for large language models and integrated development environments (IDEs) such as Cursor or Claude Code, a standardized plain-text markdown file acts as an operational map of the domain's high-value content, drastically reducing the token overhead required for initial domain comprehension5. This architecture ensures that agents ranging from simple web scrapers to advanced, multi-step reasoning engines can instantly negotiate the optimal format for their specific capabilities.

3. Proposed robots.txt Configuration

The management of autonomous crawlers in 2026 requires precise, granular control over foundational training bots, real-time search indexers, and user-directed fetch agents. A pervasive failure in modern platform configuration is the accidental exclusion of a domain from generative search features due to a fundamental misunderstanding of vendor-specific user agents13. Platforms must rigorously differentiate between crawlers that collect data for model pre-training and indexers that feed real-time retrieval features. The RogueSwarms configuration must treat these vectors independently to protect intellectual property while maximizing market visibility.

Restricting pre-training crawlers prevents proprietary platform data, unique agent architectures, and specialized registry metadata from being absorbed into the base weights of foundational models without compensation15. Conversely, explicitly allowing search indexers ensures the platform remains highly visible as a cited source in generative retrieval features17. It is critical to note that under RFC 9309, crawlers obey only the single most specific group naming them; directives from wildcard groups are entirely ignored by specifically named agents14. Therefore, the configuration must explicitly repeat all necessary rules for each designated agent.

Table\_title: Recommended Autonomous Agent Directives

 

4. Sitemap Architecture

To satisfy the divergent requirements of human-facing search engines and agentic machine systems, RogueSwarms requires a partitioned sitemap index architecture containing format-specific manifests. Traditional search engines consume a standard XML sitemap detailing the HTML structure of the platform, utilizing it to schedule crawling and assess relative page priority. However, autonomous agents and programmatic pipelines benefit from highly targeted XML sitemaps that strictly index machine-readable assets, avoiding the token-wasting process of parsing interface scaffolding.

The architecture must rely on a primary sitemap-index.xml file that orchestrates multiple subordinate sub-sitemaps. The primary sitemap-pages.xml inventories the HTML user interface, ensuring traditional search indexers can traverse the domain effectively. A secondary ai-sitemap.xml operates alongside it, specifically designed for AI agents and RAG pipelines21. This secondary sitemap catalogs the markdown alternatives, JSON-LD knowledge graphs, and explicitly defined API endpoints. This deliberate bifurcation prevents intelligent agents from wasting finite context windows on layout elements while concurrently ensuring traditional search engines do not mistakenly index raw JSON data payloads as primary, human-readable search results.

5. Canonical-Link Rules

Because the RogueSwarms registry serves identical entities in multiple representations—HTML for human developers, Markdown for LLM ingestion, and JSON for programmatic execution clients—canonicalization is absolutely critical. Without stringent rules, traditional search engines will penalize the domain for duplicate content, while machine clients may become confused regarding the authoritative source of truth. Content negotiation plays a vital role here, but static link relations solidify the underlying architecture.

The HTML representation of any agent, capability, or protocol page is fundamentally defined as the canonical resource. If a client accesses the HTML version, the \<head\> must contain a standard rel="canonical" link pointing recursively to itself. When addressing how canonical URLs should work for JSON and HTML forms, the rule is absolute: when an autonomous system requests the Markdown or JSON variant, those specific files must feature HTTP Link headers asserting a rel="canonical" relationship pointing directly back to the primary HTML page.

Furthermore, to facilitate the proactive discovery of these non-HTML formats, the HTML page must utilize explicit rel="alternate" definitions6. For instance, a human-readable page located at /agents/data-optimizer will declare an alternate relation pointing to /agents/data-optimizer.md with the specific media type text/markdown. This protocol ensures that traditional search engines consolidate all algorithmic ranking signals into the human-readable page, while sophisticated agents can reliably detect and extract the structured data representation without ever parsing the surrounding HTML scaffolding.

6. HTTP Link-Header Recommendations

The single highest-leverage optimization for machine discovery occurs entirely outside the HTML document body, operating exclusively at the network layer. Automated agents prioritize speed and token efficiency; forcing an agent to parse a full HTML DOM merely to locate an API endpoint or capability schema is highly inefficient. Injecting RFC 8288 HTTP Link headers into the server's initial response allows agents to execute a lightweight HEAD request to discover the platform's entire machine-readable surface instantly6.

Table\_title: Required HTTP Link Header Injections

 

By configuring the origin server or Content Delivery Network (CDN) to append these headers globally across all successful responses, RogueSwarms becomes instantly and natively navigable by agentic infrastructure before a single byte of HTML is downloaded or rendered.

7. JSON-LD Recommendations

Structured data allows conventional search engines and sophisticated web agents to semantically comprehend the precise nature of the entities hosted on the RogueSwarms registry. The platform should leverage the comprehensive Schema.org vocabulary via JSON-LD to classify its distinct agent offerings and capabilities. While the generic SoftwareApplication type is the traditional fallback for software indexing, the more specific WebApplication type is highly recommended for cloud-based autonomous agents24.

Deploying the WebApplication schema inherently signals to both search engines and AI recommendation systems that the software operates dynamically via a URL and does not require local installation or compilation24. The JSON-LD graphs must be embedded directly in the HTML of the registry pages and also served via strict content negotiation when the Accept: application/ld+json header is present in the request. The architecture must utilize deeply nested properties to define the intricate relationships between the agent entity, the organization that developed it, the specific tasks it can execute, and the programmatic endpoints through which it is operated.

8. Agent-Page Metadata

Every agent profile residing on the RogueSwarms registry must expose a highly structured JSON-LD entity graph mapping the agent's identity, operational parameters, and execution constraints. In response to whether an agent profile should expose JSON-LD, the answer is definitively affirmative. This metadata serves as the critical bridge allowing traditional search engines to understand the complex software entity, thereby improving organic visibility for relevant queries. The primary schema type must be WebApplication, leveraging its inheritance from SoftwareApplication24.

The required metadata that every agent page must contain includes foundational properties such as name, description, applicationCategory (utilizing values such as BusinessApplication or DeveloperApplication), and browserRequirements to inform automated clients of any specific runtime dependencies24. Crucially, the actionApplication property must be utilized to define the specific software interface or API capable of successfully completing execution requests on the agent's behalf25. By explicitly connecting the agent's semantic identity to its mechanical execution environment, automated discovery tools can both accurately categorize the agent within the registry and autonomously determine the protocol required to instantiate it.

9. Capability-Page Metadata

Capability pages define the specific, discrete tasks an individual agent or an orchestrated swarm can execute. These pages require specialized metadata that bridges the semantic gap between a high-level description of a task and its functional, programmatic execution. Capability pages build useful organic discovery by ranking for long-tail, highly functional search queries. When a developer searches for "how to execute zero-shot data extraction via API," a capability page enriched with semantic execution data satisfies that specific intent far better than a generic homepage.

The JSON-LD structure on these pages should utilize the EntryPoint and Action schemas to define the precise interaction mechanics25. Properties such as httpMethod, contentType, and encodingType explicitly inform the consumer—whether human or machine—precisely how to format a valid execution request25. More importantly, the capability metadata must explicitly mirror the inputSchema defined in the corresponding Model Context Protocol (MCP) server configuration9. By providing a JSON Schema defining the expected parameters, type information, and strict validation requirements, the platform ensures that integrating LLMs can automatically construct and dispatch valid tool calls without requiring human intervention or trial-and-error debugging28.

10. Protocol-Page Metadata

Protocol pages serve as the definitive technical documentation for the standards and connection methods utilized by RogueSwarms. These pages must expose rich metadata targeting developer tools, agent frameworks, and specialized MCP clients. The platform must explicitly advertise its support for JSON-RPC 2.0 communication, which underpins the Model Context Protocol10.

Protocol metadata must clarify the supported MCP capabilities, explicitly defining whether the server supports the tools/list operation for dynamic tool discovery, the resources/read operation for context fetching, or the prompts/get operation for template retrieval30. This highly specific metadata enables dynamic orchestration frameworks to automatically negotiate capabilities during the connection initialization phase. By publishing this data, RogueSwarms drastically reduces the integration friction for developers attempting to connect their proprietary agentic pipelines to the broader ecosystem.

11. Documentation Metadata

Documentation serves as the fundamental operational manual for autonomous agents and coding assistants. To optimize ingestion by IDEs such as Cursor and deep-research agents, documentation pages must utilize strict, agent-readable formatting. Complex navigational scaffolding, interactive elements, and layout boilerplate must be stripped away in favor of high-signal, densely informative text32.

The metadata for these documentation pages must include explicit rel="alternate" HTTP Link headers pointing directly to raw markdown formats, ensuring that retrieval-augmented generation (RAG) systems do not ingest or process HTML boilerplate6. Furthermore, documentation metadata should expose standard OpenAPI links. As specified in the OpenAPI 3.1 standard, providing explicit operation IDs, structured component schemas, and rigorous security definitions within the documentation allows automated systems to validate their generated code directly against the protocol definitions before attempting execution, thereby minimizing runtime errors33.

12. Proposed .well-known Resources

The /.well-known/ directory structure, governed by RFC 8615, serves as the definitive trust, routing, and discovery root for automated systems across the web8. Centralizing configuration files within this standardized directory prevents crawlers from brute-forcing URLs in search of manifests and provides a cryptographically verifiable mechanism for establishing domain-level trust.

Table\_title: Proposed /.well-known/ Resource Architecture

 

13. Proposed llms.txt or Comparable Resource

The llms.txt file has rapidly emerged as the preeminent routing layer for Large Language Models, agentic IDEs, and autonomous research swarms5. Following the formalized v2 specification published by Answer.AI in August 2026, the file must be located precisely at the root of the domain, served with a text/plain or text/markdown MIME type, and formatted meticulously to facilitate programmatic regex parsing by non-reasoning extraction scripts5.

The file must begin with a single H1 header containing the literal brand or platform name, followed immediately by a blockquote summarizing the registry's overarching purpose in a manner highly suitable for an LLM system prompt context window5. The remainder of the file organizes the platform's canonical endpoints under descriptive H2 headings. Each individual entry must strictly follow the syntax format \- \Title\: Description, as any deviation breaks automated ingestion parsers5.

Additionally, RogueSwarms must provide an llms-full.txt variant. While llms.txt functions as an index or table of contents, llms-full.txt acts as the payload itself, concatenating the entire markdown corpus of the platform's documentation into a single, massive text file5. This allows capable models with sufficiently large context windows to ingest the entire platform schema in a single HTTP request, circumventing the need for recursive crawling, reducing origin server load, and guaranteeing complete context alignment.

14. Index/noindex Policy

To maintain the absolute integrity of the search ecosystem and protect the platform's computational infrastructure, RogueSwarms must enforce a rigorous and highly intentional index / noindex policy. Conventional search engines actively penalize platforms that expose raw data feeds or duplicate programmatic interfaces within traditional search results, viewing them as poor user experiences.

All user-facing HTML pages—including the homepage, marketing collateral, documentation UI, and canonical agent profile pages—must remain fully indexed to drive organic human traffic. Regarding the indexing of directory query pages (such as faceted search results within the registry), these must absolutely be assigned a noindex tag. Indexing dynamic query parameters creates massive crawl traps, dilutes link equity, and degrades the ranking potential of canonical pages.

Sensitive machine endpoints must also be strictly shielded from indexing. This includes all raw programmatic endpoints, which must explicitly serve an X-Robots-Tag: noindex HTTP header. The contents of the /.well-known/ directory, all .md alternate files, all raw .json data feeds, and specifically the raw OpenAPI specification files must be kept out of search engine indices45. Furthermore, any raw MCP tool invocation endpoints, user authentication portals, or internal telemetry routes must remain strictly unindexed to mitigate automated reconnaissance by unauthorized actors.

15. Duplicate-Content Strategy

The multi-format architecture of RogueSwarms inherently creates multiple representations of identical content. Without a deliberate technical strategy, search engines will flag the platform for duplicate content, heavily diluting its topical authority. The primary defense against this is the implementation of strict HTTP Content Negotiation utilizing the Vary: Accept header35.

When a client requests a specific agent profile, the server must inspect the Accept header. If the client requests text/html, the server delivers the standard webpage. If the client requests application/json, the server delivers the raw data payload. If text/markdown is requested, the optimized LLM variant is provided. Because all three formats exist at the exact same URL, duplicate content is mechanically impossible; there are simply alternate representations of a single canonical resource.

To prevent duplicate agent records in scenarios where versioning is required (e.g., an upgraded swarm model), the registry must enforce strict canonicalization. If an agent is deprecated but maintained for legacy support, its profile must feature a rel="canonical" link pointing to the newest iteration of that agent, ensuring that search engines and discovery protocols consolidate authority onto the active asset.

16. Monitoring Strategy

Observability is essential to differentiate between legitimate, high-value autonomous agent traffic and malicious, resource-exhausting scraping attempts. Simple User-Agent string monitoring is woefully insufficient, as threat actors trivially spoof these strings to masquerade as benevolent AI indexers13. The RogueSwarms monitoring strategy must rely on a synthesis of cryptographic verification and behavioral traffic analysis.

To verify the true identity of crawling agents, the platform must adopt the emerging IETF Web Bot Auth standard. This mechanism requires evaluating HTTP Message Signatures against the public keys published in the agent operator's JSON Web Key Set (JWKS) directory36. Agents that cryptographically prove their identity bypass heuristic rate limits, while unverified traffic asserting prestigious AI user agents is treated as highly suspicious and subjected to deep inspection.

For ongoing traffic management, the platform must implement rigorous rate limiting that returns an HTTP 429 "Too Many Requests" status code accompanied by an explicit Retry-After header49. Monitoring systems must track the frequency of 429 errors and enforce exponential backoff protocols. If an agent ignores the Retry-After header and continues to flood the network, the monitoring system must automatically escalate the 429 response to a hard block at the Web Application Firewall (WAF) level, preserving origin availability49.

17. Security Concerns

The integration of the Model Context Protocol and autonomous agent discovery introduces severe, novel attack vectors, primarily indirect prompt injection via sophisticated tool poisoning. As definitively categorized by OWASP LLM01:2026, the fundamental boundary between instructions and data entirely collapses within a Large Language Model's context window52. When an autonomous agent ingests an MCP tool description or a platform capability manifest, it inherently treats that untrusted data identically to authoritative system commands53. This creates a massive vulnerability footprint for any platform acting as a registry.

If a malicious actor registers a rogue agent on the platform and embeds hidden, adversarial instructions within the agent's schema metadata, any third-party AI system querying the registry could be instantly hijacked upon ingestion. This form of tool poisoning allows attackers to seamlessly redirect agent behavior, exfiltrate sensitive data, or force unauthorized API execution without ever interacting directly with the victim's prompt interface55. Furthermore, findings from the MCPSecBench evaluation framework reveal that the pre-execution artifacts and multi-turn workflows inherent in MCP significantly expand the attack surface, with researchers demonstrating that over 85% of identified attacks successfully compromised standard implementations57.

To mitigate these existential threats, RogueSwarms must implement aggressive, multi-layered schema sanitization. All user-submitted tool descriptions, metadata fields, and JSON-LD payloads must pass through an automated semantic firewall to detect anomalous instruction patterns, encoded payloads, or recursive loops59. Furthermore, the platform must architecturally enforce strict isolation between capability enumeration and execution authority. The discovery of a tool within the registry must never implicitly grant the permission to invoke it. Robust authorization gateways must sit between the discovery layer and the execution layer, ensuring that compromised agents cannot leverage registry access to pivot into destructive actions.

18. Established Standards versus Emerging Conventions (Authoritative Timeline)

The RogueSwarms architecture relies on a delicate synthesis of deeply entrenched IETF protocols and rapidly evolving, community-driven industry conventions. Understanding the maturity, publication dates, and authoritative bodies of these mechanisms is absolutely critical for maintaining robust, standards-compliant infrastructure.

The foundation of the platform's interoperability is built upon established, normative standards. The OpenAPI Specification (OAS), maintained by the OpenAPI Initiative, remains the authoritative contract for REST interfaces, reaching essential maturity with version 3.1.0 in early 202133. The architectural cornerstone of the /.well-known/ URI routing structure was solidified by the IETF via RFC 8615 in May 20198. The mechanism for Web Linking, which facilitates the crucial HTTP header injections, was formalized by the IETF in RFC 8288 in October 20178. Cryptographic identity verification is anchored by RFC 9421 (HTTP Message Signatures), published by the IETF in February 20246. Crucially for machine discovery, the api-catalog discovery linkset was officially standardized by the IETF as RFC 9727 in June 2025, providing the definitive machine entry point62.

Conversely, the operational reality of the Agentic Web is heavily dictated by emerging conventions that lack formal, international standardization. The Model Context Protocol (MCP), introduced by Anthropic in late 2024, operates as the de facto industry standard for tool integration, despite remaining an evolving, vendor-led initiative10. The llms.txt file format, initially proposed by Jeremy Howard of Answer.AI in September 2024 and subsequently updated to the v2 specification in August 2026, functions entirely as a grassroots community convention rather than an IETF standard5. Similarly, the Web Bot Auth architecture, authored at Cloudflare and Google, entered the IETF as draft draft-meunier-webbotauth-httpsig-protocol-02 with the explicit goal of achieving Best Current Practice status by late 2026, representing the bleeding edge of cryptographic bot identity prior to final ratification36.

19. MVP Execution Checklist

To immediately instantiate this comprehensive discoverability and security architecture, the initial deployment phase requires the meticulous execution of the following foundational engineering tasks:

• Generate and publish semantic HTML scaffolding for all core platform pages to guarantee immediate indexing by conventional search engines.
• Implement explicit robots.txt directives segmenting training crawlers from retrieval indexers, repeating rules to comply with RFC 9309 constraints.
• Deploy the standard sitemap-pages.xml alongside the highly specialized ai-sitemap.xml for machine artifacts to optimize crawler budgets.
• Configure the edge network or reverse proxy to inject RFC 8288 HTTP Link headers into all successful GET responses, actively advertising alternate formats.
• Publish the llms.txt and llms-full.txt files at the absolute domain root, adhering strictly to the August 2026 v2 formatting guidelines.
• Establish the /.well-known/api-catalog endpoint, formatted precisely as an RFC 9727 application/linkset+json document, indexing the primary OpenAPI specifications.
• Deploy JSON-LD markup across all agent profile pages, utilizing the WebApplication schema and linking execution via actionApplication.
• Configure the Web Application Firewall (WAF) to issue standard HTTP 429 Retry-After headers for excessive requests, implementing and enforcing exponential backoff limits.
• Apply rigid semantic schema validation to all user-submitted metadata to proactively mitigate baseline tool poisoning and indirect prompt injection attempts.

20. Future Opportunities

As the agentic web continues its rapid maturation, RogueSwarms is strategically positioned to adopt emerging architectures that shift machine discovery from centralized, inefficient scraping to federated, zero-configuration coordination. The IETF DAWN (Discovery of Agents, Workloads, and Named Entities) framework—specifically detailed in draft-zhang-dawn-agent-discovery-framework-01, published in September 2026—proposes a highly federated discovery architecture that separates local enumeration from metadata retrieval64. Adopting the DAWN specifications will allow RogueSwarms to advertise agent capabilities seamlessly across interconnected networks without relying on traditional crawler ingestion.

Furthermore, the integration of the IETF AIPREF (AI Preferences) standards, heavily active throughout 2026, will allow the platform to explicitly tag registry content with cryptographically verifiable usage grants66. This moves the platform beyond the blunt, binary access controls of robots.txt into highly granular licensing negotiations. As the underlying infrastructure evolves, RogueSwarms should also aggressively explore monetization strategies linked directly to agentic retrieval, leveraging standardized HTTP 402 Payment Required architectures to seamlessly charge autonomous systems for executing complex tools or accessing high-value, proprietary registry data68.

Works cited

1. Does Google Use llms.txt? No, and Here Is What It Said, https://llmtxt.info/does-google-use-llms-txt/

2. Does llms.txt Actually Matter for SEO?, https://araraseo.com/blog/llms-txt-seo-geo-what-actually-matters/

3. Google Now Tells You to Optimize for Generative AI \- Digital Applied, https://www.digitalapplied.com/blog/google-official-seo-docs-generative-ai-optimization-june-2026

4. Google says normal SEO works for ranking in AI Overviews and, https://searchengineland.com/google-says-normal-seo-works-for-ranking-in-ai-overviews-and-llms-txt-wont-be-used-459422

5. LLMs.txt in 2026: The Full Guide \- Limy.ai, https://limy.ai/blog/llms-txt-in-2026-the-full-guide

6. Making the API Evangelist Network Agent-Readable, https://apievangelist.com/blog/2026/05/16/making-the-api-evangelist-network-agent-readable/

7. Exploring api-catalog, a New API Discovery Standard \- Nordic APIs, https://nordicapis.com/exploring-api-catalog-a-new-api-discovery-standard/

8. RFC 9727: api-catalog: A Well-Known URI and Link Relation to Help, https://www.rfc-editor.org/info/rfc9727/

9. Tools \- What is the Model Context Protocol (MCP)?, https://modelcontextprotocol.io/specification/2025-11-25/server/tools

10. MCP Protocol Overview \- IBM, https://www.ibm.com/docs/en/quarkus/3.33.x?topic=architecture-mcp-protocol-messages-capabilities-lifecycle

11. Changes \- llms-txt, https://llmstxt.org/changes.html

12. llms.txt does nothing for Google Search \- Baseline Labs, https://baselinelabs.ai/blog/llms-txt-google-search

13. GPTBot vs OAI-SearchBot: which OpenAI bot decides ChatGPT, https://www.attensira.com/compare/gptbot-vs-oai-searchbot

14. GPTBot vs OAI-SearchBot \- Definition and Difference \- angeo.dev, https://angeo.dev/gptbot-vs-oai-searchbot/

15. OpenAI user agents — xSeek Docs, https://www.xseek.io/docs/openai-crawlers-and-user-agents

16. AI bots robots.txt guide: GPTBot, ClaudeBot | Soar Agency, https://www.soar.sh/blog/ai-bots-robots-txt-guide

17. Overview of OpenAI Crawlers, https://developers.openai.com/api/docs/bots

18. GPTBot and OAI-SearchBot do different things, and a lot of robots.txt, https://www.reddit.com/r/DigitalMarketing/comments/1vzmp3e/gptbot\_and\_oaisearchbot\_do\_different\_things\_and\_a/

19. What Is ClaudeBot? | GEO Glossary \- Presenc AI, https://presenc.ai/glossary/claudebot

20. The AI Crawlers Split Into Three. Most robots.txt Files Still Name One., https://www.sagentix.ca/blog/ai-crawlers-split-into-three

21. Make your WordPress content easier to find and use \- cybermaps, https://cybermaps.dev/features/

22. Helping agents discover my site search with an API Catalog, https://dri.es/helping-agents-discover-my-site-search-with-an-api-catalog

23. Home — Docsector Reader, https://docsector.com/

24. WebApplication Schema: Fields, Format, and Examples \- Karpi Studio, https://www.karpi.studio/schema-glossary-types/web-application

25. EntryPoint \- Schema.org Type, https://schema.org/EntryPoint

26. actionApplication \- Schema.org Property, https://schema.org/actionApplication

27. Tools \- Model Context Protocol （MCP）, https://modelcontextprotocol.info/specification/draft/server/tools/

28. Tools \- Model Context Protocol, https://modelcontextprotocol.io/specification/draft/server/tools

29. Architecture overview \- What is the Model Context Protocol (MCP)?, https://modelcontextprotocol.io/docs/2026-07-28/learn/architecture

30. Model Context Protocol: An essential standard for AI-powered tool, https://retool.com/blog/what-is-model-context-protocol

31. Model Context Protocol (MCP) Extensions for Network Equipment, https://www.ietf.org/archive/id/draft-zw-opsawg-mcp-network-mgmt-00.html

32. What is llms.txt? Why it's important and how to create it for your docs, https://www.gitbook.com/blog/what-is-llms-txt

33. OpenAPI Specification \- Version 3.2 \- Swagger, https://swagger.io/specification/v3.2/

34. The 2026 API Discovery Stack: OpenAPI \+ llms.txt \+ RFC 9727, https://unlockedcrm.ai/blog/openapi-llms-txt-rfc-9727-discovery-stack

35. Standard Endpoints | OpenREST: An Opinionated REST Contract, https://openrest.krotscheck.net/structure/standard-endpoints/

36. What is Web Bot Auth? Cryptographic identity for agents | Agent Ready, https://agent-ready.dev/what-is-web-bot-auth

37. Web Bot Auth \- Cloudflare Docs, https://developers.cloudflare.com/bots/reference/bot-verification/web-bot-auth/

38. Security.txt \- Wikipedia, https://en.wikipedia.org/wiki/Security.txt

39. Security.txt \- Data Standards Authority \- alphagov, https://alphagov.github.io/data-standards-authority/standards/securitytxt/

40. UHP vs ARD: Discovery Layer vs Harness Execution, https://unifiedharnessprotocol.dev/uhp-vs-ard/

41. Developers | CleanForge Window Cleaning, https://www.cleanforge.co.uk/developers

42. llms.txt file \- Guide for AI ranking, https://www.botrank.ai/technical-doc/llms-txt

43. Real llms.txt examples from leading tech companies (and what they, https://www.mintlify.com/blog/real-llms-txt-examples

44. llms.txt: A Practical Implementation Guide \- CDP.com, https://cdp.com/articles/llms-txt-guide/

45. openapisearch/docs/openapi-discovery.md at main \- GitHub, https://github.com/janwilmake/openapisearch/blob/main/docs/openapi-discovery.md

46. Your Swagger Endpoint Is More Dangerous Than /help Ever Will Be, https://medium.com/ckluis/your-swagger-endpoint-is-more-dangerous-than-help-ever-will-be-fa22cdb544bb

47. The Debian Administrator's Handbook, https://l.github.io/debian-handbook/pdf/fullcolor/en-US/debian-handbook.pdf

48. draft-meunier-webbotauth-httpsig-protocol-02 \- HTTP Message, https://datatracker.ietf.org/doc/draft-meunier-webbotauth-httpsig-protocol/

49. Fix Rate Limited Errors: 10 Proven Ways to Bypass 429 Codes, https://niuproxy.com/blog/fix-rate-limited-errors-guide/

50. What is a 429 error in web scraping? | Firecrawl Glossary, https://www.firecrawl.dev/glossary/web-scraping-apis/what-is-429-error-web-scraping

51. 429 Too Many Requests Error: Meaning and How to Fix It \- Decodo, https://decodo.com/faq/error-codes/error-429

52. (PDF) Prompt Injection LLM01:2026 \-The Primary OWASP Risk for, https://www.researchgate.net/publication/414269771\_Prompt\_Injection\_LLM012026\_-The\_Primary\_OWASP\_Risk\_for\_LLM\_Applications\_How\_attacks\_work\_what\_they\_put\_at\_risk\_and\_how\_security\_teams\_should\_respond

53. The Lethal Trifecta: How Indirect Prompt Injection Is Breaking, https://medium.com/@itpro677/the-lethal-trifecta-how-indirect-prompt-injection-is-breaking-agentic-ai-and-what-security-teams-c2ecba874ed1

54. Runtime Policy Enforcement for MCP-Based LLM Agents \- MDPI, https://www.mdpi.com/2079-9292/15/13/2829

55. MCP Tool Poisoning: How It Works and How To Prevent It, https://mcpmanager.ai/blog/tool-poisoning/

56. Model Context Protocol Threat Modeling and Analysis of ... \- MDPI, https://www.mdpi.com/2624-800X/6/3/84

57. A Systematic Security Benchmark and Playground for Testing Model, https://www.researchgate.net/publication/394688522\_MCPSecBench\_A\_Systematic\_Security\_Benchmark\_and\_Playground\_for\_Testing\_Model\_Context\_Protocols

58. MCPSecBench: A Systematic Security Benchmark and Playground, https://arxiv.org/html/2508.13220v3

59. Architecting a Secure, Intent-Centric Mobile Agent Operating System, https://www.alphaxiv.org/abs/2602.10915v2

60. Denial-of-Service Attacks on LLM-Based Agent Guardrails, https://arxiv.org/html/2606.14517v1

61. draft-meunier-web-bot-auth-architecture-05 \- IETF Datatracker, https://datatracker.ietf.org/doc/html/draft-meunier-web-bot-auth-architecture

62. Only Four API Providers Publish a Real .well-known/api-catalog, https://apievangelist.com/blog/2026/05/22/four-providers-publishing-well-known-api-catalog/

63. The Rise of Model Context Protocol (MCP) Skills \- Medium, https://medium.com/@adnanmasood/the-rise-of-model-context-protocol-mcp-skills-5f0d6a1c3579

64. reopen ADR-106 as a DAWN-aligned discovery plane, keep mDNS, https://github.com/ruvnet/ruflo/issues/3269

65. History for draft-zhang-dawn-agent-discovery-framework \-01, https://datatracker.ietf.org/doc/draft-zhang-dawn-agent-discovery-framework/history/

66. IETF AI Preferences Working Group \- | StandICT.eu, https://standict.eu/discussion-groups/artificial-intelligence/267/ietf-ai-preferences-working-group

67. AI Preferences (aipref) \- IETF Datatracker, https://datatracker.ietf.org/wg/aipref/

68. A Consent and Compensation Protocol for Agentic Web Access \- arXiv, https://arxiv.org/html/2609.11152v1