Executive Summary

Full report

UAIX.org currently defines standards and best practices for AI-to-AI and agent/bot-based web interactions (e.g. chatbots reading web pages). These include the UAI-1 message format and guidance on minimal GET-only access, static fallback content, and capability-adaptive browsing. However, UAIX has limited guidance for AI agent ↔ website interactions such as programmatic APIs or agent manifests. Key UAIX documents include UAI-1 (the open message format for agent exchange); Chatbot Access (a minimal GET-access model); Capability-Adaptive Web Interaction (HTML/JSON fallbacks by agent capability level); Agent Communication Operating Model (agent identity and logging); and Agent Consent Boundaries (rules for posting/auth). UAIX reports (e.g. cross-site audits, Project Handoff vs. LLM Wiki) provide context and remind that UAIX is the canonical UAI-1 authority. These sources emphasize authenticated, auditable AI exchanges and static fallbacks (e.g. 404 or JSON format for non-GET requests).

However, there are gaps when extending UAIX to AI Agent Website Support. UAIX covers how web pages degrade for simple bot visitors, but it lacks specifications for: rich API endpoints and schemas; agent discovery/manifest formats (akin to WebMCP or llms.txt); authentication models (beyond OAuth in MCP context); capability negotiation (how agents and sites advertise features); content negotiation (hinting at formats like JSON, JSON-LD, sitemaps); privacy/consent for AI access; telemetry/logging conventions for agent use; and safety fallback policies. For example, UAIX defines no standard “agent manifest” that a site could publish to advertise its AI-accessible tools or rate limits. It also provides little guidance on site-side telemetry for AI requests or how to negotiate API versions. In short, UAIX would need new sections on agent-friendly web APIs and machine-readable site capabilities.

To fill these gaps, we surveyed industry standards and best practices:

• WebMCP and agentic web standards: The proposed W3C WebMCP (Web Model Context Protocol) lets sites register client-side “tools” (functions) via navigator.modelContext.registerTool for agents to call. It complements server-side APIs (MCP) by letting sites expose actions directly in the browser. For agent discovery, there’s an emerging concept of an OpenMCP manifest – a static /manifest.json file describing available tools, auth requirements, and rate limits (like a “robots.txt for agents”). Google’s Web.Dev guide and other expert blogs recommend using semantic HTML and the browser’s accessibility tree so agents can parse UI elements reliably.

• Agent manifests: For example, Microsoft’s Copilot platform defines a Declarative Agent Manifest (JSON schema) with fields like name, description, instructions, capabilities, and actions to specialize an LLM’s behavior. While their manifest targets assistants, the idea of a machine-readable agent manifest is instructive. We should propose a parallel concept for sites: e.g. a JSON-LD AgentManifest or site manifest enumerating supported agent endpoints, required auth, and intents.

• APIs and data formats: Best practices favor JSON/JSON-LD with explicit schemas (e.g. OpenAPI specs for endpoints, JSON-LD for structured data) and using standardized vocabularies (such as schema.org WebAPI for APIs, or signed JSON Web Tokens for security). Sites should publish OpenAPI/AsyncAPI specs (e.g. as JSON) or embed JSON-LD metadata for agents, similar to publishing a sitemap for crawlers.

• Authentication: Use standard OAuth 2.0 flows with scoped tokens for agents. Define fine-grained scope or claim restrictions so that agents get only minimal privileges (e.g. read-only vs. write). Include an agent_id or client_type=ai-agent claim for logging. Consider API keys or mutual TLS for confidential agents, and public client flows for non-confidential ones. Provide endpoints for token exchange or dynamic client registration (per OAuth extensions).

• Privacy and consent: Follow standard web privacy norms: provide clear notice if data collection occurs (e.g. via a robots.txt-like file stating AI usage policy), allow opt-in/opt-out (via something like an ai-robot.txt or Access-Control-Allow-Agent header). Inform users if content served to an AI might not have explicit consent (e.g. personalize content only with user authorization). Possibly adapt GDPR/CCPA rules to automated requests (e.g. rate limit per user).

• Accessibility: Use proper semantic HTML and ARIA so that agents (which often leverage the accessibility tree) can interpret pages. Ensure server-side rendering or prerendering so AI crawlers see content without heavy JS. Label forms and buttons (<label>, role, autocomplete tags) so agents can map fields correctly.

• Rate limiting & error handling: Clearly communicate rate limits (e.g. via response headers or manifest) and return standard HTTP status codes (429 Too Many Requests, 403 Forbidden, etc.) with machine-readable payloads. Use techniques like exponential backoff for agents. On failures, offer safe fallbacks (e.g. alternate endpoints or static content).

Based on these, we recommend the following UAIX AI-Agent Website Support extensions:

1. Architecture Options: Describe web→agent interaction models:

• Pull Model: Agent requests data via HTTP APIs (REST/GraphQL) or static content (HTML/JSON).
• Push/Notification Model: Server sends events (e.g. WebSub, Webhooks, SSE) to subscribed agents.
• Tool Invocation Model: Browser-based APIs (WebMCP) where the site registers client-side tools that agents call.

Include a conceptual diagram (below) showing an agent sending HTTP requests (authenticated) to site APIs and receiving JSON/JSON-LD responses.

   sequenceDiagram
     actor Agent
     participant WebsiteAPI
     participant DB
     Agent->>WebsiteAPI: GET /api/tasks (Accept: application/json)
     WebsiteAPI->>DB: Query tasks (authorized)
     DB-->>WebsiteAPI: [{task data...}]
     WebsiteAPI-->>Agent: 200 OK (JSON payload)
     Agent->>WebsiteAPI: POST /api/tasks (with token)
     WebsiteAPI->>DB: Insert new task
     WebsiteAPI-->>Agent: 201 Created

(AI agent requests are authenticated and use JSON, with server returning structured data.)

1. API Schemas and Examples:

• JSON/JSON-LD schemas: Encourage publishing machine-readable schemas. For example, a site might serve an OpenAPI/JSON Schema at https://example.com/.well-known/agent-api.json. An example schema snippet (in JSON) or JSON-LD using schema.org/WebAPI could look like:

     {
       "@context": "https://schema.org/",
       "@type": "WebAPI",
       "name": "ExampleAgentAPI",
       "description": "API for AI agents to manage tasks",
       "documentation": "https://example.com/api-docs",
       "apiEndpoint": "https://api.example.com/v1/tasks",
       "httpMethod": "GET",
       "authentication": "Bearer token"
     }

• C# Data Models: Provide DTO classes for key objects. For example, an agent manifest:

     using System.ComponentModel.DataAnnotations;
     public class AgentManifest
     {
         [Display(Name="Version")]
         public string Version { get; set; }

         [Display(Name="Agent Name")]
         public string Name { get; set; }

         [Display(Name="Description")]
         public string Description { get; set; }

         [Display(Name="Capabilities")]
         public List<string> Capabilities { get; set; }
     }

Each [Display(Name="…")] uses a friendly title, omitting suffixes like "Id". Similarly, example method stubs (with XML comments) might be shown for content negotiation:

     /// <summary>
     /// Returns data in format requested by the agent.
     /// </summary>
     /// <param name="agentContext">Information about agent capabilities.</param>
     [HttpGet]
     public IActionResult GetData(AgentContext agentContext) { /* ... */ }

UAIX should list supported models (e.g. OAuth 2.0 with JWTs, API keys, or signed requests). We recommend OAuth 2.0 as primary (consistent with MCP), with scopes/claims to restrict agent access. For simplicity, also allow API keys (noting their limitations). If possible, support mutual TLS for confidential agents. Include examples: e.g. OAuth Bearer token in Authorization header. Specify HTTPS and TLS usage.

1. Authentication Models:

Define how an agent learns what the site can do. Propose an Agent Manifest file (e.g. /.well-known/agent-manifest.json or /agent-metadata.jsonld) that declares available tools/endpoints, rate limits, and auth requirements. For example:

1. Capability Negotiation and Manifests:

Agents fetch this manifest before using the site, enabling pre-flight discovery (as advocated in the OpenMCP manifest idea). Include a JSON example of such a manifest in the UAIX guidance.

Sites should support multiple response formats. At minimum, accept Accept: application/json and Accept: text/html. For structured data, return JSON or JSON-LD. Use HTTP content negotiation (e.g. Vary: Accept). Encourage using common vocabularies (JSON-LD with schema.org types). Provide an example header logic or code snippet illustrating choosing JSON if application/json is in Accept. Mention fallback: if an agent only allows GET (e.g. L0 clients), provide alternate static content (like a JSON dump of data or a JSON API).

1. Content Negotiation and Data Formats:

Define how agents should identify themselves (e.g. via a custom User-Agent or X-Agent-ID header). In API logs, tag requests from AI agents (as Curity suggests, e.g. via a claim client_type=ai-agent). Log which agent (if known) and timestamp to allow audit. UAIX might provide a telemetry schema (e.g. use OpenTelemetry conventions or JSON lines). Offer example log schema or fields (agent ID, endpoint, response code, latency).

1. Telemetry and Logging:

For unpredictable AI requests (invalid JSON, huge payloads), enforce input validation and rate limiting. Define safe HTTP error responses (e.g. 400 for bad JSON, 413 for too large). Provide a documented fallback policy: if an agent’s request fails (e.g. 503 Service Unavailable), it may retry or switch to a read-only mode. Consider a “safe mode” content version if advanced features fail.

1. Safety and Fallbacks:

Require sites to disclose AI access policies. This could be via a banner or API endpoint stating “This site supports AI agents under X terms.” UAIX could adapt robots.txt syntax for agents (e.g. allow/disallow certain paths to agents) or introduce an AI-Policy header in responses. Ensure privacy notices (e.g. cookie use) apply to agent requests. If agents act on behalf of users, confirm user consent for any data sharing in the API flows.

1. Consent and Notice:

Point out that agent-friendly design is the same as general accessible design. Cite W3C ARIA guidance: use semantic elements (<button>, <nav>, <label>) so agents recognize functionality. Ensure non-visual content has accessible equivalents (alt text, ARIA labels for dynamic widgets) to aid agents using the accessibility API. UAIX should refer to existing W3C accessibility standards as part of its AI-readiness checklist.

1. Accessibility:

1. Testing, Validation, and Deployment:

• Validation: Encourage CI/CD tests using both human and agent-based frameworks (e.g. use Microsoft Playwright’s AI agent features that locate elements by accessibility roles).
• Linting: Provide JSON Schema for manifests and API requests for automated validation.
• Performance: Test under AI load (concurrent API calls).
• Checklist: UAIX can publish a deployment checklist covering: TLS active, CORS set correctly (to allow agents), OpenAPI/JSON-LD published, robots.txt/agent-manifest present, OAuth configured. Use UAIX versioning for these pages (e.g. “UAIX-WEB-01”).

Sample Agent Manifest (Table)

Example JSON-LD API Metadata

<script type="application/ld+json">
{
  "@context": "https://schema.org/",
  "@type": "WebAPI",
  "name": "ExampleShop AI API",
  "description": "API for AI agents to search and order products",
  "documentation": "https://example.com/docs/ai-api",
  "authentication": "OAuth2 Bearer",
  "endpoint": "https://api.example.com/v1/products/search"
}
</script>

Prioritized Implementation Roadmap

We recommend UAIX publish these extensions iteratively, with milestones and effort estimates:

Risks: New standards like agent manifests depend on agent vendors adopting them (interoperability risk). Security (OAuth misconfiguration) could expose data. Too many options may confuse implementers; the guidance must be clear.

Timeline: A Gantt-style timeline (illustrative):

gantt
    title AI Agent Website Support Rollout
    dateFormat  YYYY-MM
    section Specification
    AgentManifest  :a1, 2026-07, 1M
    OAuth/APIGuide :a2, after a1, 1M
    SchemaExamples :a3, 2026-09, 1M
    section Implementation
    ConsentPolicy  :a4, 2026-10, 1M
    TelemetryGuide :a5, 2026-10, 1M
    AccessibilityList :a6, 2026-11, 1M
    TestingKit     :a7, 2026-12, 2M

(Timeline shows sequential release of new guidance from mid-2026 onward.)

References

• UAIX standards and guides (UAI-1 spec, Chatbot Access, GET-Action, etc.)
• Google Web.Dev “Build agent-friendly websites” (semantics/ARIA guidance)
• Microsoft declarative agent manifest schema
• W3C WebMCP Community Group spec
• Jeremy Howard, llms.txt proposal (AI manifest file)
• Curity / Nango (API auth best practices)
• Additional readings: Schema.org WebAPI type, OAuth 2.0 RFC, W3C ARIA spec.