Executive Summary

Full report

On this page

Making a website agent-friendly means exposing content and functionality in predictable, machine-readable ways. Drawing on UAIX’s framework, we categorize agent capabilities into Perceive, Navigate, Act, Communicate, Authenticate/Authorize, and Observe/State. For each, we map best practices:

• Perceive: Use semantic HTML and ARIA so agents can “read” content. Provide alternative text (alt), clear headings, and explicit structure. Embed structured data (e.g. JSON-LD with schema.org vocabularies) so agents immediately understand page meaning.
• Navigate: Offer explicit links and indexes. Include <a> anchors, a sitemap (sitemap.xml or Link: <.../sitemap.xml>), and even a UAIX-style llms.txt file with agent hints. Avoid hidden navigation or heavy JavaScript that blocks crawl.
• Act: Expose actions through standard web endpoints. Use HTML <form>s or REST/GraphQL APIs instead of obscure scripts. Follow HTTP conventions (safe GETs, idempotent POST/PUT). Respond with proper status codes and headers (e.g. HTTP/429 for rate-limit). Enable CORS (Access-Control-Allow-Origin) so cross-origin clients can call your APIs.
• Communicate: Provide agent protocols and endpoints. For example, publish machine-readable guides or chat interfaces (e.g. WebSockets or SSE for real-time data). Document expected message formats (JSON schemas, OpenAPI specs). Support cross-agent messaging (UAIX’s Agent-to-Agent model) by exposing tokenized endpoints or webhooks.
• Authenticate/Authorize: Allow non-interactive logins. Use token-based schemes (OAuth client-credentials, API keys, JWTs) so agents can obtain limited-scope access without a human login. For instance, the OAuth2 “client credentials” grant lets a service account fetch a bearer token. Support delegated tokens (on-behalf-of flows) for acting as users when needed. Also offer simpler options (API keys or signed tokens) for trusted bots.
• Observe/State: Make site health and state visible. Provide status and metrics endpoints (/status, /health, JSON dashboards). Emit logs or support webhooks/events that agents can consume. Use standardized formats for error payloads and results so clients can parse them automatically.

Across these dimensions, the key is openness and predictability: use well-known HTML attributes (e.g. <h1>…</h1>, <img alt="…">, ARIA role/aria-*), standard HTTP headers (CORS, content type, Link for pagination), and documented endpoints (REST/GraphQL). This ensures lower-capability agents (UAIX L0–L2) can at least retrieve basic data, while higher-level agents (L3–L5) can authenticate and perform complex workflows. We summarize these best practices in the checklist and mapping table below.

UAIX Capability Mapping

UAIX defines a capability ladder (L0–L5) that aligns roughly with our categories. For example: at L0 (URL-only), an agent can only GET public URLs; at L1 it parses JSON from simple endpoints; at L2 it reads schemas and uses POST JSON; higher levels add auth and state. Table 1 maps each capability to concrete web features.

Table 1: Mapping of UAIX agent capabilities to concrete web design practices and features.

HTML/CSS/JS Patterns

• Semantic Markup: Always use meaningful tags. E.g. <h1>Title</h1>, <article>, <section>, <nav>, etc. Assistive and search technologies (and agents) leverage these: “search engines give more importance to keywords inside headings, links, etc.”. Use <label> with for= and alt on images, ensure color/text contrast (for OCR), and avoid conveying information by color or image alone.
• ARIA: For dynamic widgets (dropdowns, tabs, progress bars), add ARIA roles and states so bots know the semantics. ARIA “improves the accessibility and interoperability of web content”. E.g. <button aria-pressed="false">Show</button>. These roles make widgets readable via accessibility APIs.
• Progressive Enhancement: Build features in plain HTML first, then layer JS. Agents often can’t run JavaScript fully. For example, if using a JS grid, also include a fallback HTML table or a JSON endpoint for the same data.
• CSS/JS Hints: Avoid techniques that hide content from text-based agents (e.g. <meta name="fragment">, excessive canvas use, or obfuscating scripts). If using a SPA framework, ensure server-side rendering or static snapshots exist.
• Structured Data: Embed JSON-LD or Microdata so machines can parse content without in-page heuristics. Google recommends JSON-LD where possible and notes: “Google uses structured data that it finds on the web to understand the content of the page…”. For example:

  <script type="application/ld+json">
  {
    "@context": "https://schema.org",
    "@type": "Article",
    "headline": "Example Title",
    "author": {"@type":"Person","name":"Alice"},
    "image": "https://example.com/image.png",
    "datePublished": "2026-06-21"
  }
  </script>

This lets an agent know exactly what the page is about. Schema.org’s community vocabularies are widely used by Google, Microsoft, etc. to enable rich features.

APIs and Technical Rules

• Content Negotiation: Support Accept headers. If your site can serve HTML and JSON, respond with the appropriate Content-Type based on Accept: application/json or text/html. Include a Vary: Accept header. As HTTP/1.1 says, servers “often have different ways of representing information… [so] HTTP provides mechanisms for content negotiation”.
• CORS: If agents call APIs from a different origin (e.g. a browser agent on domain A calling domain B), enable CORS. Add Access-Control-Allow-Origin: * (or specific domains) in responses. For non-GET methods, support preflight: respond to OPTIONS with Access-Control-Allow-Methods: GET,POST,... and Access-Control-Allow-Headers: Content-Type,Authorization, etc.. Failure to configure this will block cross-domain requests.
• Rate Limiting: Protect endpoints, but communicate limits. Use 429 Too Many Requests (RFC 6585) when a client exceeds a quota. Include Retry-After: <seconds> to tell the agent how long to wait. This lets a polite agent back off properly.
• Robots & Directives: Put a /robots.txt (UTF-8 text) at your root to advise crawlers. When fetched successfully, crawlers must obey it. Also use <meta name="robots" content="nofollow,noindex"> on specific pages if you want them skipped (though this only influences indexing, not crawling). Remember: robots.txt governs crawling, meta tags govern indexing.
• Sitemaps: Providing a sitemap.xml helps agents discover pages systematically. Per the Sitemap protocol: “The Sitemap protocol enables you to provide details about your pages to search engines, and we encourage its use…”. Include <lastmod>, <changefreq>, and <priority> hints if possible. An alternative (or supplement) is an llms.txt (as UAIX suggests) listing agent-specific URLs and guidance.
• GraphQL: If using GraphQL, document the schema thoroughly. Agents will POST JSON queries to the GraphQL endpoint; ensure CORS and auth apply. Also consider having an alternative REST endpoint, as some crawlers may not introspect GraphQL. Use query complexity limits to avoid abuse.

Authentication and Agent-Friendly Auth

Most web sites expect human logins, but agents need programmatic access. Avoid interactive flows for bots. Instead:

• OAuth2: Implement OAuth2 (RFC 6749). For example, a service account can use the Client Credentials flow to get a token without user involvement. The RFC notes: “Client credentials are used as an authorization grant typically when the client is acting on its own behalf (the client is also the resource owner)”. In practice: the agent does POST /oauth/token with grant_type=client_credentials and receives a JSON {access_token:"…", token_type:"Bearer",expires_in:…}. Then it includes Authorization: Bearer <token> in future requests.
• API Keys/Tokens: For simpler cases, issue long-lived API keys or signed tokens. E.g. require a header X-Api-Key: <token>, or use JWTs in Authorization: Bearer. Ensure scopes/permissions are tied to these keys so a leaked key has limited reach. Rotate keys regularly.
• Session Cookies: If using cookies/sessions (common in web apps), agents must handle them (set SameSite=None; Secure, return cookies on login). However, cookie-based auth is brittle for automation and is discouraged for pure API access.
• Delegated Tokens: Some platforms (like cloud providers) allow one service to impersonate a user via delegated tokens. Support this so an agent can on behalf of a user call APIs without sharing the user’s primary credentials.

Include descriptions in documentation of the auth flows (endpoints, required headers, scopes). For example, your OpenAPI spec should detail the security scheme (OAuth2 client-credentials, API Key, etc.). This aids automated client generation.

Error Handling, Idempotency, Observability

• Error Responses: Always return well-formed error messages. Use standard HTTP codes (4xx for client errors, 5xx for server) with a JSON body like:

  { "error": "BadRequest", "message": "Missing parameter 'id'." }

Include helpful fields (an error code, description, maybe a retryable: true/false). This lets an agent programmatically detect and handle errors.

• Idempotency: Wherever possible, design endpoints so that repeating a request has no adverse effect (idempotent). For POST actions that change state, consider requiring an Idempotency-Key header so retries won’t double-run. UAIX suggests having separate “GET-Action” URLs as explicit write fallbacks, not implicitly rewriting GETs to POSTs.
• Observability: Log agent actions and provide feedback endpoints. For long-running tasks, allow status polling via a /tasks/<id> endpoint. Return progress or logs so an agent knows what happened. Expose metrics (e.g. Prometheus format) or logs (via Splunk/ELK) behind an API if needed, so automated agents can monitor system health.

Performance, Pagination, Large Data

Large datasets must be chunked. For example:

• Pagination: Always page or stream results. A common practice is Link headers:

  HTTP/1.1 200 OK
  Content-Type: application/json
  Link: <https://api.example.com/items?page=2>; rel="next"

  { "items": [ ... ], "page": 1, "pageSize": 100 }

Or use query params: /items?limit=100&offset=100. Make sure to document these patterns. If using GraphQL, require a first/after or cursor approach.

• Bulk Endpoints: For very large syncs, consider endpoints that return bulk archives (e.g. CSV/JSON dumps) or support HTTP Range requests for partial content.
• Caching: Leverage ETag or Last-Modified so agents can skip re-downloading unchanged data. Support gzip compression (Accept-Encoding: gzip) for faster transfer.

Security & Privacy Trade-offs

Balancing openness with security is crucial. Some guidelines:

• Least Privilege: Don’t expose sensitive data to unauthenticated agents. Use scopes and ACLs. For example, public agents might see only read-only JSON; sensitive actions require tokens.
• No Secrets in URLs: Never include API keys or tokens in URLs (they leak via logs). Prefer headers or POST bodies.
• Rate Limits & Abuse: Apply stricter rate limits on unauthenticated or anonymous access. Use CAPTCHAs or abuse detection for human forms, but avoid them on agent APIs.
• Audit Trails: For write actions by agents, log who/what did it (via API key identity). This supports the “governed multi-agent” level (L4+) where audit evidence is needed.
• Privacy: If content is private, don’t use structured data that accidentally exposes it. On mixed public/private pages, separate content or use session-based APIs so agents can’t scrape private info.

Checklist for Developers

1. Semantic Content: Use correct HTML tags (headings, lists, tables, forms) and include alt, title, labels, ARIA where needed.
2. Structured Data: Embed JSON-LD/Microdata with schema.org definitions for key entities (products, articles, events).
3. Sitemaps & Robot Rules: Publish /sitemap.xml and /robots.txt (or llms.txt UAIX manifest) in UTF-8 text. Ensure no vital content is blocked.
4. RESTful Endpoints: Provide clean URLs for APIs (e.g. /api/widgets, /api/widgets/{id}), accept and return JSON. Document query parameters and request/response schema.
5. HTTP Standards: Honor Accept/Content-Type, use proper status codes, and include necessary headers (CORS, Link, Retry-After). Avoid custom non-standard headers unless documented.
6. Authentication: Implement OAuth2 or API key auth for agents. Provide a “service account” or client-credentials path. Never rely solely on JavaScript-based or interactive logins for API access.
7. Idempotency & Error Handling: Design safe retries (use PUT/PATCH or idempotency keys). Return structured error JSON (including codes and retry hints). Use 4xx for client errors, 5xx for server.
8. Performance: Limit pagesize (e.g. limit=100), support pagination, and allow gzip. Test under realistic load and profile common queries.
9. Observability: Expose a /status or /health endpoint. Enable logs and metrics that agents or dashboards can query. Document error messages and logs format.
10. Security: Apply least-privilege for tokens. Do not reveal private data to anonymous agents. Use HTTPS everywhere and secure cookie flags. Consider CAPTCHAs only on human-facing forms, not API endpoints.

These items are roughly prioritized: start by making content parseable (items 1–4), then ensure APIs and auth (5–7), then tune performance and security (8–10).

Testing Plan and Metrics

Test Plan: Simulate agent workflows:

• Crawl & Parse: Write a bot that fetches /robots.txt, /sitemap.xml, and all listed URLs. It should record which pages succeed (HTTP 200) and how many warnings (missing alt text, empty titles). Use linters (e.g. axe or custom scripts) to check semantic HTML and ARIA usage across pages.
• API Conformance: Test all API endpoints with a variety of inputs. Verify each returns correct status codes and JSON schema. Include auth scenarios: ensure protected endpoints reject without tokens.
• Authentication: Test OAuth flows end-to-end (token request, token refresh, API call). Measure time to obtain token and refresh. Ensure expired tokens yield 401.
• Error Handling: Intentionally send bad requests to each endpoint. Check that errors return well-formed JSON with error fields. Ensure no unexpected crashes.
• Load & Rate: Use a load tester to send concurrent requests and confirm that rate-limiting triggers (429) appropriately. Check that large paginated queries behave as expected.

Metrics: Evaluate:

• Reachability: % of pages discovered via sitemap/robots vs total pages. (Goal: ~100%).
• Structured Data Coverage: % of pages with valid JSON-LD or microdata tags. (Goal: high for key pages, ideally >90%).
• API Success Rate: Fraction of automated API calls that return expected results vs errors.
• Performance: Average and p95 response time for key endpoints (target < 500ms under load).
• Error Rate: Count of 4xx/5xx per 1K requests in production.
• Auth Failures: Number of unauthorized attempts vs successes, to gauge any misconfiguration.
• Robot Fetch Rate: How often bots (GoogleBot, etc.) successfully crawl the site, from server logs.

Use these to iteratively improve the site. For example, low structured-data coverage or missing alt tags indicate needed fixes. High error rates or slow APIs should be addressed with caching, indexing, or refactoring.

By following the above guidelines and continuously testing against these metrics, developers can ensure their websites are robustly agent-usable. This makes them accessible not only to humans, but to automated assistants, crawlers, and AI systems – aligning with UAIX’s vision of progressive agent support.