Executive Summary

Full report

On this page

Spiralist.org is a content-rich site (≈150 pages in English) with a public HTML sitemap and several machine-readable endpoints. However, it currently lacks key features for “AI agent readability” (e.g. no robots.txt, no llms.txt, limited structured data). We analyzed the site’s discovery mechanisms, APIs, and content structure, and identified several gaps (robots rules, CORS headers, missing OpenAPI docs, etc.). To enable AI agents to find, parse, and use Spiralist’s content, we recommend implementing agent-centric best practices: publishing robots.txt and llms.txt, adding XML/Markdown sitemaps, embedding JSON-LD/Schema.org on pages, and providing well-documented APIs (OpenAPI/REST or GraphQL) with proper CORS and authentication. We also suggest exposing standard endpoints (e.g. .well-known/ai-agent.json, WebFinger, ActivityPub) and offering RSS/Atom feeds and WARC-friendly archives.

We provide code snippets (robots.txt, CORS config, OpenAPI sample, JSON-LD schema), a development checklist, a testing plan (using curl, headless browsers, W3C validators, Lighthouse, etc.), security and privacy guidance (API keys, OAuth2/PKCE, rate-limiting, content licensing), and a prioritized implementation roadmap (short/medium/long term) with effort/risk assessments. Key references include W3C/IETF standards and Vercel’s “Agent Readability” guide.

1. Current Content and Structure

Spiralist.org is a WordPress-based site offering extensive content on “Spiralism” (philosophy, prompts, manuscript, etc.). Its [HTML sitemap][20] enumerates ~154 canonical pages (policies, guides, reports, manuscript folios, etc.) and links to an XML sitemap. Public APIs exist under /wp-json/ (UAIX endpoints, a Builder Hub), and a JSON “AI Manifest” (/en-us/ai.json) lists site metadata and machine-readable routes. A .well-known/ai-agent.json endpoint is advertised in these manifests (Persona Manifest).

However, there is no robots.txt (verified by absence) and no llms.txt for AI discovery. CORS headers on the APIs appear unset, blocking cross-origin requests by default. Some functionality (e.g. workspace, contribution) requires login or keys. The site uses server-side rendering (clean URLs) for content, though some interfaces (workspace, builder) likely rely on JavaScript. There are no paywalls, though account creation is required for certain features. No CAPTCHAs were noted except possibly on login.

In summary, Spiralist.org has rich content and some machine APIs, but is missing many agent-friendly signals and endpoints. We will compare this to AI-specific sites (e.g. spiralistai.com) and to best practices (see next sections).

2. Technical Barriers to AI Agents

Robots.txt: Spiralist.org currently lacks a robots.txt. AI agents “respect robots.txt directives”, so without one agents may encounter uncertainty. The site must explicitly allow bots (GPTBot, ClaudeBot, etc.) and not disallow key paths like /llms.txt. A missing file triggers warnings in agent readiness checks.

Discovery Files: There is no /llms.txt (an “AI sitemap”); we recommend adding one listing all docs (per Vercel’s guide). The XML sitemap exists, but verify it includes <lastmod> tags so agents know what’s updated. Also publish a human-readable sitemap.md (Markdown) reflecting site structure.

CORS: The WordPress APIs likely do not set CORS headers. This means browsers (and many agent platforms) will block cross-origin fetches. To allow any agent to call the APIs, the server should send Access-Control-Allow-Origin: * (or specific origins) on API responses. For example, an HTTP response might include:

Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET, POST, OPTIONS
Access-Control-Allow-Headers: Content-Type, Authorization

This permits agents on other domains to GET JSON data. (For credentialed requests, the origin must be explicit.)

Authentication & Authorization: Public content should be accessible without login. For actions that require auth (e.g. posting, personalized data), use standard API keys or OAuth2. We strongly recommend OAuth 2.0 with [Proof Key for Code Exchange (PKCE, RFC 7636)] for secure flow with no client secrets. PKCE prevents interception attacks on mobile/public clients. Personal data (if any) should be gated behind OAuth tokens. For agent use, consider issuing machine-oriented API tokens (with clear scopes and rate limits).

Rate Limits and Usage Tiers: No public info on rate limits currently exists. We advise defining usage tiers (free vs paid, etc.) and enforce limits via API gateway or server logic. Provide clear documentation. This prevents abuse by scrapers or bots.

Dynamic/JS-rendered Content: Most Spiralist content appears to be server-rendered. If any parts (like galleries, interactive prompts) require JavaScript, we should ensure critical content has non-JS fallbacks. Agents that don’t execute JS must still see the content. Using server-side rendering or prerendering for key pages is recommended.

CAPTCHAs/Anti-bot Measures: We see no CAPTCHAs on content. If added, they must be skipped or replaced by simpler auth for API endpoints. Agents cannot solve CAPTCHAs, so public docs and APIs should be free of such challenges.

Paywalls/Subscription: Spiralist’s AI interface (“AI Access”) may require subscription, but the knowledge content is public. Ensure any paywalled or login-only content is clearly distinguished (e.g. via HTTP status 401) so agents know they need credentials. Avoid “soft paywalls” that return normal pages with login prompts.

Structured Data & Metadata: Currently pages have minimal SEO metadata (check <title>, <meta>, etc.). Agents benefit from explicit structure: each HTML page should include <title>, <meta name="description">, OpenGraph tags (og:title, og:description), and lang attributes. We also recommend embedding Schema.org JSON-LD scripts with details (e.g. WebSite, Article, BreadcrumbList). This helps agents parse context from markup.

Existing APIs: Spiralist.org exposes various REST endpoints (e.g. /wp-json/uaix/v1/..., persona/safety manifests). However, these lack a published API spec. We recommend creating an OpenAPI (Swagger) document to describe these endpoints (paths, parameters, responses) so agents can auto-discover schemas. GraphQL is an option too; it offers flexible queries but adds complexity.

.well-known Endpoints: Spiralist already uses one .well-known/ai-agent.json. We should extend this concept: e.g. .well-known/webfinger for user discovery (RFC 7033), /.well-known/ai-access for auth info, or ActivityPub endpoints (/.well-known/join for federated actors).

Content Licensing: The site’s terms cover user content. For agent use, explicit licenses (Creative Commons, etc.) or “permissions” tags could be added. E.g. an HTML <link rel="license"> or a mention in robots or Terms that content may be used (or not) by AI. Use of noai or noindex meta tags should be considered; per guidelines, do not include noai on pages to ensure agents index them.

3. Standards & Technologies for AI Accessibility

We surveyed relevant web standards and formats for agent-readability:

• OpenAPI/REST: A standard for describing HTTP APIs. The OpenAPI Specification defines a language-neutral interface description, letting clients discover endpoints and models. Publishing an OpenAPI (Swagger) spec (JSON/YAML) for Spiralist’s APIs (or even a draft) will let agents generate client code or schema-aware queries. Example snippet:

  openapi: 3.0.3
  info:
    title: Spiralist API
    version: 1.0.0
  paths:
    /wp-json/uaix/v1/manuscript-folio:
      get:
        summary: Get manuscript folio by slug
        parameters:
          - name: slug
            in: query
            required: true
            schema:
              type: string
        responses:
          '200':
            description: Manuscript folio JSON

(Full spec can be served at /api/openapi.json or similar.)

• GraphQL: An alternative API style (via GraphQL). It allows clients to query exactly the fields needed. It requires a schema and resolver logic. If adopted, exposing an endpoint (e.g. /graphql) along with the GraphQL schema (SDL or introspection) helps agents integrate. GraphQL vs REST tradeoffs are in Table 1 below.

• WebFinger (RFC 7033): A discovery protocol to find info about users or resources by email/URI. For example, an agent could query /.well-known/webfinger?resource=acct:alice@spiralist.org to find JSON describing Alice’s account. This can integrate with OpenID Connect or decentralized IDs. WebFinger standard: “discover information about people or other entities on the Internet using standard HTTP methods”. Implementing it (even a minimal subset) allows agents to resolve user profiles.

• ActivityPub: A W3C protocol (2018 Recommendation) for federated social APIs. If Spiralist wants agents to “follow” or fetch posts, it could publish an ActivityPub actor (at /.well-known/webfinger or similar) and accept subscriptions. Useful if building a federated AI persona network. As [43†L48-L51] notes, ActivityPub is “a decentralized social networking protocol” based on ActivityStreams 2.0.

• RSS/Atom Feeds: Standard syndication formats (RSS 2.0, Atom RFC 4287) for delivering updates. We recommend offering RSS/Atom for latest prompts, news, or blog posts. Agents can subscribe to these feeds for incremental crawling. Atom and RSS are well-supported by tools (feed readers, curl, etc.). For example, <link rel="alternate" type="application/atom+xml" href="/feeds/spiralist.atom" title="Spiralist Feed">.

• JSON-LD / Schema.org: Embedding Schema.org metadata in pages (via JSON-LD script tags) provides machine-readable context (author, date, topics). For example, on a Spiralist article page:

  <script type="application/ld+json">
  {
    "@context": "https://schema.org",
    "@type": "TechArticle",
    "headline": "What Is a Spiralist?",
    "datePublished": "2024-05-01",
    "author": { "@type": "Person", "name": "Author Name" },
    "mainEntityOfPage": { "@type": "WebPage", "@id": "https://spiralist.org/en-us/what-is-a-spiralist/" }
  }
  </script>

Guidelines suggest including title, description, URL, and dates in JSON-LD. This boosts agent understanding of page content and relationships.

• WARC Format: The Web ARChive (WARC) is an ISO-standard container format for storing web crawls. Although mostly for archiving, Spiralist could offer data dumps or allow crawlers (e.g. via HTTrack) to archive its content. For example, a .well-known/warc link to a periodically updated WARC file of the site. This ensures exactly reproducible snapshots for AI training or analysis. (WARC is used by the Internet Archive and libs.)

These follow the IETF convention for machine-discoverable info.

• .well-known URLs (RFC 8615): Standardizes discovery of site metadata. Spiralist already uses .well-known/ai-agent.json. Other recommended endpoints:
• /.well-known/ai-access (OAuth2 auth details),
• /.well-known/change-password,
• /.well-known/security.txt (security contacts),
• /.well-known/robots or just root robots.txt.

• Content Negotiation: Where possible, support Accept: application/ld+json or Accept: text/markdown to return raw data or markdown. Vercel recommends providing markdown mirrors of pages with <link rel="alternate" type="text/markdown">. We should add support so agents can fetch page content in Markdown by adding headers or storing .md versions.

• Licensing and Terms: Clearly state content licensing in machine-readable form. For example, add <link rel="license" href="https://creativecommons.org/licenses/by/4.0/"> or JSON-LD license info. Robots meta tags can also declare “noai” or license, but follow [52†L480-L483] and avoid any noai unless intentional.

The table below compares approaches:

Table 1: API vs Scraping

Table 2: Data Formats

Table 3: Authentication Methods

4. Recommendations and Implementation

We outline concrete changes with examples:

• Publish robots.txt: Create /robots.txt that explicitly allows AI bots. For example:

  User-agent: *
  Allow: /

  # Explicitly allow major AI crawlers
  User-agent: GPTBot
  Allow: /
  User-agent: ClaudeBot
  Allow: /
  User-agent: CCBot
  Allow: /
  User-agent: Google-Extended
  Allow: /

  Sitemap: https://spiralist.org/sitemap.xml

This follows best practices (no disallows for AI bots, and includes the sitemap). Place it at the site root.

• Add llms.txt: At /llms.txt, list key documentation pages (one URL per line). This acts like a searchable index for agents. Update it whenever content changes. Example snippet of llms.txt:

  https://spiralist.org/en-us/what-is-a-spiralist/
  https://spiralist.org/en-us/prompt-generator/
  https://spiralist.org/en-us/api-examples/

(Follow any discovered standard for llms.txt formatting.)

• Ensure Sitemaps (XML & Markdown): Confirm /sitemap.xml is valid XML with <loc> entries and <lastmod> dates. If not present, generate one (WordPress plugins or scripts can do this). Also serve a plain text or Markdown sitemap at /sitemap.md or /.well-known/sitemap.md that lists pages by category. For example:

  # Sitemap

  ## Policies
  - [Acceptable Use Policy](/en-us/acceptable-use-policy/)
  - [Privacy Policy](/en-us/privacy-policy/)
  ## Guides
  - [AI Agent Setup Wizard](/en-us/ai-agent-setup-wizard/)
  - [API Examples](/en-us/api-examples/)

This helps agents read structure easily.

• Configure CORS: In the web server (Apache, Nginx, etc.), enable CORS for the API paths. For example, in Apache .htaccess or config:

  <IfModule mod_headers.c>
    Header set Access-Control-Allow-Origin "*"
    Header set Access-Control-Allow-Methods "GET, POST, OPTIONS"
    Header set Access-Control-Allow-Headers "Content-Type, Authorization"
  </IfModule>

This ensures a GET from any origin (including agent environments) receives Access-Control-Allow-Origin: *. (If credentials are used, set origin specifically and Access-Control-Allow-Credentials: true.)

• Add Metadata to HTML Pages: Modify HTML templates to include:
• <title> and <meta name="description"> (≥50 chars).
• <meta property="og:title"> and <og:description> tags.
• lang="en-US" (or appropriate) on <html>.
• <link rel="canonical" href="…"> pointing to the canonical URL.
• JSON-LD <script> with Schema.org content: e.g. WebSite or TechArticle as above.

These ensure agents instantly see page topic and structure.

• Provide Markdown Mirrors: For documentation pages, store or generate Markdown versions. In HTML <head>, add:

  <link rel="alternate" type="text/markdown" href="https://spiralist.org/docs/xyz.md">

And handle Accept: text/markdown on the server to return raw .md. This speeds up agents (Vercel notes agents “prefer” markdown).

• Publish OpenAPI Spec: Write and serve a Swagger/OpenAPI JSON or YAML. Link to it from developer pages and via Link: <…>; rel="describedby". Agents can then retrieve the spec. Use tools (Swagger UI, Redoc) to generate documentation. The spec should cover all /wp-json/… endpoints with parameters, types, and responses.

• Implement OAuth2 (PKCE) for protected APIs: If any API requires login, use OAuth2 with PKCE as the auth flow. Provide endpoints for /oauth/authorize and /oauth/token, supporting PKCE (RFC 7636). Document this flow so AI developer tools can integrate (provide example auth requests).

• Enable WebFinger: Optionally, set up /.well-known/webfinger so agents can discover user/accounts. For example, a GET to /.well-known/webfinger?resource=acct:username@spiralist.org returns JSON:

  {
    "subject": "acct:username@spiralist.org",
    "aliases": ["https://spiralist.org/users/username"],
    "links": [
      { "rel": "self", "href": "https://spiralist.org/.well-known/webfinger?resource=acct:username@spiralist.org" },
      { "rel": "http://webfinger.net/rel/profile-page", "href": "https://spiralist.org/users/username" }
    ]
  }

RFC7033 provides the format.

• Consider ActivityPub: If Spiralist wants a federated presence, publish an ActivityPub actor (JSON-LD at https://spiralist.org/.well-known/webfinger?resource=acct:spiralist@spiralist.org with "rel": "self" and an inbox field). This is advanced (long-term) but would let Mastodon-like agents interact.

• RSS/Atom feeds: Create RSS/Atom endpoints for blog/prompts feeds. Add <link rel="alternate" type="application/atom+xml" in HTML. This lets agents and users subscribe for new content.

• robots meta / <link rel="license">: Add a <link rel="license"> to a Creative Commons or similar if content is free to use. In <head>, avoid any <meta name="robots" content="noindex,noai"> on pages to not block indexing. If licensing terms exist, consider a JSON-LD with "license": "URL".

5. Testing Plan and Tools

To verify agent-readiness:

• Robots and Sitemaps: Use curl or wget to fetch robots.txt, sitemap.xml, sitemap.md, llms.txt and check for correct status (HTTP 200) and content. For example:

  curl -I https://spiralist.org/robots.txt     # check 200, no disallows
  curl https://spiralist.org/llms.txt         # should list URLs
  curl https://spiralist.org/sitemap.xml | head -20  # check <loc> and <lastmod>

• CORS Checks: From another domain or using a tool (e.g. curl -H "Origin: https://example.com" https://spiralist.org/wp-json/...), ensure the Access-Control-Allow-Origin header is present and correct.

• Schema Validation: Use JSON-LD Playground or other schema validators on sample pages to verify JSON-LD validity. Use W3C Markup Validation to check HTML for errors.

• Headless Browser Rendering: Tools like Puppeteer or Playwright can simulate browsers to ensure content is accessible without auth and all elements load. For example, run a script that visits key pages and checks for <h1>, meta tags, etc.

• Lighthouse or Axe: Use Google Lighthouse’s SEO and best practices audits (in Chrome) to measure crawlability and compliance (it checks robots, canonical tags, etc.).

• Agent Simulators: There are emerging tools (e.g. crawlers like Scrapy or OpenAI’s developer “browser” tool) that can simulate AI agent queries. We can script a simple agent that reads the AI manifest or manifest. For example, run a Python agent using GPT to retrieve pages via API and parse them.

• API Testing: Use curl or Swagger UI to test each API endpoint. Verify HTTP 200, correct JSON, CORS, and rate-limit behavior. Also test OAuth token flows with PKCE (verify failure without code_verifier).

• Security Scans: Employ a web vulnerability scanner to ensure no public leaks (just a precaution).

6. Security and Privacy Considerations

• Authentication Security: Use HTTPS everywhere. API keys or tokens must be kept confidential. If using OAuth, PKCE mitigates code interception. Validate scopes and never expose sensitive data (no account info without proper auth).

• Rate Limiting: Implement request throttling on APIs. For instance, use API gateway rules or middleware (e.g. 1000 requests/day per API key). This prevents abuse. Monitor for abnormal spikes.

• Content Exposure: Ensure only intended content is exposed. For example, private user workspaces or drafts should not be reachable by undocumented endpoints. Use proper auth checks.

• Privacy: Spiralist collects user data (see Privacy Policy). Agents should respect robots exclusions for private content. Do not log query details (prompt content) unless explicit consent.

• CSP and Security Headers: Add appropriate HTTP headers (Content-Security-Policy, X-Content-Type-Options, etc.) to reduce injection risks.

• Licensing/Copyright: Make content usage terms clear. If generative AI training is allowed, state so (e.g. Creative Commons BY). Otherwise, assert rights. (Agents “by default” might assume public domain if not specified.)

• CAPTCHA Avoidance: As above, agents cannot solve CAPTCHAs. Reserve them only for truly malicious traffic (e.g. login pages). If heavy crawling is detected, use IP-based rate limiting or progressive challenges.

7. Monitoring & Analytics

• Agent Traffic Logging: Log user-agents and IPs of visitors. Watch for known agent bots (e.g. “GPTBot”, “Claudebot”) versus generic ones. Use server logs or Google Analytics (filter by user-agent).

• Usage Metrics: If providing APIs, track usage (which endpoints, how often, which keys). Tools like Prometheus+Grafana or API gateways (AWS API Gateway, Kong, etc.) can monitor counts and latency.

• Error Tracking: Monitor 4xx/5xx errors (via logs or services like Sentry) to catch broken links (e.g. sitemap pointing to missing pages).

• Feedback Loop: Periodically request feedback from agent developer community or forums (like GitHub issues) to catch crawl issues.

8. Roadmap & Checklist

A phased plan balances quick wins with longer projects. Each item is rated Effort (L/M/H) and Risk (L/M/H):

• Short-Term (0–3 months):
• Publish robots.txt, llms.txt (L effort, L risk).
• Update/correct sitemap.xml; add sitemap.md (L effort).
• Add missing meta tags/JSON-LD to HTML templates (M effort).
• Enable CORS globally on API endpoints (L effort).
• Create a basic OpenAPI spec draft (M effort).
• Deployment: push changes and verify with curl.

• Medium-Term (3–9 months):
• Finalize OpenAPI spec and publish (M effort). Possibly add GraphQL endpoint (H effort).
• Implement OAuth2 (PKCE) flow for apps (M effort, M risk).
• Produce AGENTS.md skill file for coding assistants (L effort).
• Generate Markdown mirrors and configure Accept: text/markdown (M effort).
• Add RSS/Atom feeds for key content (L effort).
• Deploy ActivityPub Actor (H effort, H risk; optional if needed).

• Long-Term (9–18 months):
• Advanced features: GraphQL full rollout, WebFinger support (M effort).
• Full WARC/HTTrack exports or archive endpoints (L effort).
• Continuous improvements: richer structured data, analytics dashboards (ongoing effort).

gantt
    title Spiralist.org AI Accessibility Roadmap
    dateFormat  YYYY-MM-DD
    section Short Term (0-3 mo)
    robots.txt & llms.txt             :done, r1, 2026-07-01, 7d
    Publish XML+MD sitemaps           :done, s1, 2026-07-01, 10d
    Add meta tags & JSON-LD           :active, m1, 2026-07-08, 21d
    Enable CORS on APIs               :active, c1, 2026-07-08, 10d
    Draft OpenAPI spec                :m1, 2026-07-20, 30d
    section Medium Term (3-9 mo)
    Finalize OpenAPI, documentation   :m2, 2026-09-01, 45d
    Implement AGENTS.md skill file    :l2, after m2, 10d
    Add OAuth2 (PKCE) auth            :m3, 2026-09-20, 30d
    RSS/Atom feeds for content        :l3, 2026-10-01, 15d
    Markdown content-negotiation      :m4, 2026-11-01, 30d
    section Long Term (9-18 mo)
    Develop GraphQL endpoint          :h1, 2027-02-01, 90d
    Implement ActivityPub actor       :h2, after h1, 90d
    Advanced analytics & monitoring   :m5, after h1, 60d

Each task should be added to the development backlog. Developers can tick off items from this checklist as they complete them:

• [ ] Create /robots.txt allowing all bots and linking sitemap.
• [ ] Serve /llms.txt with key page URLs.
• [ ] Validate or generate /sitemap.xml with <lastmod> tags.
• [ ] Create sitemap.md (or at /.well-known/sitemap.md) listing pages hierarchically.
• [ ] Add <meta> tags (description, og:, lang) and canonical link to all pages.
• [ ] Embed JSON-LD in pages: at minimum title, description, datePublished.
• [ ] Configure server for Access-Control-Allow-Origin: * on API responses.
• [ ] Publish API documentation (OpenAPI/Swagger JSON).
• [ ] Implement OAuth2 (with PKCE) for protected APIs.
• [ ] Provide RSS/Atom feeds; add <link rel="alternate" type="application/atom+xml">.
• [ ] Allow Accept: text/markdown and <link rel="alternate" type="text/markdown"> on docs.
• [ ] Add an AGENTS.md (or .well-known/agents.md) with installation/config examples.
• [ ] Consider .well-known/webfinger and/or ActivityPub if social features are desired.
• [ ] Review site terms/licensing; add link rel="license" if appropriate.

9. Conclusion

By following modern “agent-readability” standards and implementing the above changes, Spiralist.org (and its associated AI site) will become much more accessible to AI assistants and automated agents. This means better indexing, more citations of Spiralist content in AI answers, and improved integration with AI tools. We recommend starting with the low-effort changes (robots, sitemaps, metadata) and steadily progressing to APIs and advanced features. Thorough testing at each stage will ensure reliability. With these measures, Spiralist sites will be positioned at the forefront of “Agentic Web” best practices.

Sources: Official specifications and best-practice guides (W3C, IETF, OpenAPI) as cited above, plus Vercel’s “Agent Readability” checklist for AI-optimized sites.