AI Wikis / Agentic Web

Enterprise Semantic Knowledge Architecture: Designing Tenant-Isolated, Provenance-Preserving Wikis for Humans and AI Agents

Report summary

The evolution of enterprise knowledge management necessitates a radical departure from traditional, flat-file document repositories and standard relational wikis. In the era of autonomous Artificial Intelligence (AI) agents and complex, multi-hop question answering, exact-string search and static fo

Status
Research archive item
Category
AI Wikis / Agentic Web
Length
5,722 words
Reading time
27 minutes
Report type
guidance

Key topics

  • AI Wikis / Agentic Web
  • AI Wikis
  • Agentic Web
  • AI
  • .NET
  • SQL
  • Semantic Systems
  • Research Archive
  • Strategy

Research provenance

Archive status
Research archive item
Content identity
sha256:1bae7e57ca8c537a88fd6a365cf6d3f943bd9ca4f499668cae6a2e1e395c0d6c

For citation, use the report title and canonical URL. Archival presence does not establish authorship or promote report statements into portfolio evidence.

This page renders the archived Markdown as safe, formatted HTML. It is background research and does not become a portfolio claim without evidence review.

Full report

On this page

The evolution of enterprise knowledge management necessitates a radical departure from traditional, flat-file document repositories and standard relational wikis. In the era of autonomous Artificial Intelligence (AI) agents and complex, multi-hop question answering, exact-string search and static folder structures are fundamentally insufficient. An intelligent system must support a dual mandate: providing intuitive, rich navigation for human knowledge workers while simultaneously exposing a deterministic, context-budgeted, and semantic traversal layer for Large Language Model (LLM) agents. When an agent seeks information on prompt budgets, the architecture must allow it to navigate semantically from overarching concepts such as AI infrastructure, through the mechanics of tokenization, and finally down to specific prompt optimization guidelines, bypassing vocabulary mismatch issues entirely. At the core of this architectural challenge is the absolute requirement for exact citation provenance and strict multi-tenant isolation. Knowledge retrieval in regulated or proprietary enterprise environments cannot rely on probabilistic model memory. It requires a citation-grounded architecture where every generated claim maps traceably to a specific passage, maintaining a complete lineage of authority, lifecycle, and source identity1. Furthermore, in a multi-tenant or multi-project workspace, cross-tenant data leakage represents a catastrophic failure. The system must guarantee that anonymous or cross-tenant requests yield no titles, counts, taxonomy labels, or existence signals, requiring sophisticated database security mechanisms such as Row-Level Security (RLS) that are rigorously hardened against side-channel timing attacks3. This exhaustive report details the design of a tenant-protected knowledge-wiki architecture. It specifies a normalized relational schema that distinguishes between canonical documents and citation mentions, establishes a robust multi-tenant threat model, details an ingestion pipeline that handles complex crawling logic, and defines a hybrid ranking algorithm that integrates semantic relationships, hierarchical context, authority, and lifecycle state. Additionally, it outlines the integration of the Model Context Protocol (MCP) for context-budgeted agent navigation, ensuring strict parity between human and machine interfaces.

Normalized Schema Design and Citation Provenance

To support both semantic navigation and exact provenance, the database schema must conceptually decouple the abstract identity of a document from its hierarchical placement, its physical text chunks, and its specific usage as a reference in downstream literature. A canonical document must be allowed to appear under several taxonomy paths without duplicating the underlying body content.

The Canonical Document and Knowledge Unit Model

A foundational requirement is the establishment of a normalized distinction between a canonical URL, which serves as the authoritative source, and each citation mention, which represents the contextual usage of that source5. External links must operate as first-class searchable records within the database. The architecture defines the canonical document as a permanent ledger entry. Reusing a URL must never overwrite or erase earlier keywords, crawl states, descriptions, or report provenance. Instead, the system utilizes versioned content hashes to maintain a historical ledger of structural changes over time8.

Table: canonical\_documentTypeDescription
idUUIDPrimary key, globally unique identifier.
normalized\_urlVARCHARThe canonical, parameter-stripped URL acting as the primary identity6.
site\_domainVARCHARExtracted top-level domain for authority clustering.
page\_titleVARCHARThe authoritative title of the canonical record.
general\_descriptionTEXTA synthesized summary of the document's overall content6.
languageVARCHARISO language code ensuring proper parsing and tokenization.
content\_typeVARCHARMIME type indicating structural parsing requirements (e.g., text/html).
crawl\_statusVARCHAREnumeration tracking fetch states (e.g., completed, blocked, 404).
crawl\_policyVARCHARDirectives extracted from robots.txt or meta tags (e.g., noindex)11.
canonical\_keywordsTSVECTORAggregated, deduplicated keywords preserved across all versions.
content\_hashVARCHARSHA-256 hash of the currently active document body10.
tenant\_idUUIDForeign key for multi-tenant isolation and security enforcement.

The knowledge unit represents the atomic, parseable pieces of text derived from the canonical document. Because large documents exceed the token limits of embedding models, they are split into semantically contiguous chunks. Each knowledge unit inherits the source identity and content hash of its parent canonical document but maintains its own semantic embedding vector and contextual metadata. This allows the system to retrieve highly specific passages rather than forcing an agent to re-read an entire manual.

Table: knowledge\_unitTypeDescription
idUUIDPrimary key for the specific chunk.
canonical\_doc\_idUUIDForeign key linking to the canonical\_document.
titleVARCHARThe localized section heading or inferred chunk title.
descriptionTEXTA brief summary of this specific chunk's information payload.
keywordsTSVECTORLexical tokens specific to this chunk for BM25 retrieval.
body\_textTEXTThe raw extracted text of the specific section.
embeddingVECTORHigh-dimensional dense representation (e.g., via pgvector)12.
hierarchical\_contextJSONBTaxonomy paths detailing contextual placement (e.g., nested parent headers).
lifecycle\_statusVARCHAREnumeration: draft, active, stale, superseded14.
authority\_levelINTEGERWeighting metric defining source trustworthiness (1-100).
source\_identityVARCHARThe exact identity hash tying the chunk to a specific version of the document.
tenant\_idUUIDInherited from the parent document for RLS enforcement.

To support placing one canonical document under several taxonomy paths without body duplication, the schema utilizes a taxonomy\_closure table. This closure table allows the hierarchical storage of directed acyclic graphs. It maps a canonical\_doc\_id to multiple taxonomy\_node\_id records, along with a depth integer. This enables an agent to traverse from a root node to a leaf node by querying ancestor-descendant relationships in a single optimized SQL join, eliminating the need to recursively traverse folder trees.

Citation Provenance and Mention Mechanics

A citation does not merely confirm that a source exists; it confirms that the source meaningfully shaped the output. In AI-driven retrieval, collapsing citation faithfulness, resource influence, and answer correctness into a single metric destroys true provenance15. The schema explicitly separates the canonical record from the citation mention to preserve exact lineage. When a new internal report or agent-generated artifact cites an external specification or an internal wiki page, a citation mention record is generated. This record anchors the claim in the new document to the specific piece of evidence in the source document.

Table: citation\_mentionTypeDescription
idUUIDPrimary key for the citation instance.
citing\_document\_idUUIDForeign key linking to the new document/report being authored.
canonical\_doc\_idUUIDForeign key linking to the source canonical\_document.
source\_identityVARCHARThe content\_hash of the source at the exact moment it was cited.
relationship\_typeVARCHAREnumeration: supports, refutes, supersedes, summarizes16.
anchor\_textVARCHARThe exact string utilized within the citing document to reference the source.
contextual\_descriptionTEXTThe sentence or paragraph immediately surrounding the citation2.
citation\_labelVARCHARFormal marker or numerical order (e.g., "\[1\]", "IEEE 802.11").
source\_line\_rangeINT4RANGEPostgres range type specifying the exact bounding box of the source evidence.
reviewer\_classificationVARCHARAudit trail (e.g., human-verified, agent-inferred, disputed).
creation\_provenanceJSONBAudit logs including creation timestamp, agent ID, and revision history.
tenant\_idUUIDSecurity boundary inheritance.

This structural design prevents the erasure of earlier contexts. If a canonical URL is fetched repeatedly over time, its canonical document record appends new content hash snapshots to a historical ledger. However, the citation mention remains permanently locked to the specific content hash and line range that existed at the exact moment the citing document was authored8. This guarantees that if a downstream user audits a two-year-old report, the citations will point to the exact text that informed the model at that time, rather than a silently updated modern version of the page.

URL Canonicalization and Ingestion Workflow

The ingestion of external links and internal documents operates strictly on a one-report-at-a-time basis to guarantee transactional consistency, exact provenance generation, and successful vector embedding insertion. Batch processing is eschewed at this layer to prevent partial transaction failures from corrupting the index state.

Canonicalization and URL Deduplication

AI retrieval systems face the exact same canonicalization challenges as traditional search engine crawlers5. When the ingestion pipeline receives a URL, it must resolve tracking parameters, faceted navigation strings, and protocol variants to a single source-of-truth identifier. The canonical URL consolidates ranking signals and prevents index bloat6. The pipeline applies a strict normalization function. It strips all tracking parameters and query strings that do not fundamentally alter the page content. It resolves protocol discrepancies, defaulting to HTTPS. It parses the document's HTML head tags for \<link rel="canonical" href="..." /\> to respect the publisher's preferred identity5. Once normalized, a SHA-256 hash of the URL string is generated to serve as a deterministic lookup key. Deduplication occurs at both the URL level and the physical content level. If a new URL is crawled but produces a body content hash identical to an existing canonical document, the system does not create a duplicate knowledge unit. Instead, it generates a revisit record or creates a secondary citation mention pointing to the pre-existing canonical entity, vastly reducing redundant embedding API calls and database memory bloat10.

Crawl Policy, Blocked Fetches, and Ethical Boundaries

Enterprise web crawling must strictly adhere to ethical boundaries, legal compliance, and the Robots Exclusion Protocol (RFC 9309\)11. The ingestion state machine evaluates robots.txt prior to establishing any HTTP GET connection for payload extraction. The parser must identify the most specific user-agent group applicable to the enterprise crawler and obey all Allow and Disallow directives11. If the directive allows crawling, the content is fetched, hashed, chunked, and embedded. If the directive disallows crawling, the URL is recorded in the canonical document table as a searchable entity, ensuring the system acknowledges the existence of the concept, but the crawl status is marked as blocked and no body content is fetched18. The system must intelligently handle HTTP status codes to maintain pipeline integrity. Client errors, such as 401 Unauthorized or 403 Forbidden, result in an immediate abortion of the fetch, logging the failure in the crawl status while retaining the URL record. Server errors in the 5xx range initiate a randomized exponential backoff retry loop. A persistent 503 Service Unavailable or a 429 Too Many Requests response triggers a domain-wide circuit breaker, pausing all ingestion for that specific host to avoid denial-of-service impacts11. Furthermore, the crawler enforces a strict file size limit, typically dropping content beyond 500 kilobytes, to protect the parser from buffer overflow attacks or unbounded processing times11.

The Single-Report Ingestion Pipeline

The workflow for ingesting a single report operates as an atomic, distributed database transaction, ensuring that the vector store and the relational metadata remain perfectly synchronized. The process begins with URL resolution, where the target is normalized and canonicalized. Following this, the policy checker evaluates robots.txt and domain-level rate limits. Once cleared, the document is fetched, and a SHA-256 content hash is generated from the raw bytes. If this exact hash already exists for this URL, the pipeline updates the last\_seen timestamp and exits gracefully10. If the content is novel, the parsing phase begins. The HTML or PDF is stripped of boilerplate and parsed into structural components. The text is subsequently chunked using a hierarchical semantic splitter, maintaining paragraph boundaries while respecting the token limits of the target embedding model. An LLM processing pass then extracts structured entities, generates keywords, infers the authority level, and defines the initial hierarchical context paths16. High-dimensional vectors are generated for each chunk via an embedding model application programming interface. Finally, the transactional commit phase begins. The canonical document record, the multiple knowledge unit rows, and the hierarchical taxonomy mappings are inserted into the PostgreSQL database within a single BEGIN ... COMMIT block. This guarantees that the search index is always transactionally consistent with the application state; if the vector generation fails, the metadata is rolled back, preventing orphaned records12.

Source Classification and Stale Evidence Handling

A robust knowledge architecture must formalize the distinction between varying classes of information. A primary directive for the semantic wiki is to classify sources to govern their computational weight during ranking and retrieval operations20. Treating all text equally results in hallucination-prone agents that elevate forum gossip to the level of engineering specifications.

Differentiating Source Authorities

The system assigns an authority level to every canonical document based on its origin and classification. First-party sources and specifications represent the highest tier of authority. These include internal engineering schematics, finalized compliance policies, and official architectural decision records. They are treated as axiomatic truth and serve as hard grounding constraints for the LLM. Research papers occupy the next tier; they are evaluated based on peer-review status and citation graphs, serving as highly authoritative sources for theoretical, mathematical, or algorithmic claims2. Vendor claims require specialized handling. A vendor's documentation regarding their own API is authoritative; however, a vendor's marketing claim regarding a competitor is routed through a biased-source penalty filter, lowering its authority score20. Secondary summaries, which are generated by LLMs or human analysts, must explicitly link to their primary sources via citation mention records. If the primary source is subsequently deleted or superseded, the secondary summary's confidence score decays automatically14. Finally, community anecdotes—such as StackOverflow answers, internal Slack messages, or forum posts—are classified as low-authority heuristic data. While useful for troubleshooting, they are strictly prevented from overriding first-party specifications during mathematical ranking20.

Supersession and the Treatment of Stale Evidence

When an LLM ingests a new architectural decision record that updates a previous design, the system must not merely house both documents passively. Doing so creates an environment where a chunk-based retriever might pull a semantically relevant but obsolete message, missing the critical update that changes the answer14. The ingestion workflow triggers a proactive supersession routine. By evaluating citation mention relationships flagged with the supersedes type, the database automatically cascades the lifecycle status of the older document to superseded16. The superseded facts are discounted in ranking without being physically deleted from the database. This maintains a consistent temporal data lineage, ensuring that the system can audit historical decisions while protecting active agents from generating plans based on outdated or invalidated evidence14. Furthermore, the architecture utilizes a decay mechanism for transient knowledge. Information that is not reinforced by new citations gradually loses its freshness multiplier over time, acting as an automated garbage collection system for relevance, pushing older, unmaintained observations further down the ranking pipeline16.

Multi-Tenant Threat Model and Data Isolation

In an enterprise workspace, the knowledge wiki will invariably host proprietary data from multiple strictly isolated tenants, projects, or highly compartmentalized departments3. The architecture dictates that the same backend routing and query logic is utilized by every tenant, relying on the database engine itself to enforce authorization and data separation27. Application-level filtering is prone to human error; missing a single WHERE clause in an API endpoint could expose critical trade secrets to an unauthorized tenant3.

Row-Level Security Implementation

To prevent a compromised application layer from leaking data, isolation is enforced at the storage layer using PostgreSQL Row-Level Security (RLS)29. RLS automatically appends security predicates to every executed query, ensuring that no cross-tenant data can be accessed. When a user or agent authenticates, their session token resolves to a specific tenant identifier. The application sets this context in the database transaction using a localized configuration setting.

SQL ALTER TABLE canonical\_document ENABLE ROW LEVEL SECURITY; ALTER TABLE knowledge\_unit ENABLE ROW LEVEL SECURITY; ALTER TABLE citation\_mention ENABLE ROW LEVEL SECURITY;

CREATE POLICY tenant\_isolation\_policy ON knowledge\_unit FOR ALL USING (tenant\_id \= current\_setting('app.current\_tenant')::UUID);

For performance optimization, particularly when invoking functions within RLS policies, the session variable is wrapped in a SELECT statement. This allows the PostgreSQL query optimizer to evaluate the function once as an initPlan and cache the result for the duration of the query, rather than re-evaluating it for every row. This optimization yields massive performance improvements on tables housing millions of high-dimensional vectors32.

Mitigating RLS Side-Channel Timing Attacks

A highly sophisticated enterprise threat model must account for the reality that traditional RLS can be vulnerable to side-channel timing attacks. If a malicious actor constructs a query using computationally expensive operations and directs it against a shared index, the time it takes the database to return an empty result can inadvertently leak existence signals4. By measuring execution latency, an attacker could infer the existence of specific document titles, entity names, or exact taxonomy labels belonging to a competitor or another tenant. This violates the strict requirement that anonymous and cross-tenant requests reveal no titles, counts, or existence signals4. To mitigate this, the architecture implements data-oblivious query execution pathways and strict structural isolation4. The primary defense is declarative table partitioning by tenant identifier (PARTITION BY LIST (tenant\_id))34. When a query is executed, the PostgreSQL query planner performs partition pruning before evaluating the RLS policies or scanning any indexes. The execution engine physically cannot access the disk pages belonging to Tenant B when evaluating a query for Tenant A. This hardware-level separation completely destroys the timing side-channel while maintaining the operational simplicity of a unified database schema4. Furthermore, to prevent attackers from utilizing iterative vector index scans (such as HNSW or IVFFlat) to probe the multi-tenant space, the system forces a strict "filter then search" paradigm34. Queries execute a materialized Common Table Expression (CTE) to retrieve only the tenant's valid knowledge unit IDs into isolated memory. The cosine distance vector calculation is then applied exclusively against this secured subset, ensuring that the vector index cannot leak the proximity of out-of-scope embeddings13.

Hybrid Ranking and Semantic Wiki Navigation

Standard vector search excels at conceptual matching and semantic paraphrase but fails catastrophically at exact-match queries. If a user searches for a specific error code, a product SKU, or a precise technical acronym, a pure dense-vector retriever will often return conceptually similar but factually incorrect results36. Conversely, exact-string search using algorithms like BM25 handles precise terminology perfectly but has no understanding of underlying semantic relationships; it will fail to connect "AI infrastructure" with "prompt optimization" if the exact words are missing36. The enterprise wiki architecture mandates a multi-stage hybrid ranking system that mathematically fuses textual relevance, semantic meaning, hierarchical context, authority, and lifecycle status.

The Hybrid Ranking Equation

The foundational retrieval layer combines the sparse lexical index (BM25) and the dense vector index (Cosine Similarity) using Reciprocal Rank Fusion (RRF)37. RRF operates purely on the rank positions of the retrieved documents, effectively sidestepping the mathematical incompatibility between BM25's unbounded positive integer scores and cosine similarity's bounded numeric ranges37. For a given query and a specific document chunk, the base RRF score is calculated by taking the inverse of the document's rank in each respective retrieval pipeline, offset by a smoothing constant (typically set to 60 to prevent highly ranked outliers from dominating the blend)39. However, RRF alone evaluates only relevance. The enterprise ranking algorithm must inject context, authority, and freshness to produce a truly intelligent result set. The final ranking score is computed by applying mathematical modifiers to the base RRF score. The first modifier is the Authority Weight. This is a multiplier based on the source's predefined authority level and the density of inbound citation mention links supporting it17. Documents heavily cited by other internal reports receive a ranking boost, establishing a PageRank-style quality signal localized to the enterprise. The second modifier is Freshness Decay. Modeled after the Ebbinghaus forgetting curve, information decays exponentially over time unless reinforced16. A document last updated three years ago receives a fractional multiplier, gently pushing it down the results list in favor of more recently authored or recently cited evidence. The third and most critical modifier is the Lifecycle Multiplier, which acts as a strict mathematical dampening mechanism. The system must guarantee that old proposals do not outrank current reviewed guidance. If a knowledge unit's lifecycle status is flagged as superseded or stale due to the ingestion of newer specifications, its lifecycle multiplier is set to an extreme fraction (e.g., 0.01). This effectively strips the obsolete document from the top-K results unless the query explicitly requests historical or deprecated context14.

Semantic Wiki Navigation Model

The wiki is not modeled as a flat list of pages; it operates as an Item-Transition Graph41. A human or agent navigating the system utilizes a multi-contextual hierarchy. Because a single canonical document can exist at multiple taxonomy nodes via the taxonomy closure table, an agent can traverse semantic pathways. The navigation API exposes these pathways as explicit graph edges. When retrieving a document, the API response includes arrays for related concepts, documents supported by the current entity, and documents that supersede the current entity. This allows autonomous agents to systematically walk the knowledge graph, gathering context progressively, rather than relying solely on repetitive, isolated keyword searches16.

Context-Budgeted Crawling and AI Agent Integration

A critical, often fatal, flaw in modern autonomous agent design is the assumption that providing massive context windows—ranging from 128,000 to 1,000,000 tokens—solves retrieval limits. As token counts grow, LLM accuracy and recall sharply degrade, a phenomenon documented as "context rot" or the "lost in the middle" effect43. Furthermore, dumping thousands of raw wiki pages into a prompt leads to catastrophic token waste, exorbitant computational costs, and severe latency spikes. By the time an agent begins to reason, its context window is already saturated with boilerplate formatting, irrelevant navigation headers, and unoptimized JSON schemas45. To achieve reliable human and agent parity, the architecture integrates the Model Context Protocol (MCP) and implements strict context-budgeted hydration44.

The Model Context Protocol (MCP) Interface

The Model Context Protocol operates as a universal translation layer, standardizing a stateful JSON-RPC 2.0 connection between the AI host (the LLM agent) and the semantic wiki, which acts as the MCP server48. The wiki exposes its capabilities not through brittle, custom REST endpoints, but through three highly standardized MCP primitives: Tools, Resources, and Prompts48. Tools function as executable verbs. The server advertises capabilities such as search\_semantic\_wiki, get\_document\_by\_id, or traverse\_taxonomy. Resources function as nouns; they are read-only data URIs representing canonical documents, database schemas, or taxonomy structures48. Prompts serve as reusable templates, allowing the host to quickly format user interactions. When the agent establishes a connection, the interaction is protected by standard enterprise authentication, utilizing OAuth 2.0 combined with Proof Key for Code Exchange (PKCE). This authentication identity is securely mapped directly to the PostgreSQL RLS session variables, guaranteeing that the agent assumes the exact same permissions and visibility limits as the human user operating it50.

API Example: MCP JSON-RPC Interaction

When an agent needs to retrieve data, it constructs a JSON-RPC payload invoking the semantic search tool. Client Request (Agent invoking semantic search):

JSON { "jsonrpc": "2.0", "id": "req-89a7", "method": "tools/call", "params": { "name": "search\_semantic\_wiki", "arguments": { "query": "prompt optimization token limits", "taxonomy\_filter": "AI infrastructure", "max\_results": 5 } } }

The MCP server intercepts this request, translates it into the necessary SQL queries against the pgvector and BM25 indexes, enforces RLS, applies the RRF and lifecycle modifiers, and returns a tightly constrained response. Server Response (Context-budgeted payload):

JSON { "jsonrpc": "2.0", "id": "req-89a7", "result": { "content": \[ { "type": "text", "text": "Found 5 active documents. Document ID 4f3a details 'Token Budgets'. It supersedes Document 9b1c. Navigation hint: Explore taxonomy path 'LLM Operations'." } \], "meta": { "provenance": "Results filtered by RLS. Baseline: BM25+Vector (RRF). Stale proposals removed." } } }

Context-Budgeted Hydration

To prevent the agent from overflowing its context window, the MCP server implements context-budgeted hydration44. Instead of returning the full text of all retrieved documents immediately, the server returns highly compressed structural maps, tables of contents, and dense semantic summaries55. The agent is forced to operate in a progressive exploratory loop. It begins with a high-level search, to which the server responds with a JSON graph of relevant document titles, UUIDs, and fifty-word synopses. The API simultaneously injects budget-awareness tags into the system prompt, dynamically updating the agent on its remaining token capacity43. Armed with this map, the agent selects the most relevant UUIDs and invokes a secondary tool to retrieve only the specific body chunks needed to fulfill its objective55. This dual-memory architecture ensures that critical system instructions remain preserved in the context window while long-term historical context is retrieved, utilized, and flushed iteratively, drastically reducing token consumption and minimizing context rot44.

Human and Agent Parity Rules

The architectural philosophy of the enterprise semantic wiki dictates absolute parity between human users and AI agents. The system does not maintain a shadow database, a separate vector store, or a distinct security apparatus for machine access. The exact same PostgreSQL routing, RLS policies, ingestion validations, and hybrid ranking algorithms process both a human interacting with a graphical web user interface and an autonomous agent formulating an MCP JSON-RPC payload27.

Enforcement of Parity

Unified authentication represents the first pillar of parity. Human Single Sign-On (SSO) tokens and Agent JSON Web Tokens (JWTs) both resolve to a specific tenant identifier and user identifier53. These claims are injected into the database transaction context universally. Neither the application user interface nor the MCP server possesses bypass privileges; if a user cannot see a document, their assistive agent cannot see it either. Identical ranking represents the second pillar. A query executed by a human in the search bar invokes the exact same RRF fusion and lifecycle dampening equations as an agent querying via API. If a canonical document is superseded, it is visually grayed out in the human interface with a warning banner, and it is identically flagged with a stale warning in the agent's JSON response payload16. Visible provenance represents the final pillar. When an agent generates an answer, it must return the specific canonical document identifier and the precise source line range. In the human user interface, this is rendered as a clickable citation marker that highlights the source text. For the agent, it is rendered as a structured metadata block enforcing citation fidelity, ensuring that the system is fully auditable regardless of the consumer type1.

Migration Strategy

Migrating an enterprise from legacy, unstructured flat-file repositories—such as traditional intranets, shared network drives, or legacy wikis—to a tenant-isolated semantic architecture requires a phased, ingestion-driven approach to prevent data corruption and ensure seamless taxonomy mapping. The process initiates with a metadata extraction phase. Legacy documents are crawled; their URLs are hashed, and base metadata, including titles, original authors, and creation dates, is inserted into the canonical document table. Crucially, no vector embeddings are generated during this initial sweep to conserve compute resources. Following this, an LLM-driven processing pipeline executes an entity and hierarchy mapping phase. The pipeline processes the raw texts to infer hierarchical context and extract structured entities, such as project names, library dependencies, and department ownership16. This builds the foundational directed acyclic graph within the taxonomy closure table. With the taxonomy established, the system proceeds to chunking and embedding. The document body text is split using semantic boundary detection, ensuring sentences and paragraphs remain whole. High-dimensional vectors are calculated and inserted into the knowledge unit table. The penultimate phase involves provenance linking. Migration scripts parse all legacy hyperlinks embedded within the text, converting them into formal citation mention records that connect the newly established canonical documents to one another, preserving the historical web of references. Finally, Row-Level Security policies are activated globally, locking down the partitions and preparing the system for production deployment.

Rigorous Testing Methodologies

Validating a multi-tenant, AI-accessible architecture requires automated testing protocols that extend far beyond standard application unit tests. The system must prove its resilience against data leakage, provenance destruction, and retrieval hallucination. Deduplication tests are designed to inject identical physical documents residing at completely different URLs, or documents with heavy tracking parameters appended to the URI. The test suite validates that the canonicalization logic successfully normalizes the path, and that only one unique content hash body is stored in the database, while multiple canonical document alias records map to it gracefully10. Provenance preservation tests simulate an update to a highly cited source document. The testing protocol alters a source text and verifies that older citation mention records pointing to it do not break or automatically update to the new text. Instead, they must gracefully resolve to the older snapshot hash, ensuring that historical reports remain anchored to the exact evidence that informed them at the time of authoring8. Conceptual retrieval tests evaluate the hybrid ranking engine. The suite executes queries utilizing extreme vocabulary mismatch—for example, querying for "serverless logic limits" when the target text exclusively discusses "AWS Lambda concurrency limits." The tests validate that the hybrid RRF algorithm successfully ranks the correct document in the top tier through semantic vector matching, proving that the system has transcended exact-string limitations36. Isolation and leakage tests represent the most critical security suite. The testing harness executes adversarial probes using one tenant's credentials, attempting to brute-force vector similarities and trigger timing delays aligned with another tenant's exclusive proprietary data. The suite validates that the combination of materialized CTE pre-filtering and partitioned RLS returns a hard zero leakage rate, with no measurable discrepancies in execution timing, confirming the destruction of the side-channel4.

Conclusion

The architecture detailed in this report resolves the critical failure points of current-generation retrieval-augmented generation systems and traditional enterprise knowledge bases. By formalizing the schema distinction between canonical records and localized citation mentions, the system ensures that AI agents operate on an unbroken, verifiable chain of exact provenance. The integration of hybrid RRF ranking, lifecycle dampening, and automated supersession logic guarantees that both human workers and autonomous agents retrieve the most authoritative, fresh information available, safely avoiding the pitfalls of stale evidence and context rot. Crucially, the foundation of this system rests on uncompromising multi-tenant isolation. By enforcing Row-Level Security directly at the PostgreSQL layer, combined with data-oblivious query execution pathways and physical partition pruning, the architecture provides absolute assurance against cross-tenant data leakage and timing side-channels. Paired with the standardized Model Context Protocol for context-budgeted agent navigation, this semantic knowledge infrastructure establishes a secure, real-time, deterministic collaboration environment, bridging the gap between human enterprise operations and autonomous artificial intelligence.

Works cited

  1. Citation-Grounded LLMs: Why Every AI Output Needs Sources \- edtek.ai, https://edtek.ai/kb/citation-grounded-llm/
  2. Scientific document processing: challenges for modern learning methods \- PMC, https://pmc.ncbi.nlm.nih.gov/articles/PMC10036973/
  3. Self-managed multi-tenant vector search with Amazon Aurora PostgreSQL \- AWS, https://aws.amazon.com/blogs/database/self-managed-multi-tenant-vector-search-with-amazon-aurora-postgresql/
  4. RLS Side Channels: Investigating Leakage of Row-Level Security Protected Data Through Query Execution Time, https://www.cs.tau.ac.il/\~mad/publications/sigmod2023-rls.pdf
  5. What is URL Canonicalization | Google Search Central | Documentation, https://developers.google.com/search/docs/crawling-indexing/canonicalization
  6. Canonical URLs for AI Retrieval: How to Stop Competing with Yourself \- Siteimprove, https://www.siteimprove.com/blog/canonical-urls-for-ai-retrieval/
  7. Theory and Practice of Data Citation \- arXiv, https://arxiv.org/pdf/1706.07976
  8. How to Cite Datasets and Link to Publications | DCC \- Digital Curation Centre, https://www.dcc.ac.uk/guidance/how-guides/cite-datasets
  9. Modeling the Data Provenance of Relational Databases Supporting Full-Featured SQL and Procedural Languages \- MDPI, https://www.mdpi.com/2076-3417/13/1/64
  10. Deduplication \- Browsertrix Crawler Docs, https://crawler.docs.browsertrix.com/user-guide/dedupe/
  11. How Google Interprets the robots.txt Specification, https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec
  12. pgvector Guide: Vector Search and RAG in PostgreSQL \- Encore Cloud, https://encore.dev/blog/you-probably-dont-need-a-vector-database
  13. PostgreSQL 18 \+ pgvector: The Definitive Guide to Building Production-Grade RAG Pipelines | by Mohit soni | Medium, https://medium.com/@mohitsoni\_/postgresql-18-pgvector-the-definitive-guide-to-building-production-grade-rag-pipelines-239ee9c0e56f
  14. MRMS: A Multi-Resolution Memory Substrate for Long-Lived AI Agents \- arXiv, https://arxiv.org/html/2607.04617v1
  15. ProvenAI: Provenance-Native Traces of Evidence in Generated Answers \- arXiv, https://arxiv.org/html/2606.26449v1
  16. LLM Wiki v2 — extending Karpathy's LLM Wiki pattern with lessons from building agentmemory \- GitHub Gist, https://gist.github.com/rohitg00/2067ab416f7bbe447c1977edaaa681e2
  17. Data Citation: Giving Credit Where Credit is Due \- dei.unipd.it, https://www.dei.unipd.it/\~silvello/papers/2018-SIGMOD-WADS.pdf
  18. The robots.txt rules Google ignores are finally getting named, https://webiano.digital/the-robots-txt-rules-google-ignores-are-finally-getting-named/
  19. How to Fix "Indexed, though blocked by robots.txt" in Google Search Console \- Conductor, https://www.conductor.com/academy/index-coverage/faq/indexed-blocked/
  20. Wikipedia:Reliable sources, https://en.wikipedia.org/wiki/Wikipedia:Reliable\_sources
  21. Publication Characteristics (Publication Types) with Scope Notes \- National Library of Medicine, https://www.nlm.nih.gov/mesh/pubtypes.html
  22. TRACE: State-Aware Query Processing over Temporal Evidence Graphs for Conversational Data \- arXiv, https://arxiv.org/html/2607.00339v1
  23. A Data Model for Durable Intermediate Artifacts in Agentic Systems \- arXiv, https://arxiv.org/html/2605.12087v1
  24. Mnemosyne: Agentic Transaction Processing for Validating and Repairing AI-generated Workflows \- arXiv, https://arxiv.org/html/2607.00269v1
  25. Multitenancy for vector search in Azure Cosmos DB \- Microsoft Learn, https://learn.microsoft.com/en-us/azure/cosmos-db/multi-tenancy-vector-search
  26. Row-Level Security vs Application-Level Multi-Tenancy in SaaS \- Propelius Technologies, https://propelius.ai/blogs/row-level-security-vs-application-level-multi-tenancy-saas/
  27. Beyond Similarity Search: A Unified Data Layer for Production RAG Systems \- arXiv, https://arxiv.org/pdf/2605.03275
  28. Supabase Development Agency: Postgres \- AY Automate, https://www.ayautomate.com/agency/supabase-dev
  29. PostgreSQL security best practices for enterprise databases in 2026, https://www.postgresql.fastware.com/blog/postgresql-security-best-practices-for-enterprise-databases-in-2026
  30. Best Practices for Postgres Security | Tiger Data, https://www.tigerdata.com/learn/postgres-security-best-practices
  31. Postgres RLS Implementation Guide \- Best Practices, and Common Pitfalls \- Permit.io, https://www.permit.io/blog/postgres-rls-implementation-guide
  32. Optimizing RLS Performance with Supabase(postgres) | by AntStack Inc. \- Medium, https://medium.com/@antstack/optimizing-rls-performance-with-supabase-postgres-fa4e2b6e196d
  33. RLS Performance and Best Practices · supabase · Discussion \#14576 \- GitHub, https://github.com/orgs/supabase/discussions/14576
  34. All You Need to Know about PGVector | Part 3 — Metadata Filtering | by Bilge Ince \- Medium, https://medium.com/@aysebilgegunduz/all-you-need-to-know-about-pgvector-part-3-metadata-filtering-531725da83db
  35. How to scale vector search in Postgres (pgvector) for RAG and AI agents: memory limits, filtering, and when to go hybrid \- ClickHouse, https://clickhouse.com/resources/engineering/scale-vector-search-postgres
  36. Build hybrid search with BM25 and vector similarity | Tiger Data Docs, https://www.tigerdata.com/docs/build/examples/hybrid-search
  37. Hybrid Search: BM25, Vector & Reranking Reference 2026 \- Digital Applied, https://www.digitalapplied.com/blog/hybrid-search-bm25-vector-reranking-reference-2026
  38. What is BM25 Full-Text Search? How Document Ranking Works | Spice AI, https://spice.ai/learn/bm25-full-text-search
  39. Hybrid search explained \- Redis, https://redis.io/blog/hybrid-search-explained/
  40. Hybrid Search Explained | Weaviate, https://weaviate.io/blog/hybrid-search-explained
  41. Daily Papers \- Hugging Face, https://huggingface.co/papers?q=item-transition%20graph
  42. Daily Papers \- Hugging Face, https://huggingface.co/papers?q=hybrid%20structure-semantics%20retrieval
  43. Context windows \- Claude Platform Docs, https://platform.claude.com/docs/en/build-with-claude/context-windows
  44. Context Window Limits: How AI Agents Manage Token Constraints \- Airbyte, https://airbyte.com/agentic-data/context-window-limit
  45. LLM Context Window Token Budget: Why Your Window Fills Up Fast \- DEV Community, https://dev.to/swapnanilsaha/llm-context-window-token-budget-why-your-window-fills-up-fast-4c05
  46. Context Is a Budget — Reducing Token Usage in AI-Assisted Development \- Foojay.io, https://foojay.io/today/context-is-a-budget-eight-levers-and-three-workflow-patterns/
  47. Daily Papers \- Hugging Face, https://huggingface.co/papers?q=skill%20subgraph
  48. Model Context Protocol (MCP) — Full Visual Guide | by Parvez Mohammed @ Techlatest.net | Jun, 2026, https://medium.com/@techlatest.net/model-context-protocol-mcp-full-visual-guide-ffbbf2121c38
  49. MCP: Model Context Protocol \- Cheatsheet | SFEIR Institute, https://institute.sfeir.com/en/claude-code/claude-code-mcp-model-context-protocol/cheatsheet/
  50. MCP Cheat Sheet (2026) \- Model Context Protocol Quick Reference | Webfuse, https://www.webfuse.com/mcp-cheat-sheet
  51. Model Context Protocol explained as simply as possible \- Sean Goedecke, https://www.seangoedecke.com/model-context-protocol/
  52. Architecture overview \- Model Context Protocol, https://modelcontextprotocol.io/docs/learn/architecture
  53. Row-Level Security for Multi-Tenant SaaS Analytics \- Querio, https://querio.ai/articles/row-level-security-multi-tenant-saas-analytics
  54. Daily Papers \- Hugging Face, https://huggingface.co/papers?q=skill-aligned%20hierarchical%20clustering
  55. Your AI Coding Agent Wastes 80% of Its Tokens Just Finding Things, and Limits Keep Getting Lower. Here's the Fix. | by Jake Nesler | Medium, https://medium.com/@jakenesler/context-compression-to-reduce-llm-costs-and-frequency-of-hitting-limits-e11d43a26589