Ontological Machine Intelligence Observability and Evaluation Architecture

Full report

On this page

A. Research Metadata

The transition from stochastic, generative architectures to genuine Ontological Machine Intelligence (OMI) represents a fundamental paradigm shift in computational design. Legacy artificial intelligence models operate fundamentally via next-token prediction over vast parameters, demonstrating exceptional fluency in linguistic synthesis while suffering from profound, structural epistemological deficits1. These systems routinely conflate the mechanical transport of information with the rigorous, logical justification of that information. In contemporary literature, this phenomenon is diagnosed as "semantic laundering"—an architectural realization of the philosophical Gettier problem where a machine accepts weakly warranted propositions as ground truth simply because they cross an internal, trusted API boundary1. To achieve true machine intelligence, an autonomous agent must possess an explicit, structural epistemic architecture capable of delineating raw observation from logical inference, and untested hypotheses from verified knowledge1. The underlying representational mechanisms rely on hyperdimensional computing and vector-symbolic architectures (VSA), utilizing mathematical operations such as bundling (superposition), binding (role-filler association), and unbinding (associative retrieval) to bridge continuous latent spaces with formal symbolic ontologies2. Consequently, observability for such systems cannot merely track API latency or compute utilization; it must mathematically verify the transition of evidence through an epistemic ledger, capturing the exact provenance of all assertions, causal interventions, and metacognitive abstentions1. This architecture leverages the OpenTelemetry (OTel) GenAI semantic conventions alongside the OpenInference standard to normalize telemetry across hybrid Python and Rust implementations4. Evaluation orchestration relies on the Inspect AI framework, applying deterministic, cross-language JSON fixtures to validate system boundaries without demanding identical internal implementations7.

B. Observability Model

The observability model for an Ontological Machine is divided into six deterministic boundaries that map to the system's core cognitive loop. Standard distributed tracing tracks the execution flow, while an append-only event ledger tracks the epistemic state transitions to guarantee deterministic replay capabilities3. The first boundary manages the ingestion of raw data, mapping the observation into a strict typed ontology. Duplicate entities and unknown types are rejected immediately, emitting normalized error metrics to the telemetry collector3. Traces at this layer document the exact topological integration of new data against the existing ontological graph. The second boundary encompasses the epistemic ledger, where observations are transformed into formal claims. Telemetry here records the promotion of claims through a strict state machine: an assertion becomes a hypothesis, supported by cryptographic evidence digests, before being promoted to belief and ultimately knowledge (or demoted to falsified)3. The third boundary covers causal evaluation. Causal mechanisms evaluate the live and counterfactual state of the system. Observability spans isolate live conditioning from causal interventions, ensuring that the system's counterfactual reasoning is independently measurable3. The fourth boundary involves goal and skill orchestration. Desired goals and skills propose appropriate actions, generating agentic tool-proposal spans that follow OpenInference semantic conventions to document the cognitive planning phase5. The fifth boundary separates capability from action. External actions require explicit capability authority and validation from a capability broker before execution. Telemetry explicitly captures capability denials (e.g., rejecting an unauthorized write:device request) as neutral boundary rejections rather than internal system crashes, maintaining the integrity of the selected plan8. Finally, the sixth boundary ensures that every state transition is serialized into an append-only memory stream. Canonical events are assigned monotonic sequence numbers, allowing the entire cognitive history of the Ontological Machine to be exported, hashed, and deterministically replayed for regression testing and root-cause analysis3.

C. English Guide

The architecture of modern machine intelligence suffers from a representational divide between continuous latent spaces and formal symbolic ontologies2. Traditional observability centers on identifying runtime exceptions, tracking remote procedure calls, and validating expected mechanical behavior11. Large Language Model (LLM) observability extends this paradigm to monitor dynamic, stochastic outputs using standard semantic conventions—tracking token usage, cost, and latency5. However, Ontological Machine observability requires a third tier: epistemic and causal tracing. When the system ingests an observation, the observability pipeline must record the cryptographic digest of the source data, trace its translation into a formal hypothesis, and log the causal interventions that elevate it to verified knowledge3. The implementation heavily utilizes the OpenTelemetry GenAI semantic conventions. Historically, tracing systems used generic labels, but the GenAI Special Interest Group (SIG) established a standardized vocabulary under the gen\_ai.\* namespace. Early attributes like gen\_ai.system have been deprecated in favor of gen\_ai.provider.name, while operational properties rely on gen\_ai.request.model, gen\_ai.response.model, and gen\_ai.usage.input\_tokens5. The Ontological Machine framework injects these standard attributes into its Rust and Python execution contexts. Furthermore, the system implements the OpenInference specification to provide insight into complex agent orchestrations, external tool usage, and retrieval-augmented generation processes6. OpenInference spans capture the rich semantic payload of tool proposals and capability evaluations, allowing external platforms like Arize Phoenix or Datadog to render complex execution graphs12. Because OMI systems are designed as executable proofs, time independence is critical. Wall-clock time must never determine the semantic output of cognitive operations, nor should it influence the deterministic execution of the epistemic ledger8. The observability framework captures canonical event streams that allow for exact replays. If a regression occurs during development or production, the system exports the exact event lineage. The verification harness replays this lineage, comparing state hashes across executions byte-for-byte. By doing so, the framework isolates the exact cognitive step where a divergence occurred, eliminating the probabilistic debugging typically required in generative AI systems8. Evaluation of these systems transcends simple text-matching or LLM-as-a-judge stochastic grading. The Ontological Machine employs the Inspect AI framework, developed by the UK AI Security Institute, to orchestrate cross-language evaluation fixtures14. Inspect AI allows developers to define evaluation datasets, attach solvers (which represent the agent's prompting and tool-use strategies), and apply scorers that validate the output7. The OMI verification harness uses language-neutral JSON fixtures to validate invariants at module boundaries across all supported languages (Python, Rust, C\#, Java, C)8. Capability denials are strictly evaluated; the suite proves that when a requisite capability is withheld by the broker, the execution plan is safely neutralized, generating a missing-capability error code without corrupting the internal goal state8. Privacy is maintained at the absolute edge of the ingestion layer. Using high-performance redaction mechanisms modeled on Microsoft Presidio, the system performs named entity recognition (NER) and pattern matching to strip personally identifiable information (PII) before it ever enters the epistemic ledger17. This ensures that the cryptographic hashes used for evidence tracking are devoid of sensitive data, allowing raw traces and event logs to be safely exported to centralized OpenTelemetry collectors without violating compliance requirements.

D. Simplified Chinese Guide

现代机器智能架构的困境在于连续潜在空间与形式化符号本体之间的表征鸿沟2。传统的可观测性侧重于识别运行时异常、追踪远程过程调用（RPC）以及验证预期的机械行为11。大语言模型（LLM）的可观测性在此基础上进行了扩展，使用标准语义约定来监控动态的、随机的输出——例如追踪 Token 使用量、成本和延迟5。然而，本体机器（Ontological Machine）的可观测性需要第三个层级：认识论与因果追踪。当系统摄取某个观察结果时，可观测性管道必须记录源数据的密码学摘要，追踪其转化为形式化假设的过程，并记录将其提升为已验证知识的因果干预步骤3。 该实现大量利用了 OpenTelemetry GenAI 语义约定。在过去，追踪系统使用通用的标签，但 GenAI 特别兴趣小组（SIG）在 gen\_ai.\* 命名空间下建立了一套标准化的词汇表。早期的属性（如 gen\_ai.system）已被弃用，取而代之的是 gen\_ai.provider.name，而操作属性则依赖于 gen\_ai.request.model、gen\_ai.response.model 和 gen\_ai.usage.input\_tokens5。本体机器框架将这些标准属性注入其 Rust 和 Python 执行上下文中。此外，该系统实现了 OpenInference 规范，以深入洞察复杂的智能体编排、外部工具调用以及检索增强生成（RAG）过程6。OpenInference 跨度（Spans）捕获了工具提案和能力评估的丰富语义负载，使得外部平台（如 Arize Phoenix 或 Datadog）能够渲染复杂的执行图12。 由于 OMI 系统被设计为可执行的证明，时间独立性至关重要。挂钟时间（Wall-clock time）绝不能决定认知操作的语义输出，也不应影响认识论账本的确定性执行8。可观测性框架捕获规范的事件流，从而允许极其精确的重放。如果在开发或生产过程中发生回归错误，系统将导出确切的事件谱系。验证工具（Verification Harness）将重放这一谱系，并逐字节比较不同执行过程中的状态哈希值。通过这种方式，该框架隔离了发生分歧的确切认知步骤，彻底消除了生成式 AI 系统中通常需要的概率性调试过程8。 对此类系统的评估超越了简单的文本匹配或“以 LLM 作为裁判”（LLM-as-a-judge）的随机评分。本体机器采用了由英国人工智能安全研究所（UK AI Security Institute）开发的 Inspect AI 框架，来编排跨语言的评估测试夹具14。Inspect AI 允许开发者定义评估数据集、附加求解器（Solvers，代表智能体的提示和工具使用策略）并应用评估器（Scorers）来验证输出7。OMI 验证工具使用语言无关的 JSON 格式夹具，在所有支持的语言（Python、Rust、C\#、Java、C）中验证模块边界处的不变性8。系统对能力拒绝（Capability denials）进行严格评估；评估套件证明，当代理程序缺少必要的能力授权时，执行计划会被安全阻断，产生 missing-capability 错误代码，而不会破坏内部的目标状态8。 隐私保护在数据摄取层的最边缘进行。系统采用基于 Microsoft Presidio 模型的性能优化脱敏机制，通过命名实体识别（NER）和模式匹配，在个人身份信息（PII）进入认识论账本之前将其彻底剥离17。这确保了用于证据追踪的密码学哈希值完全不包含敏感数据，从而使得原始追踪和事件日志能够安全地导出到集中的 OpenTelemetry 收集器，而不会违反任何合规性要求。

E. Code Examples

The following 18 runnable examples demonstrate the integration of OpenTelemetry, OpenInference, Microsoft Presidio, and Inspect AI into the Ontological Machine runtime across Python and Rust environments.

1. Structured Logging

Source (logging\_example.rs):

Rust use tracing::{info, span, Level}; use tracing\_subscriber::FmtSubscriber;

pub fn main() { let subscriber \= FmtSubscriber::builder().json().finish(); tracing::subscriber::set\_global\_default(subscriber).unwrap();

let event\_id \= "evt\_001\_alpha"; let ontology\_node \= "entity\_user\_profile"; info\!(event\_id, ontology\_node, "Observation normalized into ontology boundaries"); }

Commands: rustc logging\_example.rs && ./logging\_example Output: {"timestamp":"2026-08-31T14:46:58Z","level":"INFO","fields":{"event\_id":"evt\_001\_alpha","ontology\_node":"entity\_user\_profile","message":"Observation normalized into ontology boundaries"}} Tests: Assert the JSON output parses correctly and contains the key ontology\_node. Privacy Notes: Operates entirely on abstract ontology nodes; no personal data logged. Links:11

2. Correlation and Trace Identifiers

Source (correlation.rs):

Rust use opentelemetry::trace::TraceContextExt; use tracing\_opentelemetry::OpenTelemetrySpanExt; use tracing::{info\_span, Span};

pub fn main() { let span \= info\_span\!("omi\_ingest\_observation"); let ctx \= span.context(); let trace\_id \= ctx.span().span\_context().trace\_id(); println\!("Correlation Trace ID: {}", trace\_id); }

Commands: cargo run \--bin correlation Output: Correlation Trace ID: 4bf92f3577b34da6a3ce929d0e0e4736 Tests: Assert trace\_id is a valid 32-character hexadecimal string representing a W3C trace parent. Privacy Notes: Trace IDs are cryptographically random and non-identifying. Links:6

3. Nested Spans

Source (nested\_spans.rs):

Rust use tracing::{info\_span, info};

pub fn main() { let root \= info\_span\!("omi\_cognitive\_loop"); let \_root\_enter \= root.enter();

let ontology \= info\_span\!("ontology\_grounding", domain="physics\_engine"); let \_ont\_enter \= ontology.enter(); info\!("Grounding spatial entities"); drop(\_ont\_enter);

let epistemology \= info\_span\!("evidence\_promotion", claim="clm\_01\_beta"); let \_epi\_enter \= epistemology.enter(); info\!("Promoting formal hypothesis to structural belief"); }

Commands: cargo run \--bin nested\_spans Output: A hierarchical trace graph showing omi\_cognitive\_loop [Figure omitted from source export] ontology\_grounding followed by evidence\_promotion. Tests: Validate the span evidence\_promotion recognizes omi\_cognitive\_loop as its parent ID. Privacy Notes: Internal operational metadata only; completely safe for external export. Links:3

4. Metrics Counters and Histograms

Source (metrics.rs):

Rust use opentelemetry::metrics::MeterProvider; use opentelemetry::KeyValue;

pub fn main() { let provider \= opentelemetry\_sdk::metrics::SdkMeterProvider::builder().build(); let meter \= provider.meter("omi\_runtime");

let token\_counter \= meter.u64\_counter("gen\_ai.usage.input\_tokens").init(); token\_counter.add(2048, &\[KeyValue::new("gen\_ai.request.model", "gpt-4o")\]);

let latency\_hist \= meter.f64\_histogram("gen\_ai.server.request.duration").init(); latency\_hist.record(0.85, &\[KeyValue::new("gen\_ai.operation.name", "chat")\]); }

Commands: cargo run \--bin metrics Output: GenAI metric attributes registered in the OpenTelemetry global registry. Tests: Query the OTel SDK memory exporter to ensure gen\_ai.usage.input\_tokens equals 2048\. Privacy Notes: Highly aggregated statistical metrics containing zero user context. Links:5

5. Error Classification

Source (error\_class.rs):

Rust use tracing::{error, span, Level};

pub fn trigger\_denial(error\_code: &str) { let s \= span\!(Level::ERROR, "capability\_check", error.type \= error\_code); let \_enter \= s.enter(); error\!("Action cleanly rejected by external capability broker"); }

pub fn main() { trigger\_denial("missing-capability"); }

Commands: cargo run \--bin error\_class Output: Span capability\_check enriched with the standardized attribute error.type=missing-capability. Tests: Validate that the serialized JSON log explicitly maps error.type to "missing-capability". Privacy Notes: Standardized error taxonomy ensures sensitive stack traces and user payloads are not leaked in boundary rejections. Links:6

6. Sensitive-Field Redaction

Source (redaction.py):

Python from presidio\_analyzer import AnalyzerEngine from presidio\_anonymizer import AnonymizerEngine

def redact\_observation(text: str) \-\> str: analyzer \= AnalyzerEngine() anonymizer \= AnonymizerEngine() results \= analyzer.analyze(text=text, language="en") anonymized \= anonymizer.anonymize(text=text, analyzer\_results=results) return anonymized.text

if \_\_name\_\_ \== "\_\_main\_\_": raw\_input \= "User Jane Doe logged in from 192.168.1.1." print(redact\_observation(raw\_input))

Commands: python3 redaction.py Output: User \<PERSON\> logged in from \<IP\_ADDRESS\>. Tests: Assert \<PERSON\> and \<IP\_ADDRESS\> are present in the output, while "Jane Doe" is strictly absent. Privacy Notes: Executes Microsoft Presidio at the exact system ingestion boundary to anonymize PII before it can contaminate the epistemic ledger. Links:17

7. Model-Request Timing and Token or Cost Accounting

Source (accounting.rs):

Rust use tracing::{info\_span, info};

pub fn main() { let span \= info\_span\!("llm\_inference", gen\_ai\_provider\_name \= "anthropic", gen\_ai\_request\_model \= "claude-3-5-sonnet-20241022", gen\_ai\_usage\_input\_tokens \= 1024, gen\_ai\_usage\_output\_tokens \= 256, gen\_ai\_usage\_cost\_usd \= 0.007 ); let \_guard \= span.enter(); info\!("Inference cycle complete and accounted"); }

Commands: cargo run \--bin accounting Output: Spans enriched with deterministic token and cost attributes compliant with OpenTelemetry GenAI v1.37+. Tests: Extract attributes from the span context and verify strict mapping to the gen\_ai.usage.\* namespace. Privacy Notes: Telemetry captures commercial volume and cost; the actual prompt payload is omitted to protect proprietary agentic logic. Links:5

8. Evidence-Transition Trace

Source (epistemic.rs):

Rust use tracing::info\_span;

pub fn main() { let claim\_id \= "clm\_492"; let evidence\_digest \= "sha256:8f434346648f6b96df89dda901c5176b10a6d83961dd3c1ac88b59b2dc327aa4";

let span \= info\_span\!("epistemic\_transition", claim \= claim\_id, evidence \= evidence\_digest, from\_state \= "hypothesis", to\_state \= "belief" ); let \_guard \= span.enter(); }

Commands: cargo run \--bin epistemic Output: Span epistemic\_transition emitted mapping a cryptographic evidence digest to a state transition. Tests: Validate to\_state rigorously equals belief and that the digest format matches sha256:\*. Privacy Notes: Evidence is tracked entirely by mathematical digest. The mutable source text is never inlined as authority, protecting systemic privacy. Links:3

9. Causal-Intervention Trace

Source (causal.rs):

Rust use tracing::info\_span;

pub fn main() { let span \= info\_span\!("causal\_intervention", target\_entity \= "engine\_valve", variable \= "pressure\_psi", threshold \= 45.5, world\_state \= "counterfactual" ); let \_guard \= span.enter(); }

Commands: cargo run \--bin causal Output: Span documenting the derivation of a counterfactual world state. Tests: Assert world\_state is logically separated from live state to prevent causal pollution. Privacy Notes: Floating point thresholds represent physical or logical laws and do not leak user-specific system content. Links:3

10. Goal-Selection Trace

Source (goal.rs):

Rust use tracing::info\_span;

pub fn main() { let span \= info\_span\!("goal\_proposal", goal\_id \= "goal\_optimize\_query", selected\_skill \= "skill\_sql\_rewrite", status \= "pending\_capability\_check" ); let \_guard \= span.enter(); }

Commands: cargo run \--bin goal Output: Span charting the internal orchestrator's goal proposal logic. Tests: Assert that the span status transitions precisely to pending\_capability\_check before any execution occurs. Privacy Notes: Represents internal machine logic entirely isolated from user-supplied prompt strings. Links:3

11. Tool-Proposal and Capability-Denial Trace

Source (denial.rs):

Rust use tracing::{error, info\_span};

pub fn main() { let tool \= "sys\_write\_file"; let required\_cap \= "write:filesystem";

let span \= info\_span\!("capability\_broker", tool\_name \= tool, required \= required\_cap); let \_guard \= span.enter(); error\!(error.type \= "missing-capability", "Execution safely denied by capability broker"); }

Commands: cargo run \--bin denial Output: Boundary rejection log emitted with error.type="missing-capability". Tests: Validate that the broader agent plan remains unmodified post-rejection. Privacy Notes: Security assertions are permanently logged; the rejected payload is discarded to prevent memory bloat. Links:8

12. Event Replay Trace

Source (replay.rs):

Rust use tracing::{info\_span, info};

pub fn main() { let span \= info\_span\!("deterministic\_replay", start\_seq \= 0, end\_seq \= 1045); let \_guard \= span.enter(); info\!("Replaying append-only memory events to reconstruct epistemic state"); }

Commands: cargo run \--bin replay Output: A trace orchestrating the sequential replay of the append-only log. Tests: The final replayed state hash must be cryptographically compared against the original run's state hash. Privacy Notes: Replay operations utilize canonical event hashes; any underlying PII was excluded during initial ingestion by Presidio. Links:3

13. Deterministic Comparison Between Two Runs

Source (compare\_runs.py):

Python import json

def compare\_hashes(run1\_path: str, run2\_path: str) \-\> bool: with open(run1\_path) as f1, open(run2\_path) as f2: hash1 \= json.load(f1).get("final\_state\_hash") hash2 \= json.load(f2).get("final\_state\_hash")

assert hash1 \== hash2, f"Nondeterminism detected: {hash1} \!= {hash2}" print("Exact deterministic match confirmed.") return True

if \_\_name\_\_ \== "\_\_main\_\_": compare\_hashes("run1.json", "run2.json")

Commands: python3 compare\_runs.py Output: Exact deterministic match confirmed. Tests: Execute with structurally identical inputs across two different architectures (e.g., Rust and Python) to guarantee byte-for-byte exactness. Privacy Notes: The script operates solely on 256-bit cryptographic hashes of the state vectors. Links:8

14. Local Trace Export

Source (local\_export.rs):

Rust use tracing\_subscriber::fmt::writer::MakeWriterExt; use std::fs::File;

pub fn main() { let file \= File::create("omi\_trace.log").unwrap(); let subscriber \= tracing\_subscriber::fmt().json().with\_writer(file).finish(); tracing::subscriber::with\_default(subscriber, || { tracing::info\!(event \= "local\_export\_init", "Exporting telemetry to local file system"); }); }

Commands: cargo run \--bin local\_export Output: Well-formed JSON structures serialized to omi\_trace.log. Tests: Programmatically read omi\_trace.log to ensure it contains valid newline-delimited JSON (NDJSON). Privacy Notes: The local file is written to secure, ephemeral CI storage and is automatically wiped after the pipeline terminates. Links:11

15. OpenTelemetry-Compatible Export

Source (otlp\_export.rs):

Rust use opentelemetry\_otlp::WithExportConfig; use tracing\_subscriber::prelude::\*;

pub fn main() { let tracer \= opentelemetry\_otlp::new\_pipeline() .tracing() .with\_exporter(opentelemetry\_otlp::new\_exporter().tonic().with\_endpoint("http://localhost:4317")) .install\_batch(opentelemetry::runtime::Tokio) .unwrap();

let telemetry \= tracing\_opentelemetry::layer().with\_tracer(tracer); tracing\_subscriber::registry().with(telemetry).init(); }

Commands: cargo run \--bin otlp\_export Output: Telemetry traces asynchronously exported over gRPC to an external OTLP collector. Tests: Interrogate the local Jaeger or OpenObserve instance to verify spans are successfully indexed. Privacy Notes: Production exports strictly require mutual TLS (mTLS) to prevent man-in-the-middle interception of trace data. Links:6

16. Evaluation Dataset Runner

Source (eval\_runner.py):

Python from inspect\_ai import Task, task, eval from inspect\_ai.dataset import json\_dataset from inspect\_ai.scorer import model\_graded\_fact from inspect\_ai.solver import generate

@task def omi\_ontology\_validation(): return Task( dataset=json\_dataset("fixtures.json"), solver=generate(), scorer=model\_graded\_fact() )

if \_\_name\_\_ \== "\_\_main\_\_": eval(omi\_ontology\_validation(), model="openai/gpt-4o")

Commands: inspect eval eval\_runner.py \--model openai/gpt-4o Output: The Inspect Viewer TUI launches, displaying evaluation transcripts, sample completion rates, and final scores. Tests: The evaluation suite asserts that the structural score must exceed a 0.95 threshold for formal ontology conformance. Privacy Notes: The evaluation JSON fixtures contain purely synthetic, artificially generated data scenarios. Links:7

17. Regression Dashboard Input

Source (regression.py):

Python import json

def output\_regression\_metrics(): \# Extracted from negative\_conformance tests metrics \= { "accepted": 1, "codes": \["missing-capability", "illegal-transition", "stale-generation"\], "rejected": 4 } with open("dashboard\_input.json", "w") as f: json.dump(metrics, f, indent=2)

if \_\_name\_\_ \== "\_\_main\_\_": output\_regression\_metrics()

Commands: python3 regression.py Output: A dashboard\_input.json payload containing sorted and normalized taxonomy counts. Tests: Execute a JSON schema validation to ensure the output strictly conforms to the expected dashboard ingestion schema. Privacy Notes: The payload is completely anonymized, consisting solely of statistical capability counts. Links:8

18. Root-Cause Report Assembled from Traces and Events

Source (root\_cause.py):

Python import json

def generate\_rca(trace\_log: str): try: with open(trace\_log) as f: spans \= \[json.loads(line) for line in f if line.strip()\]

errors \= \[s for s in spans if s.get("level") \== "ERROR"\] print(f"Root Cause Report: {len(errors)} anomalies detected.") for e in errors: err\_type \= e.get('fields', {}).get('error.type', 'unknown') print(f"- Boundary failure: {err\_type}") except FileNotFoundError: print("Trace log not found. Skipping RCA.")

if \_\_name\_\_ \== "\_\_main\_\_": generate\_rca("omi\_trace.log")

Commands: python3 root\_cause.py Output: A synthesized summary mapping the missing-capability attribute directly to the failed capability\_broker span. Tests: Assert the standard output string contains the exact phrase Boundary failure: missing-capability. Privacy Notes: Operates efficiently on fully redacted JSON trace logs, guaranteeing safety for widespread developer distribution. Links:8

F. Project Directory

omi-observability/ ├── Cargo.toml ├── src/ │ ├── main.rs │ ├── tracing\_init.rs │ ├── epistemic/ │ │ └── ledger\_traces.rs │ ├── causal/ │ │ └── intervention\_spans.rs │ └── agent\_runtime/ │ └── broker\_denials.rs ├── eval/ │ ├── pyproject.toml │ ├── requirements.txt │ ├── eval\_runner.py │ ├── compare\_runs.py │ └── fixtures.json └── dashboard/ ├── dashboard\_input.json └── root\_cause.py The project separates high-performance execution tracking in src/ (Rust) from the evaluation, regression detection, and dashboard assembly in eval/ and dashboard/ (Python).

G. Evaluation Recipes

The automated validation of the OMI framework requires explicit testing against multiple cognitive failure modes. Inspect AI and DeepEval serve as the orchestration engines for these tests16.

1. Exact-Match Tests: Asserts that deterministic inputs yield identical canonical hashes across language implementations. For example, executing a complex ontological query in Rust must produce a final\_state\_hash that perfectly matches the Python reference implementation byte-for-byte8.

2. Structured-Output Validity: Validates that all emitted OpenTelemetry attributes strictly conform to specification. Using a JSON schema validator, the pipeline checks that traces never emit deprecated gen\_ai.system attributes and strictly utilize gen\_ai.provider.name5.

3. Retrieval Quality: Evaluates the associative retrieval of the hyperdimensional ontology kernel. Utilizing DeepEval's ContextualPrecisionMetric, the system proves that the retrieved semantic concepts are highly relevant to the provided observation2.

4. Evidence Citation Accuracy: A custom Inspect AI scorer scans the epistemic state transition logs to ensure every knowledge claim maps directly to an immutable evidence\_digest. A zero-evidence promotion attempt results in an immediate evaluation failure3.

5. Capability-Denial Correctness: Uses table-driven conformance tests. The framework deliberately requests execution without write:filesystem permissions. The test verifies that this action triggers an explicit missing-capability rejection without modifying the active plan state8.

6. Latency Distribution: Analyzes the gen\_ai.server.request.duration histogram percentiles to isolate time-to-first-token (TTFT) metrics, distinguishing between network lag and computational generative interventions6.

7. Resource Use: Tracks tokens consumed versus cached tokens read per intervention, evaluating efficiency by parsing gen\_ai.usage.input\_tokens and usage.cache\_read.input\_tokens attributes5.

8. Nondeterminism Detection: The system continuously replays event streams across isolated CI environments. Any detected mismatch in the final\_state\_hash identifies structural nondeterminism and automatically halts the deployment pipeline8.

H. Decision and Privacy Matrices

I. Source Ledger

The following 30 projects form the core ecosystem for implementing OMI observability.

J. Integration JSON

JSON { "integration": { "version": "1.0", "architecture": "OntologicalMachine", "telemetry": { "tracing\_protocol": "OTLP/gRPC", "metrics\_schema": "OpenTelemetry\_GenAI\_v1.37", "agent\_schema": "OpenInference\_v0.3.22" }, "epistemic\_ledger": { "transitions": \["assertion", "hypothesis", "belief", "knowledge", "falsified"\], "event\_store": "AppendOnlyMemory", "redaction": "Microsoft\_Presidio" }, "evaluation": { "orchestrator": "Inspect\_AI", "metrics\_engine": "DeepEval", "conformance\_fixtures": "language-neutral-json", "deterministic\_replay": true }, "capability\_broker": { "default\_policy": "deny", "error\_taxonomy": \[ "missing-capability", "illegal-transition", "duplicate-id", "stale-generation" \] } } }

Works cited

1. Machine Epistemology — Full report \- OntologicalMachine.com, https://ontologicalmachine.com/en-us/C/research/machine-epistemology/full

2. Hyperdimensional OMI — Full report \- OntologicalMachine.com, https://ontologicalmachine.com/en-us/CSharp/research/hyperdimensional-omi/full

3. Deep OMI Reference Implementation — Rust, https://ontologicalmachine.com/en-us/Rust/reference-implementation

4. GitHub \- Arize-ai/openinference: OpenTelemetry Instrumentation for, https://github.com/Arize-ai/openinference

5. How OpenTelemetry Traces LLM Calls, Agent Reasoning, and MCP, https://greptime.com/blogs/2026-05-09-opentelemetry-genai-semantic-conventions

6. OpenTelemetry for LLMs: How we instrument a multi-provider AI, https://www.truefoundry.com/blog/opentelemetry-llm-gateway-instrumentation

7. Inspect AI Evals Tutorial (UK AISI) — 2026 Guide \- QASkills.sh, https://qaskills.sh/blog/inspect-ai-evals-tutorial-2026

8. Verification Harness and Cross-Language Conformance — Python, https://ontologicalmachine.com/en-us/Python/build/verification-harness

9. GitHub \- Arize-ai/openinference at analysepodcast.com, https://github.com/Arize-ai/openinference?ref=analysepodcast.com

10. Agent Runtime and Capability Separation \- OntologicalMachine.com, https://ontologicalmachine.com/en-us/Rust/reference-implementation/agent-runtime

11. OpenTelemetry for LLMs: Complete SRE Guide for 2026, https://openobserve.ai/blog/opentelemetry-for-llms/

12. Best AI Agent Evaluation Tools for Production Teams (2026), https://www.augmentcode.com/tools/best-ai-agent-evaluation-tools

13. OpenTelemetry Instrumentation \- Datadog Docs, https://docs.datadoghq.com/llm\_observability/instrument/otel\_instrumentation/

14. inspect-ai \- PyPI, https://pypi.org/project/inspect-ai/

15. Inspect: A framework for large language model evaluations · GitHub, https://github.com/UKGovernmentBEIS/inspect\_ai

16. Inspect, https://inspect.aisi.org.uk/

17. How to Build PII Detection \- OneUptime, https://oneuptime.com/blog/post/2026-01-30-llmops-pii-detection/view

18. \[2026\] OpenTelemetry for LLM Observability — Self-Hosted Setup, https://dev.to/rapidclaw/2026-opentelemetry-for-llm-observability-self-hosted-setup-335o

19. AI Observability metrics reference \- Dynatrace Documentation, https://docs.dynatrace.com/docs/observe/dynatrace-for-ai-observability/metrics-reference

20. Setting Up LLM Observability Pipelines in 2026 \- MLflow, https://mlflow.org/articles/setting-up-llm-observability-pipelines-in-2026/

21. Presidio PII Masking with LiteLLM \- Complete Tutorial, https://docs.litellm.ai/docs/tutorials/presidio\_pii\_masking

22. Arize-ai/arize-otel-python \- GitHub, https://github.com/Arize-ai/arize-otel-python

23. Evaluating models with Inspect \- Hugging Face, https://huggingface.co/docs/inference-providers/guides/evaluation-inspect-ai

24. deepeval · PyPI, https://pypi.org/project/deepeval/3.2.3/