Machine Intelligence Runtime (MIR) Plan

Full report

On this page

Executive Summary: A Machine Intelligence Runtime (MIR) builds on the MiRuntime conceptual framework by providing an implementation for orchestrating AI-driven tasks. This report audits the existing MIR developer resources, compares leading model-serving and orchestration platforms, and derives functional/nonfunctional requirements. We explore architecture alternatives (monolith vs microservices, edge vs cloud, serverless vs dedicated, hardware acceleration), propose a concrete API/SDK design with endpoints, authentication, and telemetry hooks, and outline deployment patterns (containers, Helm/operator, CI/CD). Finally we present an implementation roadmap with milestones, testing plan, and success metrics. Leading open-source projects (KServe, Seldon, BentoML, Ray Serve, TorchServe, NVIDIA Triton, MLflow) and cloud-native practices inform our recommendations.

MIR Specification Audit

The MiRuntime developer portal defines a conceptual runtime contract but no code. Key elements include: a versioned JSON runtime contract capturing task intent (objective, success criteria), context (data sources, memory scopes), authority (allowed tools, policies, approvals), execution state (budget, checkpoints), and evidence (events, artifacts). For example, their JSON template has fields like "intent": {"objective": ...}, "execution": {"state": "running", "budgets": ...}, "evidence": {"events": [...], ...}. Tool contracts are likewise specified: each tool/action has an ID, input schema, permission class, side-effects, credential scope, timeouts, idempotency, rollback policy, and evidence requirements. An evidence schema defines event logs (types like run.created, tool.completed, etc.) with fields for run ID, sequence, timestamp, actor, subject, data, and hashes. An implementation checklist enumerates requirements: clear system boundaries, explicit identity/permission models, schema validation, finite budgets, idempotency, checkpoints before side-effects, and comprehensive logging of policy decisions and artifacts. In short, MIR currently provides a design blueprint (contracts, events, policies) for a safe AI runtime; our task is to turn this blueprint into an architecture and product design.

Key MIR principles are: keep policy decisions outside the model, make side-effects recoverable and auditable, and treat components (models, tool backends, memory stores) as replaceable adapters. These guide the requirements below.

Industry Platforms & Runtimes

Many open-source systems exist for serving and orchestrating ML/AI workloads. The table below compares prominent projects:

These platforms illustrate common patterns: most use containers on Kubernetes (KServe, Seldon, Triton, BentoML on K8s) or cluster frameworks (Ray), and expose REST/gRPC endpoints with autoscaling support. KServe and Seldon provide high-level CRDs to declaratively deploy models, while BentoML and TorchServe focus on Python-based model packaging. Triton is the de facto high-performance GPU server, often run inside an operator (e.g. KServe can launch Triton pods). MLflow provides convenient standardized endpoints but has no native autoscaling or multi-tenant management.

Core Requirements

From MIR’s design and industry practices, we derive core requirements:

• API Surface: The runtime should expose a REST/gRPC API (and language SDKs) to submit tasks (“runs”), query status, and retrieve results. Endpoints might include /runs (POST new run), /runs/{id} (GET status/metadata), /runs/{id}/cancel, /tools (GET available tools), etc. Each run’s input includes the runtime contract (intent, context, policies) and any initial data. The API must support pagination and filters (e.g. list runs by user/project).

• SDKs: Client libraries in C#, Python, and (optionally) TypeScript, to wrap API calls. These should include data models matching the contract schema. (For example, a C# RunRequest class with [Display(Name="Objective")] attributes on properties, and similar for tool invocations.)

• Model Inference: Ability to invoke one or more AI models per run. Support both synchronous (request/response) and asynchronous or streaming modes (useful for chat or LLM tokens). Support batching where possible to improve throughput (e.g. group multiple requests or tokens). Support common model formats: TorchScript, ONNX, TensorFlow SavedModel, HuggingFace pipelines, etc. Possibly use standard protocols (e.g. ONNX Runtime or KFServing’s V1/V2 protocol).

• Tool and Agent Orchestration: The runtime must manage “tools” or external actions invoked by the model (per MIR’s tool contracts). This includes schema-checking inputs, enforcing permission and credential scopes, and capturing tool outputs. Workflows should support branching logic (if/else), loops, and human approval steps. Complex tasks may involve chaining multiple model calls or tool calls in sequence or graph (InferenceGraph).

• State and Memory: Support storing and retrieving conversation state or knowledge (“memory”) across steps or runs. Provide scoped data stores (user session memory, project memory) while respecting data classification and retention policies.

• Policy and Approval: Enforce authority and governance per run. For example, certain actions require explicit approval (as per the “requiredApprovals” field in the contract). The system must halt for approvals and log decisions as evidence.

• Evidence & Logging: Every state change and side-effect must emit structured events for audit. Following the evidence schema, log events like run.created, tool.started/completed, checkpoint.created, run.completed/failed with metadata (actor, timestamp, hash). Store intermediate artifacts (models outputs, diffs) and any unresolved uncertainties at termination.

• Security: Strong multi-tenancy and isolation. Authenticate API users and agents (e.g. tokens or OAuth). Authorize each action via tool permission classes. Network egress must be controlled (only allow approved destinations). Secrets (credentials) must never leak to models or logs. Enforce least privilege (MIR checklist: “Tools are default-deny and categorized by side effect”).

• Scalability & Performance: Target high throughput and low-latency. Support horizontal autoscaling (e.g. K8s HPA on custom metrics). For LLMs or heavy models, support batching and multi-GPU serving. Provide service-level agreements (SLAs), e.g. 99th percentile latency budgets. Use async processing for long-running tasks with checkpoints to avoid loss on restart.

• Reliability: Durable state and checkpointing before side effects. Support retry/circuit-breaker logic and graceful termination (MIR checklist: idempotency, timeouts, checkpoints before effects). Ensure high availability (run control plane with redundancy, e.g. K8s control plane vs dedicated service).

• Observability: Emit metrics (requests/sec, queue depth, model load times) and integrate tracing across components. Use OpenTelemetry conventions for traces/logs. Provide dashboards for run status and system health. Allow operators to detect stuck or looping runs via telemetry.

• Cost Efficiency: Optimize resource usage (multi-model sharing, GPU memory scheduling). Prefer serverless scale-to-zero for idle services to save cost (as in KServe). Provide cost accounting by run or project.

Architecture Options & Tradeoffs

We consider several architectural dimensions:

• Monolithic vs Microservices:
• Monolithic (all logic in one service): simpler deployment and testing, single code base. Downside: harder to scale individual components, risk of a single failure, slower development for parallel teams.
• Microservices (separate services for API gateway, scheduler, model servers, policy engine, evidence store, etc.): enables independent scaling (e.g. scale model inference pods separately from API front-end), technology heterogeneity (use Python for ML, Go/C# for control plane), and fault isolation. But it adds complexity (networking, service discovery) and operational overhead (multiple images/infra).
• Trade-off: For a MIR with diverse functions (API, orchestration, logging), a microservices/Kubernetes-based approach is favored for scalability and resilience, though a simple prototype could be monolithic.

• Edge vs Cloud:
• Edge-deployed (on-prem or on-device inference): Offers low latency and local data use (e.g. offline or private settings), but limited compute (fewer GPUs, resource-constrained devices) and complex deployment (heterogeneous environments). Harder to implement multi-node training or global coordination.
• Cloud-based (centralized servers or clusters): Easier to scale (virtually infinite GPUs), simplifies updates and orchestration, and allows global resource pooling. But involves network latency and cloud costs, and raises data privacy concerns if sensitive data is sent off-prem.
• Trade-off: Likely start cloud-first (Kubernetes on cloud or on-prem clusters), but design so parts of the runtime (e.g. execution or tools) could run on local nodes (a “hybrid” local-first control as MIR suggests) if needed.

• Serverless vs Dedicated:
• Serverless (FaaS / scale-to-zero): Functions or Knative services that spin up on demand (as in KServe’s Knative mode). Pro: cost-saving for intermittent workloads, automatic elasticity for spiky traffic. Con: cold starts (inference warm-up delay), limitations on execution time and request size, less control over hardware (e.g. no dedicated GPU support in many FaaS).
• Dedicated (persistent pods/services): Keep model servers always hot, ensuring low-latency responses and ability to use GPUs continuously. But costs accumulate when idle.
• Trade-off: Use a hybrid model: critical low-latency endpoints or chatbots stay in dedicated services; batch jobs and low-traffic services use scale-to-zero. KServe supports both “Serverless” and “RawDeployment” modes.

• Hardware Acceleration:
• Support GPUs (NVIDIA/AMD) for large models and large-batch throughput. Enable multi-instance GPU (MIG) or NVIDIA MPS for sharing (e.g. KServe/CNV leverages MIG). Use TPUs, IPUs, or other AI accelerators where available.
• Use frameworks like NVIDIA Triton (with dynamic batching) to maximize GPU utilization. Triton’s architecture (Fig. below) schedules incoming inference requests per model and batches them before invoking the GPU backend. This yields high throughput for deep models while exposing HTTP/gRPC endpoints.

Figure: NVIDIA Triton Inference Server high-level architecture (requests via HTTP/GRPC, model scheduler, framework backends)

• Data-plane Separation: Adhering to MIR’s separation of control vs runtime plane, one can split the system into a control plane (API, policy engine, scheduler, metadata store) and data plane (model-serving pods, transformers, memory store). This aligns with KServe’s design (Control Plane CRDs vs Data Plane predictors/transformers).

In summary, we recommend a Kubernetes-based microservices architecture with CRD/operator support. GPUs and Triton can accelerate inference; policy and evidence capture can be separate services/pods. Use serverless scaling for non-critical flows and dedicated pods for core inference. Edge deployment can be considered as a variant (maybe via a Kubernetes cluster on-prem).

API & SDK Design

We propose a RESTful API with JSON and gRPC options, plus client SDKs (C#, Python, TS). Key endpoints:

Authentication & Versioning: Each call requires an Authorization bearer token representing the user/agent. Use OAuth2/JWT with scopes (run.create, run.read, tool.invoke). Support API versioning (e.g. POST /v1/runs). TLS is mandatory. Store tokens outside of task context.

Telemetry: All API and SDK calls should emit structured logs and metrics. Use OpenTelemetry or Prometheus for metrics (e.g. request count, latency, run queue length, GPU utilization). Include trace IDs in responses for correlation. Hooks: each runtime action (model call, tool call) should be traced and logged, and evidence events should be written to a durable event store (could be ElasticSearch, Kafka, etc.). Operators will use these logs to detect anomalies (per MIR checklist).

Sample SDK (C#): A C# data model for the run request might be:

/// <summary>Request body to start a new runtime task.</summary>
public class RunRequest
{
    [Display(Name = "Objective")]
    public string Objective { get; set; }

    [Display(Name = "Risk Class")]
    public string RiskClass { get; set; }

    [Display(Name = "Success Criteria")]
    public List<string> SuccessCriteria { get; set; }

    [Display(Name = "Sources")]
    public List<string> DataSources { get; set; }

    [Display(Name = "Memory Scopes")]
    public List<string> MemoryScopes { get; set; }

    [Display(Name = "Prohibited Data Classes")]
    public List<string> ProhibitedDataClasses { get; set; }

    [Display(Name = "Allowed Tools")]
    public List<string> AllowedTools { get; set; }

    [Display(Name = "Write Policy")]
    public string WritePolicy { get; set; }

    [Display(Name = "Required Approvals")]
    public List<string> RequiredApprovals { get; set; }

    [Display(Name = "Checkpoint Policy")]
    public string CheckpointPolicy { get; set; }

    [Display(Name = "Budgets")]
    public BudgetsRequest Budgets { get; set; }
}

/// <summary>Budget constraints for a run.</summary>
public class BudgetsRequest
{
    [Display(Name = "Steps")]
    public int Steps { get; set; }  // e.g. max tool invocations

    [Display(Name = "Elapsed Seconds")]
    public int ElapsedSeconds { get; set; }  // wall-clock timeout
}

(The BudgetsRequest structure includes fields like step count and time limit.)

The SDK would have async methods like CreateRunAsync(RunRequest req), GetRunStatusAsync(string runId), etc. Libraries should serialize/deserialize ISO-8601 timestamps (UTC). For telemetry, SDK calls should log at INFO level (with correlation IDs) and expose metrics (via the client or use platform OTEL integration).

Deployment Patterns

• Containers: Package each service (API server, model runners, tool executors, database) as Docker containers. Use Kubernetes (or equivalent) to deploy. Each model might be served by a separate pod or share a multi-model server (Triton).
• Helm & Operators: Provide a Helm chart or Kubernetes Operator for installing MIR. This could include CRDs for RunRequest or Tool, similar to KServe’s InferenceService. Operators can handle tasks like rolling updates and scaling of inference pods.
• CI/CD Pipelines: Use GitOps (e.g. ArgoCD) or CI (GitHub Actions, Jenkins) to build and push images on commits, and to apply Helm charts. Incorporate automated tests in CI: unit tests for code, contract compliance tests, and simple deployment tests (e2e smoke on a dev cluster).
• Configuration: Store configuration (model registry, tool policy, secrets) in Kubernetes ConfigMaps/Secrets or HashiCorp Vault. Use Kubernetes RBAC to isolate runtime components by namespace/tenant.
• Monitoring: Deploy Prometheus/Grafana and ELK stack (or OTEL collector + backends) to collect metrics and logs from all pods. Use Kubernetes Horizontal Pod Autoscaler (with custom metrics like GPU usage or queue depth) to auto-scale inference pods.
• Storage: Use shared storage (S3/GCS) for large models and artifacts. A distributed database (like PostgreSQL or CockroachDB) for metadata (run status, contract versions). Optionally use Kafka or Pulsar to stream evidence events to a log store.

Implementation Roadmap

Below is a sample 12–18 month timeline with milestones, tests, and metrics. Tasks overlap as indicated.

gantt
    title 2026–2027 MIR Implementation Roadmap
    dateFormat  YYYY-MM-DD
    section Requirements & Design
    Research & Requirements        :done,    2026-06-24, 30d
    Architecture Design           :active, 2026-07-24, 45d
    API/SDK Specification         :2026-09-07, 30d
    section Development
    Prototype Core Runtime        :2026-10-07, 60d
    SDK & API Development         :2026-10-07, 45d
    Integrate Model & Tools       :2026-12-06, 45d
    section Testing & Validation
    Unit & Integration Tests      :2027-01-20, 30d
    Performance & Security Tests  :2027-02-19, 30d
    Pilot (Alpha) Deployment      :2027-03-21, 45d
    section Launch
    Beta Release & Feedback       :2027-05-05, 30d
    Production Release v1.0       :2027-06-04, 15d
    Operational Monitoring Setup  :2027-06-19, 15d

Milestones & Tests: Early deliverables include architecture review and API prototype. The Core Runtime Prototype implements basic contract ingestion, a sample model server (e.g. a toy LLM), and tool invocation. Throughout development, build automated contract compliance tests (ensuring no invalid contract states), fault-injection tests (simulate model failures), and security audits (pen-testing for unauthorized tool use). Performance targets (e.g. 95% tail latency under load, 99.9% uptime) should be defined. For each release, define success metrics: correctness of evidence logs, resource usage per request, and handling of edge cases (cancellation, circuit breakers).

Validation: Pre-release QA must cover the MIR Implementation Checklist items: simulate high-risk runs (requiring manual approval), test idempotency and recovery, verify audit logs, and ensure no sensitive data is exposed. Collect telemetry to validate autoscaling rules (e.g. ensure pods scale on queue length as intended). Feedback from pilot users will drive refinements before v1.0.

Open-Source & References: Our design draws on many official sources: KServe’s docs, BentoML’s docs, Ray Serve docs, TorchServe docs, NVIDIA Triton’s user guide, and MLflow’s docs. Primary literature on ML serving patterns (e.g. Kubeflow/MLOps whitepapers) and CNCF/Kubernetes best practices should guide detailed implementation. All key decisions and architecture diagrams should be referenced from these authoritative sources.