Building a Fully Custom C ML Runtime for Direct LLM Interaction

Full report

On this page

Executive summary

A fully custom C# ML runtime that talks directly to frontier LLM APIs and local model backends is feasible, but the most robust design is not a single monolith. The strongest architecture is a managed-first control plane in C# with pluggable transports and backends: cloud adapters for OpenAI and Anthropic, an OpenAI-compatible server adapter for Meta-oriented or local deployments via Llama Stack or llama.cpp, and an in-process local inference layer for ONNX Runtime and, if you choose to go fully “against the metal,” a custom GGUF/GGML execution backend. That recommendation is consistent with the attached research corpus, which emphasizes managed-code control, GGUF parsing, memory-mapped weights, SIMD, KV-cache engineering, and staged replacement of native dependencies rather than a naïve rewrite. It also aligns with Teleodynamic’s architecture of a fast execution loop, a slower structural-governance loop, explicit resource budgeting, immutable traces, and “no-op dominance” when a change is not justified.

The key design move is to separate token-generation mechanics from runtime governance. The fast loop owns tokenization, prompt assembly, retrieval, prefill, decode, streaming, and structured-output enforcement. The slow loop owns model routing, template revisions, adapter loading, plugin lifecycle, cost controls, rollback, and human-review gates. Teleodynamic’s resource law and operator library are especially useful here: they provide a rigorous way to model whether a new plugin, prompt template, adapter, cache index, or worker topology should be added, merged, retired, or left unchanged. In practical terms, that becomes a resource governor over latency, token spend, memory pressure, review burden, failure rate, and uncertainty.

For protocol choice, OpenAI’s Responses API is the most feature-rich direct HTTP surface for multimodal inputs, tool calling, stateful interactions, background jobs, and SSE streaming; its Realtime stack adds WebSocket, WebRTC, and SIP options for low-latency audio and live sessions. Anthropic’s Messages API is cleaner if you want a narrower, event-typed SSE model with very explicit tool-use deltas and a strong batch-processing story. Meta’s most practical current integration surface is not a raw “Llama 2 API,” but Llama Stack, which presents an OpenAI-compatible /v1/responses surface and an Anthropic-compatible /v1/messages surface over pluggable providers; alternatively, llama.cpp exposes a lightweight OpenAI-compatible local HTTP server over GGUF models. ONNX Runtime is the best in-process “safer local” path for C# if you want cross-platform execution providers, quantized graph execution, and edge/mobile friendliness. A custom GGUF runtime is the highest-effort, highest-control path and should be pursued only when you need deep control over quantization formats, KV paging, tokenizer fidelity, structured decoding, or hardware-specific kernels.

The attached research strongly supports a staged execution plan: begin with a provider-agnostic C# runtime contract and observability model, then add hosted adapters, then local server adapters, then ONNX in-process inference, and only after that implement the custom GGUF pathway with memory mapping, tokenizer metadata support, SIMD kernels, and paged KV cache. That sequence minimizes delivery risk while preserving a path to a genuinely custom runtime. It also respects Teleodynamic’s claim boundaries: architecture pages, roadmaps, and resource controls are presented as explanatory engineering patterns rather than proof of a deployed autonomous runtime, so this report uses them as design heuristics, not certification claims. Where the attached documents are silent, I make explicit assumptions and mark them as such.

Research basis and assumptions

The attached research set appears to converge on five architectural commitments. First, the runtime should be managed-first, with C# owning orchestration, memory strategy, and most of the execution surface rather than merely wrapping a native library. Second, it should treat GGUF as the canonical local model artifact, using metadata-rich loading, quantization-aware execution, and—ideally—memory mapping rather than bulk deserialization. Third, the runtime should optimize decode throughput through SIMD, fused execution, row-aware layout choices, and paged KV cache management. Fourth, it should support structured decoding, adapter loading, and eventual multimodal sidecars. Fifth, the work should proceed in phases, preserving parity and testability while replacing dependency-heavy native pieces one layer at a time.

Teleodynamic contributes a complementary governance model rather than a drop-in inference engine. Its most relevant ideas for this runtime are the two-timescale architecture; an endogenous resource budget R(t) that blocks unaffordable growth; an operator library of split, merge, add, retire, and no-op; claim-boundary discipline; and local sandboxing with explicit safety gates. Teleodynamic also repeatedly states that its public architecture and roadmap pages are explanatory and review-gated, not proof of a live autonomous safety system. That matters: the right way to apply Teleodynamic here is as a design discipline for your runtime’s control plane, not as evidence that any given policy is already operationally safe.

The main assumptions in this report are straightforward. I assume the unspecified parts of the attached research would reinforce—not contradict—the visible themes of managed execution, local model control, auditability, and performance engineering. I also assume you want a runtime that can switch among hosted providers and local backends without changing the application contract, because that is the lowest-risk way to combine direct LLM interaction with future custom inference work. Finally, I assume that “fully custom” means control of the runtime surface and evolution path, not necessarily that day one must include a from-scratch transformer kernel for every supported model family. That assumption follows both the staged rewrite logic in the attached material and Teleodynamic’s bias toward no-op over overclaim.

Target architecture

The recommended runtime has three layers. The control plane is pure C#: request admission, budget governance, prompt assembly, state management, policy checks, provider routing, trace emission, and plugin/tool orchestration. The transport layer exposes adapters for HTTPS + SSE, gRPC, and WebSockets, depending on backend and latency needs. The execution layer hosts backends for OpenAI, Anthropic, OpenAI-compatible local servers, ONNX Runtime, and eventually custom GGUF execution. This preserves a single application-facing contract while allowing the backend mix to evolve. That architecture is consistent with Teleodynamic’s fast loop/slow loop model, Llama Stack’s server-abstraction argument, and the attached research’s staged migration away from native wrappers.

flowchart TB
    A[Client Applications] --> B[Runtime Gateway]
    B --> C[Admission and Policy Layer]
    C --> D[Session Orchestrator]
    D --> E[Prompt and Template Engine]
    D --> F[Resource Budget Governor]
    D --> G[Evidence and Trace Ledger]

    E --> H[Tokenizer and Chat Template Resolver]
    H --> I[Provider Router]

    I --> J[OpenAI Adapter]
    I --> K[Anthropic Adapter]
    I --> L[Llama Stack or llama.cpp Adapter]
    I --> M[ONNX Runtime Backend]
    I --> N[Custom GGUF Backend]

    J --> O[Streaming Aggregator]
    K --> O
    L --> O
    M --> O
    N --> O

    O --> P[Safety and Structured Output Layer]
    P --> Q[Tool and Plugin Host]
    P --> R[Client Stream]

    F --> I
    F --> Q
    G --> S[Metrics and Telemetry Export]

The fast loop should contain only request-scoped decisions: template rendering, retrieval assembly, tokenization, prefill, decode, tool-call marshalling, stream aggregation, moderation or safety checks, and final output shaping. The slow loop should own topology and policy changes: model routing updates, prompt-template promotion, plugin activation or retirement, adapter versioning, cache-policy adjustment, model hot-swap rules, and rollback. Teleodynamic’s resource gate says structural changes should occur only if expected local gain repays action cost while preserving the viability floor; in runtime terms, you should not promote a new tool, adapter, cache index, or model default if it increases latency, spend, memory burden, or review burden faster than it improves outcomes.

flowchart LR
    subgraph Fast Loop
        A[Input]
        B[Template Render]
        C[Tokenize]
        D[Prefill]
        E[Decode and Stream]
        F[Structured Output and Safety]
        A --> B --> C --> D --> E --> F
    end

    subgraph Slow Loop
        G[Observe Traces]
        H[Evaluate Cost Latency Memory Review]
        I[Choose Add Merge Retire or No-op]
        J[Promote Config or Roll Back]
        G --> H --> I --> J
    end

    F --> G
    H --> K[R(t) Viability Gate]
    K --> I

A practical component map is below.

The memory and state model should be deliberately layered rather than ad hoc. Request state should hold correlation IDs, deadlines, cancellation tokens, and incremental stream state. Session state should hold conversation handles, provider-specific response IDs, template selections, and approved tool context. Model state should hold tokenizer metadata, weights or sessions, KV-cache handles, adapter handles, and warm-up facts. Shared state should hold prefix caches, embedding caches, grammar/schema caches, and plugin registries. Governance state should store blocked actions, warnings, review receipts, and rollout decisions. That last category is where Teleodynamic adds real value: “blocked action telemetry” is not noise; it is evidence that resource closure is doing its job.

For local backends, weights should not become giant managed objects if you can avoid it. GGUF was designed as a binary format for storing inference models, explicitly targeting single-file deployment, extensibility, and mmap compatibility, and the attached research argues for zero-allocation or near-zero-allocation loading with unmanaged pointers, aligned buffers, and page-driven access. That is the right direction for a custom GGUF backend. ONNX Runtime is the more conservative alternative: it is a cross-platform accelerator with execution providers, graph optimizations, and a C# API, which makes it well suited for an initial in-process backend while you build the deeper custom path.

Protocol and API choices

Protocol selection should be pragmatic. Use HTTPS + SSE as the default outward protocol because both OpenAI Responses and Anthropic streaming expose event streams over standard HTTP, and because it integrates cleanly with HttpClient, reverse proxies, and most enterprise networking. Use WebSockets when you need genuinely interactive duplex behavior, especially for Realtime sessions or browser/mobile event loops. Use gRPC primarily for east-west internal communication between your gateway and model workers, not as the default public ingress, because typed contracts, deadlines, and bidi streaming are valuable internally, but stream/channel limits and load-balancing behavior make it a more deliberate choice than plain HTTP. Reuse clients, channels, and pools in all cases.

The transport choices inside the runtime should also be standardized.

Prompt engineering and templating should be backend-aware rather than universalized too aggressively. OpenAI supports prompt objects and provider-side instructions; GGUF can embed tokenizer.chat_template and even tokenizer.huggingface.json; Teleodynamic’s developer guidance argues that outputs should carry warnings, confidence, and evidence context rather than bare authoritative strings. The consequence is architectural: your runtime should compile a canonical internal prompt object into provider-specific envelopes instead of treating the raw upstream message format as your domain model.

Core C# implementation patterns

The central implementation principle is to define a provider-neutral execution contract and make every provider or local backend conform to it. That lets the rest of your application remain oblivious to whether the response comes from OpenAI, Anthropic, Llama Stack, ONNX Runtime, or your own GGUF executor. It also creates a single place for Teleodynamic-inspired governance: budgets, no-op rules, plugin approvals, and evidence traces.

Request pipeline pattern

using System.Collections.Concurrent;
using System.Collections.ObjectModel;
using System.ComponentModel.DataAnnotations;
using System.Diagnostics;
using System.Runtime.CompilerServices;

namespace CustomLlmRuntime;

/// <summary>
/// Represents a provider-neutral LLM request.
/// </summary>
public sealed record LlmRequest(
    [property: Display(Name = "Provider")] string Provider,
    [property: Display(Name = "Model")] string Model,
    [property: Display(Name = "System Prompt")] string SystemPrompt,
    [property: Display(Name = "User Prompt")] string UserPrompt,
    [property: Display(Name = "Conversation Key")] string? ConversationKey,
    [property: Display(Name = "Enable Streaming")] bool EnableStreaming,
    [property: Display(Name = "Max Output Tokens")] int MaxOutputTokens,
    [property: Display(Name = "Temperature")] double Temperature,
    [property: Display(Name = "Metadata")] IReadOnlyDictionary<string, string>? Metadata);

/// <summary>
/// Represents a streamed response fragment or a final completion item.
/// </summary>
public sealed record LlmResponseChunk(
    [property: Display(Name = "Text")] string Text,
    [property: Display(Name = "Is Final")] bool IsFinal,
    [property: Display(Name = "Request Id")] string? RequestId,
    [property: Display(Name = "Provider")] string Provider,
    [property: Display(Name = "Model")] string Model);

/// <summary>
/// Provides a unified streaming contract for hosted and local backends.
/// </summary>
public interface ILlmProviderAdapter
{
    /// <summary>
    /// Streams a provider response for the specified request.
    /// </summary>
    /// <param name="request">The provider-neutral request payload.</param>
    /// <param name="cancellationToken">A token that cancels the in-flight request.</param>
    /// <returns>A sequence of streamed response chunks.</returns>
    IAsyncEnumerable<LlmResponseChunk> StreamAsync(
        LlmRequest request,
        CancellationToken cancellationToken = default);
}

/// <summary>
/// Renders provider-ready prompts from canonical prompt inputs.
/// </summary>
public interface IPromptRenderer
{
    /// <summary>
    /// Renders the final provider-facing prompt text.
    /// </summary>
    /// <param name="request">The request to render.</param>
    /// <returns>The rendered prompt.</returns>
    string Render(LlmRequest request);
}

/// <summary>
/// Applies resource-budget and policy decisions before a request is sent.
/// </summary>
public interface IRequestPolicy
{
    /// <summary>
    /// Throws if the request violates policy or exceeds current runtime budget.
    /// </summary>
    /// <param name="request">The request being evaluated.</param>
    /// <param name="cancellationToken">A token that cancels evaluation.</param>
    ValueTask EnforceAsync(LlmRequest request, CancellationToken cancellationToken = default);
}

/// <summary>
/// Coordinates rendering, policy, routing, and streaming for the runtime.
/// </summary>
public sealed class LlmRuntimePipeline
{
    private static readonly ActivitySource ActivitySource = new("CustomLlmRuntime.Pipeline");
    private readonly IReadOnlyDictionary<string, ILlmProviderAdapter> _providers;
    private readonly IPromptRenderer _renderer;
    private readonly IRequestPolicy _policy;

    /// <summary>
    /// Initializes a new pipeline instance.
    /// </summary>
    /// <param name="providers">The available provider adapters keyed by provider name.</param>
    /// <param name="renderer">The prompt renderer.</param>
    /// <param name="policy">The request policy.</param>
    public LlmRuntimePipeline(
        IReadOnlyDictionary<string, ILlmProviderAdapter> providers,
        IPromptRenderer renderer,
        IRequestPolicy policy)
    {
        _providers = providers;
        _renderer = renderer;
        _policy = policy;
    }

    /// <summary>
    /// Executes a request through policy, rendering, and provider dispatch.
    /// </summary>
    /// <param name="request">The request to execute.</param>
    /// <param name="cancellationToken">A token that cancels execution.</param>
    /// <returns>A streamed sequence of output chunks.</returns>
    /// <exception cref="InvalidOperationException">Thrown when the configured provider is unavailable.</exception>
    public async IAsyncEnumerable<LlmResponseChunk> ExecuteAsync(
        LlmRequest request,
        [EnumeratorCancellation] CancellationToken cancellationToken = default)
    {
        using var activity = ActivitySource.StartActivity("llm.request");
        activity?.SetTag("llm.provider", request.Provider);
        activity?.SetTag("llm.model", request.Model);
        activity?.SetTag("llm.streaming", request.EnableStreaming);
        activity?.SetTag("llm.max_output_tokens", request.MaxOutputTokens);

        await _policy.EnforceAsync(request, cancellationToken).ConfigureAwait(false);

        if (!_providers.TryGetValue(request.Provider, out var provider))
        {
            throw new InvalidOperationException($"Unknown provider '{request.Provider}'.");
        }

        // Render once so all downstream adapters receive a consistent canonical prompt.
        var rendered = _renderer.Render(request);
        var materialized = request with { UserPrompt = rendered };

        await foreach (var chunk in provider.StreamAsync(materialized, cancellationToken).ConfigureAwait(false))
        {
            yield return chunk;
        }
    }
}

This pattern keeps the boundary stable while allowing OpenAI, Anthropic, Llama Stack, llama.cpp, ONNX Runtime, or a custom GGUF executor to plug in behind the same contract. It also gives you a single place to attach ActivitySource spans and apply an R(t)-style resource policy before any transport work begins. OpenTelemetry’s .NET guidance recommends creating your ActivitySource once and using it as the stable tracer source for meaningful units of work.

SSE streaming handler pattern

using System.ComponentModel.DataAnnotations;
using System.Diagnostics;
using System.Net.Http.Headers;
using System.Runtime.CompilerServices;
using System.Text;
using System.Text.Json;

namespace CustomLlmRuntime.OpenAi;

/// <summary>
/// Configuration for the OpenAI Responses adapter.
/// </summary>
public sealed class OpenAiOptions
{
    [Display(Name = "Base Address")]
    public required Uri BaseAddress { get; init; }

    [Display(Name = "Api Key")]
    public required string ApiKey { get; init; }

    [Display(Name = "Project")]
    public string? Project { get; init; }

    [Display(Name = "Organization")]
    public string? Organization { get; init; }
}

/// <summary>
/// Streams Responses API events over HTTP using server-sent events.
/// </summary>
public sealed class OpenAiResponsesAdapter : ILlmProviderAdapter
{
    private static readonly JsonSerializerOptions JsonOptions = new(JsonSerializerDefaults.Web);
    private readonly HttpClient _httpClient;
    private readonly OpenAiOptions _options;

    /// <summary>
    /// Initializes a new adapter.
    /// </summary>
    /// <param name="httpClient">The shared HTTP client instance.</param>
    /// <param name="options">The adapter configuration.</param>
    public OpenAiResponsesAdapter(HttpClient httpClient, OpenAiOptions options)
    {
        _httpClient = httpClient;
        _options = options;
        _httpClient.BaseAddress = options.BaseAddress;
        _httpClient.DefaultRequestHeaders.Authorization =
            new AuthenticationHeaderValue("Bearer", options.ApiKey);

        if (!string.IsNullOrWhiteSpace(options.Project))
        {
            _httpClient.DefaultRequestHeaders.Remove("OpenAI-Project");
            _httpClient.DefaultRequestHeaders.Add("OpenAI-Project", options.Project);
        }

        if (!string.IsNullOrWhiteSpace(options.Organization))
        {
            _httpClient.DefaultRequestHeaders.Remove("OpenAI-Organization");
            _httpClient.DefaultRequestHeaders.Add("OpenAI-Organization", options.Organization);
        }
    }

    /// <summary>
    /// Streams response chunks from the OpenAI Responses API.
    /// </summary>
    /// <param name="request">The canonical runtime request.</param>
    /// <param name="cancellationToken">A token that cancels the HTTP request and stream processing.</param>
    /// <returns>An asynchronous stream of textual chunks.</returns>
    public async IAsyncEnumerable<LlmResponseChunk> StreamAsync(
        LlmRequest request,
        [EnumeratorCancellation] CancellationToken cancellationToken = default)
    {
        var clientRequestId = Guid.NewGuid().ToString("D");

        var payload = new
        {
            model = request.Model,
            input = request.UserPrompt,
            instructions = request.SystemPrompt,
            max_output_tokens = request.MaxOutputTokens,
            temperature = request.Temperature,
            stream = true,
            metadata = request.Metadata
        };

        using var message = new HttpRequestMessage(HttpMethod.Post, "v1/responses")
        {
            Content = JsonContent.Create(payload)
        };

        message.Headers.Add("X-Client-Request-Id", clientRequestId);

        using var response = await _httpClient.SendAsync(
            message,
            HttpCompletionOption.ResponseHeadersRead,
            cancellationToken).ConfigureAwait(false);

        response.EnsureSuccessStatusCode();

        var requestId = response.Headers.TryGetValues("x-request-id", out var values)
            ? values.FirstOrDefault()
            : null;

        await using var stream = await response.Content.ReadAsStreamAsync(cancellationToken).ConfigureAwait(false);
        using var reader = new StreamReader(stream, Encoding.UTF8, leaveOpen: false);

        string? eventName = null;
        while (!reader.EndOfStream)
        {
            var line = await reader.ReadLineAsync(cancellationToken).ConfigureAwait(false);
            if (line is null)
            {
                break;
            }

            if (line.Length == 0)
            {
                eventName = null;
                continue;
            }

            if (line.StartsWith("event:", StringComparison.Ordinal))
            {
                eventName = line["event:".Length..].Trim();
                continue;
            }

            if (!line.StartsWith("data:", StringComparison.Ordinal))
            {
                continue;
            }

            var json = line["data:".Length..].Trim();
            if (json == "[DONE]")
            {
                yield return new LlmResponseChunk(
                    Text: string.Empty,
                    IsFinal: true,
                    RequestId: requestId,
                    Provider: request.Provider,
                    Model: request.Model);

                yield break;
            }

            using var document = JsonDocument.Parse(json);
            if (TryReadDeltaText(document.RootElement, eventName, out var text) &&
                !string.IsNullOrEmpty(text))
            {
                yield return new LlmResponseChunk(
                    Text: text,
                    IsFinal: false,
                    RequestId: requestId,
                    Provider: request.Provider,
                    Model: request.Model);
            }
        }
    }

    /// <summary>
    /// Attempts to normalize OpenAI streaming events into a text fragment.
    /// </summary>
    /// <param name="root">The parsed JSON event payload.</param>
    /// <param name="eventName">The SSE event name.</param>
    /// <param name="text">The extracted text fragment.</param>
    /// <returns><c>true</c> if a text fragment was found; otherwise <c>false</c>.</returns>
    private static bool TryReadDeltaText(JsonElement root, string? eventName, out string text)
    {
        text = string.Empty;

        // Supports common textual-delta event shapes. Extend this switch as the provider schema evolves.
        if (root.TryGetProperty("delta", out var delta) &&
            delta.ValueKind == JsonValueKind.String)
        {
            text = delta.GetString() ?? string.Empty;
            return text.Length > 0;
        }

        if (root.TryGetProperty("output_text", out var outputText) &&
            outputText.ValueKind == JsonValueKind.String)
        {
            text = outputText.GetString() ?? string.Empty;
            return text.Length > 0;
        }

        if (root.TryGetProperty("text", out var textElement) &&
            textElement.ValueKind == JsonValueKind.String &&
            eventName is not "response.completed")
        {
            text = textElement.GetString() ?? string.Empty;
            return text.Length > 0;
        }

        return false;
    }
}

OpenAI streams Responses data over SSE when stream: true is set, and the API documents request IDs and client-supplied request IDs for troubleshooting. The .NET guidance is to reuse HttpClient instances and set connection lifetimes rather than creating new clients per request. That combination is the right baseline for provider adapters.

Tokenizer and chat-template integration pattern

using System.ComponentModel.DataAnnotations;
using System.Collections.ObjectModel;

namespace CustomLlmRuntime.Tokenization;

/// <summary>
/// Minimal metadata extracted from GGUF or equivalent model manifests.
/// </summary>
public sealed class TokenizerSpec
{
    [Display(Name = "Model Type")]
    public required string ModelType { get; init; }

    [Display(Name = "Bos Token")]
    public int? BosToken { get; init; }

    [Display(Name = "Eos Token")]
    public int? EosToken { get; init; }

    [Display(Name = "Chat Template")]
    public string? ChatTemplate { get; init; }

    [Display(Name = "Hugging Face Tokenizer Json")]
    public string? HuggingFaceTokenizerJson { get; init; }

    [Display(Name = "Vocabulary")]
    public IReadOnlyList<string> Vocabulary { get; init; } = Array.Empty<string>();
}

/// <summary>
/// Represents a tokenizer implementation.
/// </summary>
public interface ITokenizer
{
    /// <summary>
    /// Encodes input text into token IDs.
    /// </summary>
    /// <param name="text">The input text to encode.</param>
    /// <returns>The encoded token IDs.</returns>
    IReadOnlyList<int> Encode(string text);

    /// <summary>
    /// Decodes token IDs into text.
    /// </summary>
    /// <param name="tokenIds">The token IDs to decode.</param>
    /// <returns>The decoded text.</returns>
    string Decode(IReadOnlyList<int> tokenIds);
}

/// <summary>
/// Resolves the correct tokenizer implementation from model metadata.
/// </summary>
public sealed class TokenizerRegistry
{
    /// <summary>
    /// Creates a tokenizer from extracted metadata.
    /// </summary>
    /// <param name="spec">The tokenizer specification.</param>
    /// <returns>A tokenizer implementation appropriate for the model.</returns>
    public ITokenizer Create(TokenizerSpec spec)
    {
        if (!string.IsNullOrWhiteSpace(spec.HuggingFaceTokenizerJson))
        {
            return new HuggingFaceTokenizer(spec.HuggingFaceTokenizerJson);
        }

        return spec.ModelType switch
        {
            "llama" or "gpt2" or "replit" or "rwkv" => new GgmlVocabularyTokenizer(spec),
            _ => throw new NotSupportedException(
                $"Tokenizer model '{spec.ModelType}' is not supported by this runtime.")
        };
    }

    /// <summary>
    /// Renders a provider-neutral conversation into model-specific prompt text.
    /// </summary>
    /// <param name="spec">The tokenizer specification containing any embedded chat template.</param>
    /// <param name="systemPrompt">The system prompt content.</param>
    /// <param name="userPrompt">The user prompt content.</param>
    /// <returns>The rendered prompt text.</returns>
    public string RenderChatPrompt(TokenizerSpec spec, string systemPrompt, string userPrompt)
    {
        if (!string.IsNullOrWhiteSpace(spec.ChatTemplate))
        {
            // Production implementation should compile the Jinja-compatible template once,
            // sandbox rendering, and validate the resulting token boundaries per model family.
            return spec.ChatTemplate!
                .Replace("{{ system }}", systemPrompt, StringComparison.Ordinal)
                .Replace("{{ user }}", userPrompt, StringComparison.Ordinal);
        }

        return $"System: {systemPrompt}\nUser: {userPrompt}\nAssistant:";
    }
}

/// <summary>
/// Placeholder Hugging Face tokenizer integration.
/// </summary>
public sealed class HuggingFaceTokenizer : ITokenizer
{
    private readonly string _tokenizerJson;

    public HuggingFaceTokenizer(string tokenizerJson) => _tokenizerJson = tokenizerJson;

    public IReadOnlyList<int> Encode(string text) =>
        throw new NotImplementedException("Bridge to a HF-compatible tokenizer library or service.");

    public string Decode(IReadOnlyList<int> tokenIds) =>
        throw new NotImplementedException("Bridge to a HF-compatible tokenizer library or service.");
}

/// <summary>
/// Placeholder GGML vocabulary tokenizer.
/// </summary>
public sealed class GgmlVocabularyTokenizer : ITokenizer
{
    private readonly TokenizerSpec _spec;

    public GgmlVocabularyTokenizer(TokenizerSpec spec) => _spec = spec;

    public IReadOnlyList<int> Encode(string text) =>
        throw new NotImplementedException("Implement model-family-specific tokenization rules.");

    public string Decode(IReadOnlyList<int> tokenIds) =>
        string.Concat(tokenIds.Select(id => id >= 0 && id < _spec.Vocabulary.Count ? _spec.Vocabulary[id] : string.Empty));
}

GGUF explicitly documents tokenizer metadata such as tokenizer.ggml.model, token lists, BOS/EOS IDs, optional tokenizer.huggingface.json, and tokenizer.chat_template. That means your tokenizer system should not be hard-coded to one model family; it should dispatch from metadata and preserve model-specific prompt rendering. This is also the right place to enforce Teleodynamic-style “bounded interpretation” behavior for model families whose tokenization is approximate or incomplete.

Cache pattern for prefix reuse and provider prompt caching

using System.ComponentModel.DataAnnotations;
using System.Security.Cryptography;
using System.Text;
using Microsoft.Extensions.Caching.Memory;

namespace CustomLlmRuntime.Caching;

/// <summary>
/// Represents a reusable prefix-cache entry.
/// </summary>
public sealed class PrefixCacheEntry
{
    [Display(Name = "Key")]
    public required string Key { get; init; }

    [Display(Name = "Rendered Prefix")]
    public required string RenderedPrefix { get; init; }

    [Display(Name = "Created Utc")]
    public required DateTimeOffset CreatedUtc { get; init; }

    [Display(Name = "Last Accessed Utc")]
    public required DateTimeOffset LastAccessedUtc { get; set; }

    [Display(Name = "Provider Cache Key")]
    public string? ProviderCacheKey { get; init; }
}

/// <summary>
/// Provides local reusable prefix caching for prompts.
/// </summary>
public sealed class PrefixCache : IDisposable
{
    private readonly MemoryCache _cache = new(new MemoryCacheOptions
    {
        SizeLimit = 50_000
    });

    /// <summary>
    /// Gets a stable cache key from immutable prompt parts.
    /// </summary>
    /// <param name="provider">The target provider or backend.</param>
    /// <param name="model">The target model.</param>
    /// <param name="systemPrompt">The canonical system prompt.</param>
    /// <param name="toolSchemaFingerprint">The tool/schema fingerprint.</param>
    /// <returns>A stable cache key.</returns>
    public static string BuildKey(
        string provider,
        string model,
        string systemPrompt,
        string toolSchemaFingerprint)
    {
        var material = $"{provider}\n{model}\n{systemPrompt}\n{toolSchemaFingerprint}";
        var hash = SHA256.HashData(Encoding.UTF8.GetBytes(material));
        return Convert.ToHexString(hash);
    }

    /// <summary>
    /// Tries to get a prefix-cache entry.
    /// </summary>
    /// <param name="key">The stable cache key.</param>
    /// <param name="entry">The cache entry if found.</param>
    /// <returns><c>true</c> if an entry exists; otherwise <c>false</c>.</returns>
    public bool TryGet(string key, out PrefixCacheEntry? entry)
    {
        if (_cache.TryGetValue(key, out PrefixCacheEntry? result))
        {
            result!.LastAccessedUtc = DateTimeOffset.UtcNow;
            entry = result;
            return true;
        }

        entry = null;
        return false;
    }

    /// <summary>
    /// Stores a prefix-cache entry with bounded retention.
    /// </summary>
    /// <param name="entry">The entry to store.</param>
    /// <param name="ttl">The cache retention window.</param>
    public void Set(PrefixCacheEntry entry, TimeSpan ttl)
    {
        _cache.Set(entry.Key, entry, new MemoryCacheEntryOptions
        {
            AbsoluteExpirationRelativeToNow = ttl,
            Size = Math.Max(1, entry.RenderedPrefix.Length / 128)
        });
    }

    public void Dispose() => _cache.Dispose();
}

This local prefix cache should be paired with provider-side prompt caching when available. OpenAI documents prompt_cache_key and optional retention up to 24h; Anthropic documents prompt caching in its context-management section and also exposes Message Batches for asynchronous cost reduction. The architectural point is to have one internal cache identity that can drive both local cache hits and provider cache keys, while your budget governor decides whether cloud caching, local caching, or no caching is the correct choice for a given route.

Internal transport patterns for gRPC and WebSockets

For internal model-worker communication, a thin gRPC contract works well when you need typed incremental deltas, strict deadlines, and bidirectional control, but you should reuse channels/stubs and only introduce channel pools when stream concurrency actually saturates connection limits. For Realtime-style user experiences, ClientWebSocket is appropriate and .NET supports both HTTP/1.1 and HTTP/2 websockets, including an overload that lets you reuse pooled connections through an HttpMessageInvoker.

Performance, cost, and observability

At the kernel level, the attached research points to the same performance frontier that modern local inference engines pursue: memory-mapped weights, quantization-aware kernels, SIMD acceleration, fused QKV and feed-forward dispatch, KV-cache paging, row-interleaved repacking, speculative decoding, and NUMA-aware or topology-aware threading. GGUF itself is designed for fast loading, extensibility, and mmap compatibility, and llama.cpp documents broad hardware support, quantization options, speculative decoding, concurrent requests, and benchmark tooling. In other words, if you truly want a custom local kernel, the performance problem is mostly a memory-bandwidth, scheduling, and cache-locality problem, not just a “write the matrix multiply in C#” problem.

A good practical split is this. Let hosted providers handle the hardest frontier-model serving cases early on. Let ONNX Runtime cover in-process local execution where graph export and execution providers are adequate. Reserve the custom GGUF path for the models and workloads where you need one or more of the following: exact tokenizer/chat-template control, nonstandard quantization support, deeper evidence-bearing traceability, local-only privacy posture, hard control of KV layout, or the ability to experiment with scheduling/structured decoding that upstream servers do not expose. That staged split reflects both the attached research and ONNX Runtime’s role as a cross-platform accelerator rather than a full runtime-governance framework.

The chart below is a planning model, not an empirical benchmark. It visualizes the typical tradeoff you should expect when you apply continuous batching or micro-batching to decode and prefill: small batches improve throughput amortization, but after a moderate point, queueing, cache pressure, worker saturation, and stream limits push p95 latency up sharply. That shape matches the attached research’s emphasis on fused dispatch and paged caching, plus gRPC’s warnings about channel/stream saturation under long-lived streaming loads.

xychart-beta
    title "Illustrative p95 latency per request vs micro-batch size"
    x-axis "Batch Size" [1, 2, 4, 8, 16, 32]
    y-axis "Estimated p95 Latency ms" 0 --> 450
    line [90, 102, 125, 170, 265, 410]

Under that model, the right operational strategy is usually adaptive batching, not “maximize batch size.” For interactive chat, keep batches relatively small and bias for deadline compliance. For queueable summarization, extraction, or embedding work, batch more aggressively. For Anthropic batch workloads, use Message Batches where latency tolerance exists because the API explicitly positions that route as asynchronous and cites a 50% cost reduction. For OpenAI, exploit prompt caching and, where appropriate, background execution so long-running jobs do not occupy interactive capacity. For local backends, use prefix caches and speculative decoding before reaching for larger batches that will degrade tail latency.

A concise optimization matrix is below.

Observability should be first-class, not an afterthought. .NET’s guidance is to instrument work with ActivitySource for traces and Meter/Counter/other instruments for metrics, using IMeterFactory when the runtime is constructed through DI. On top of that foundation, you should carry provider request IDs in every trace: OpenAI documents x-request-id and Anthropic documents request-id. The runtime should tag every span and metric with provider, model, route, tool, cache_hit, batch_size, prefill_tokens, decode_tokens, blocked_action, and safety_outcome. Teleodynamic’s blocked-action idea is especially valuable operationally: “no-op due to insufficient budget” should emit a first-class metric, not disappear as an internal branch.

A good baseline metric set is:

Security, testing, and deployment

Security in this runtime is a combination of provider auth, local-host hardening, sandbox boundaries, and disciplined claims about what the runtime actually guarantees. OpenAI accepts bearer credentials from API keys or short-lived access tokens via workload identity federation and explicitly warns not to expose API keys client-side. Anthropic requires either x-api-key or bearer authorization plus an anthropic-version header, and likewise supports workload identity federation. Those practices point to a clear runtime rule: secrets live server-side, preferably in project-scoped service accounts or workload-identity flows, and provider adapters receive them through configuration or a vault-backed abstraction rather than directly from callers.

Teleodynamic adds two useful security disciplines. First, read-only public artifacts stay separate from mutation surfaces: public documentation and diagnostics should explain behavior, not directly mutate symbol registries, embeddings, or ontologies. Second, local execution is not automatically safe: local/offline sandboxes can reduce latency, cost volatility, privacy exposure, and external dependency, but they do not remove safety risk, and they should not implicitly grant arbitrary code execution or blind tools. Those principles map directly onto plugin and tool-host design in your runtime.

A concise security checklist is below.

Testing should be differential, replayable, and stress-oriented. For hosted providers, maintain golden contract tests for non-streaming request/response payloads and event-replay tests for streaming protocols. Anthropic’s SSE event taxonomy is explicit enough that you can build deterministic parsers and regression fixtures around message_start, content-block deltas, message_delta, message_stop, ping, and error events. OpenAI streaming and Realtime should receive the same treatment, but their flexibility makes your adapter normalization layer more important. For local backends, differential tests should compare your executor against a reference server—typically llama.cpp or a known-good ONNX route—on tokenization, BOS/EOS handling, stop conditions, grammar enforcement, and logit masking behavior.

A high-value testing matrix is:

Deployment should differ by backend. For hosted-provider routing, containers or serverless functions work well because the runtime needs only outbound HTTPS and a warm in-memory cache. For local model workers, containers are the default operational boundary: llama.cpp documents Docker usage, and Llama Stack explicitly publishes a Docker Hub route. For edge and on-device scenarios, ONNX Runtime is the most natural fit because its documentation explicitly covers C#, mobile, IoT, edge, and multiple execution providers. A fully custom GGUF worker is usually best deployed as a long-lived container or dedicated service process because cold starts and model mapping costs are significant.

Extensibility should be governed the same way. Introduce new model families, adapters, or plugins behind internal contracts such as ILlmProviderAdapter, IToolPlugin, ITokenizer, IGrammarConstraint, and IStateStore. Then apply Teleodynamic operator logic to lifecycle decisions: add a plugin when value repays maintenance, merge overlapping routes, retire low-utility components, and choose no-op when evidence is weak. That is a better long-term control regime than endlessly adding bespoke branches to the request pipeline.

Migration roadmap and references

The right migration plan is incremental. Start by stabilizing the application contract and the evidence/trace model. Then add hosted adapters and local-server adapters. Then introduce ONNX in-process execution. Only after you have robust differential tests, tokenizer fidelity, and cache accounting should you commit to the custom GGUF backend. That sequence mirrors the attached research’s staged rewrite logic and Teleodynamic’s public implementation-roadmap style: directional planning, bounded claims, and visible review gates.

gantt
    title Recommended implementation roadmap
    dateFormat  YYYY-MM-DD
    axisFormat  %b %d

    section Foundations
    Runtime contract and DTOs           :active, a1, 2026-06-22, 14d
    Trace ledger and OpenTelemetry      :a2, after a1, 14d
    Budget governor and policy gates    :a3, after a1, 21d

    section Hosted adapters
    OpenAI Responses adapter            :b1, 2026-07-06, 21d
    Anthropic Messages adapter          :b2, 2026-07-13, 21d
    Streaming replay fixtures           :b3, after b1, 14d

    section Local serving
    Llama Stack or llama.cpp adapter    :c1, 2026-07-27, 21d
    Prefix cache and schema cache       :c2, after c1, 14d
    Differential tests vs reference     :c3, after c1, 21d

    section In-process inference
    ONNX Runtime backend                :d1, 2026-08-17, 28d
    Execution-provider tuning           :d2, after d1, 14d
    Edge packaging path                 :d3, after d1, 14d

    section Custom local kernel
    GGUF loader and metadata parser     :e1, 2026-09-14, 28d
    Tokenizer and chat-template engine  :e2, after e1, 21d
    KV cache and paging layer           :e3, after e1, 28d
    SIMD kernels and fused dispatch     :e4, after e3, 35d

    section Hardening
    Security review and fuzzing         :f1, 2026-10-26, 21d
    Performance regression harness      :f2, after e4, 21d
    Controlled production rollout       :f3, after f1, 21d

A realistic decision rule for phase promotion is simple. Do not advance a backend just because it is “more custom.” Advance it when it wins on a measurable combination of cost, privacy posture, latency, failure rate, maintenance burden, and audit quality. In Teleodynamic terms, the structural change only deserves promotion if local gain repays its maintenance cost without violating the viability floor. That rule will keep the runtime from overcommitting to a bespoke kernel too early.

References

The attached research corpus used as design input: managed-first runtime rewrite strategy, local execution concerns, GGUF/kernel engineering, and staged migration planning.

Teleodynamic source pages most relevant to this architecture: start page, architecture, operator library, resource economy, work-constraint cycle, local sandboxes, claim boundary ledger, developer integration guide, implementation roadmap.

OpenAI official documentation used here: Responses API reference, Realtime guide, API overview/authentication, prompt caching, streaming, and request-ID guidance.

Anthropic official documentation used here: API overview, Messages, Streaming Messages, authentication headers, request IDs, batches, and SDK capabilities.

Meta and local-serving references used here: Llama Stack documentation and llama.cpp repository documentation for OpenAI-compatible local serving, concurrency, speculative decoding, grammar constraints, and Docker usage.

Model-format and local-execution references used here: GGUF specification, tokenizer metadata and chat-template fields, ONNX Runtime overview, execution providers, C# API, model-validation warnings, and performance surfaces.

.NET and transport references used here: HttpClient lifetime guidance, WebSocket support, .NET tracing with ActivitySource, .NET metrics with Meter and IMeterFactory, DI lifetime guidance, and gRPC C# plus performance guidance.