Building a Fully Custom C LLM Runtime Informed by Teleodynamic Theory

Full report

On this page

Executive Summary

A fully custom C# LLM runtime is technically feasible, but the strongest architecture is not a monolith that tries to do everything in managed code from day one. The best design is a layered runtime: a provider-agnostic orchestration core in C#, a memory-safe/high-throughput buffer and batching substrate, and a pluggable execution layer that can target both hosted APIs and local models. The local path should support native backends such as llama.cpp/GGUF first, then optionally evolve toward deeper managed reimplementation of kernels or graph execution where profiling proves it is worth the risk. This recommendation follows current provider capabilities, .NET runtime realities, and Teleodynamic’s repeated emphasis on bounded action, explicit review, and no-op dominance rather than speculative overreach.

Teleodynamic.com is especially useful as a governance-and-control vocabulary for the runtime, not as an empirical proof that a live system is already teleodynamic. The site explicitly presents itself as a static theoretical and architectural lens, not a deployed agent platform or live telemetry system. Its most valuable contributions to a C# runtime are: resource-bounded decision gates, claim-boundary enforcement, operator libraries with an explicit no-op path, review gates, evidence packets, public-safe metrics, and lane separation between orchestration, memory, telemetry, review, and external tool use. Those concepts map cleanly to runtime modules such as a budget governor, memory ledger, tool router, evaluator, audit store, and policy/claim boundary engine.

For runtime targets, the current .NET support picture strongly favors .NET 10 as the preferred baseline and .NET 8 only as a compatibility target if required. Microsoft’s current support policy shows .NET 10 as active LTS through November 2028, while .NET 8 is in maintenance and ends support in November 2026. For most greenfield work in mid-2026, that argues for net10.0 as the main target, optionally multi-targeting net8.0 only if ecosystem dependencies force it.

For hosted-model integration, the runtime should expose a common abstraction over streaming, conversation state, tool use, structured outputs, embeddings, and prompt caching. OpenAI’s Responses API supports stateful interactions through prior responses, tool invocation, structured output, and SSE streaming; Anthropic exposes SSE streaming with content-block deltas and explicit tool-use blocks; Gemini exposes streaming endpoints and embedding APIs, including multimodal embeddings. That means a single C# orchestration surface can normalize a surprisingly large set of provider features without collapsing to the least common denominator.

For local execution, a pragmatic first step is to target GGUF + native executor interop rather than immediately rewriting all inference kernels in C#. GGUF is optimized for quick model loading/save and carries standardized metadata; llama.cpp remains a prominent C/C++ inference stack; SentencePiece and BPE-family tokenizers remain central tokenization patterns. A custom runtime can still provide differentiated value in C# by owning the orchestration loop, batching, memory management, tool plane, telemetry, safety policy, and retrieval pipeline even when matrix kernels remain native at first.

The attached research corpus appears, from the uploaded file set, to converge on a similar direction: a custom C# LLM runtime, “bare-metal” or “directly against the metal” execution, possible llama.cpp rewriting, enterprise architecture planning, and performance-oriented pipeline control. However, the body text of those attachments was not queryable in the tool context available here, so any fine-grained claims from them are unspecified in this report. Directionally, they reinforce the report’s main conclusion: own the runtime architecture in C#, but introduce native-kernel replacement only after measurement justifies the cost. Unspecified details from the attachments are marked as such.

Source Basis and Design Posture

Teleodynamic’s own claim boundaries matter for how its ideas should be used. The site says Teleodynamic AI is a theoretical and architectural lens for constraint-maintaining intelligence; it explicitly says the site does not run agents, train models, certify products, prove AGI, or claim consciousness. Its claim-boundary ledger further warns against widening these ideas into claims of deployed autonomous intelligence or live runtime closure without evidence. In practice, that means Teleodynamic should guide runtime structure, auditability, policy discipline, and control loops, not be presented as proof that your runtime has reached some stronger scientific status.

The strongest translated design principle from Teleodynamic is resource closure under bounded budgets. Its resource-economy page frames viable action as a budgeted decision in which compute, memory, review, and uncertainty all consume internal budget, and expensive structural actions should be blocked when affordability fails. A custom C# runtime can operationalize this as real quota checks on context growth, retrieval fan-out, tool calls, batch expansion, speculative decoding, and fine-tuning jobs. Put simply: every expensive action in the runtime should pass an affordability gate.

The second critical principle is no-op dominance. Teleodynamic’s operator library treats split, merge, add, retire, and no-op as explicit structural operators, with no-op framed as an active positive control signal when no affordable edit improves local viability. For LLM runtimes, that principle is extremely valuable: when the retriever has weak evidence, when a tool schema mismatches, when context trimming would harm fidelity, or when a fine-tune dataset is too noisy, “do nothing and log why” is often the most robust engineering decision.

The third principle is review-gated evidence. Teleodynamic’s evaluation lab emphasizes metric families beyond accuracy alone, including stability, traceability, review pressure, and operational viability, and its onboarding/roadmap pages repeatedly frame changes as review-gated and bounded. That suggests a runtime that treats prompt changes, tool schema migrations, model upgrades, and memory compaction policies as auditable artifacts with replayable evidence rather than informal code edits.

Goals, Use Cases, and High-Level Architecture

A serious custom C# runtime should explicitly support the following use cases: low-latency interactive inference, batch offline inference, structured tool use, stateful conversations, persistent memory, retrieval-augmented generation, embeddings, evaluation workflows, and optional fine-tuning orchestration. Hosted providers already expose stateful conversations, tools, structured outputs, embeddings, and streaming, while current research and serving literature show why retrieval, tool use, and memory-aware serving are no longer “extras” but core parts of modern LLM systems.

For inference, the runtime should serve both request/response and incremental token streaming. OpenAI streams via server-sent events when stream: true; Anthropic streams SSE events with structured event types and block deltas; Gemini exposes streamGenerateContent endpoints over SSE. A provider-independent C# abstraction can therefore normalize streaming as IAsyncEnumerable<TokenEvent> or Channel<TokenEvent> regardless of provider.

For fine-tuning, the runtime should distinguish between hosted optimization workflows and local parameter-efficient adaptation. Hosted providers continue to expose model optimization/fine-tuning surfaces, but the strongest local strategy for a custom runtime is usually LoRA or QLoRA rather than full-model retraining. LoRA freezes pretrained weights and injects low-rank trainable matrices, dramatically reducing trainable parameters without adding inference latency; QLoRA pushes this further by fine-tuning 4-bit quantized models and showed 65B fine-tuning on a single 48 GB GPU in the original paper.

For prompt engineering, the runtime should treat prompts as versioned executable assets rather than string literals. OpenAI’s current APIs expose prompt templates, conversation state, structured outputs, and prompt cache controls; Anthropic and Gemini also make prompt structure a first-class concern through tools, context management, and streaming semantics. That favors a runtime with prompt registries, prompt metadata, evaluation hooks, prompt cache keys, and rollback support.

For tool use and stateful agents, the runtime should not hardwire a “single autonomous agent” abstraction into its core. ReAct and Toolformer are useful here: both show that the value comes from interleaving reasoning with actions and deciding when to invoke external systems. Teleodynamic’s L0–L6 capability framing also supports this separation: a runtime can support everything from deterministic workflows to adaptive, domain-bounded agents without falsely claiming fully general autonomous agency.

flowchart LR
    Client[Clients<br>ASP.NET | Desktop | CLI | Serverless] --> Orchestrator[Runtime Orchestrator]
    Orchestrator --> Session[Session and State Manager]
    Orchestrator --> Policy[Policy and Claim Boundary Engine]
    Orchestrator --> Budget[Resource Budget Governor]
    Orchestrator --> Router[Model Router]

    Router --> ApiExec[Hosted Provider Executors]
    Router --> LocalExec[Local Model Executors]

    ApiExec --> OpenAI[OpenAI]
    ApiExec --> Anthropic[Anthropic]
    ApiExec --> Gemini[Gemini]
    ApiExec --> Azure[Azure OpenAI]

    LocalExec --> GGUF[GGUF Loader]
    LocalExec --> NativeInterop[Native Backend Interop]
    LocalExec --> Tokenizer[Tokenizer Layer]

    Orchestrator --> Tools[Tool Plane]
    Orchestrator --> Memory[Memory and Retrieval Plane]
    Orchestrator --> Eval[Evaluation and Benchmarking]
    Orchestrator --> Obs[Observability]

    Memory --> Embed[Embedding Service]
    Memory --> Vector[Vector Index]
    Memory --> Store[Document and Session Store]

    Obs --> Metrics[Metrics]
    Obs --> Traces[Tracing]
    Obs --> Logs[Structured Logs]

The corresponding process model should separate responsibilities across at least three loops: a fast token loop for streaming/inference, a mid-speed orchestration loop for state, retrieval, and tools, and a slow structural loop for evaluation, compaction, prompt revision, and model selection. That slow-loop idea aligns directly with Teleodynamic’s operator library and resource-economy framing: expensive structural edits belong outside the token loop.

Teleodynamic-to-runtime component mapping

Architecture Decisions for a Production-Grade C# Runtime

At the threading and async layer, the runtime should standardize on TPL + async/await for orchestration and Channels for producer/consumer token flow. Microsoft’s TPL guidance frames Task as the core abstraction for asynchronous operations and recommends Task.Run where additional scheduling control is not needed; Channels implement an asynchronous producer/consumer FIFO queue with built-in synchronization. That combination is a natural fit for streamed tokens, background embeddings, retrieval prefetch, and batched tool execution.

At the buffer layer, Span<T>, ReadOnlySpan<T>, Memory<T>, ReadOnlyMemory<T>, and IMemoryOwner<T> should be the default vocabulary. Microsoft’s guidance is clear: prefer Span<T> for synchronous APIs when possible, keep strict ownership/lease rules, and dispose or transfer IMemoryOwner<T> exactly once. In practice, that means tokenizer hot paths, JSON framing, SSE parsing, and embedding vector manipulation should minimize copies and make leases explicit.

At the interop layer, native integration should use source-generated P/Invoke where possible. Microsoft’s LibraryImportAttribute support generates marshalling code at compile time, removes the runtime IL stub, and can allow inlining, which is particularly attractive for frequent small interop boundaries such as tokenizer functions, quantized matmul wrappers, and KV-cache control calls. For a custom runtime, that is meaningfully better than hand-waving about “unsafe code” without a disciplined interop strategy.

At the serialization layer, the default choice should be System.Text.Json with source generation unless profiling shows a particular bottleneck that meaningfully justifies alternatives. Microsoft’s source-generation-backed serializer supports both metadata and serialization-optimized modes through JsonSerializerContext, which is relevant for high-volume provider traffic, structured outputs, persisted state, and telemetry envelopes.

At the HTTP client layer, use IHttpClientFactory with typed clients for hosted providers. Microsoft documents several benefits that matter directly here: central configuration of logical clients, outgoing middleware via delegating handlers, underlying handler lifetime management to avoid common DNS/lifetime problems, and integrated logging. That is the right default for OpenAI, Anthropic, Gemini, Azure OpenAI, and internal tool services.

At the hosting layer, design the core runtime as a .NET Generic Host library first, then expose host-specific shells. ASP.NET Core is the obvious server host; desktop shells can still host ASP.NET Core and gRPC APIs with Microsoft.AspNetCore.App even in non-web projects; Azure Functions isolated worker gives process control and .NET-version independence; AWS Lambda supports managed .NET runtimes, custom runtimes, and container images; Cloud Run offers a .NET path plus autoscaling and concurrency controls; Kubernetes Deployments give declarative rollouts and rollback semantics.

flowchart TD
    Host[Generic Host] --> Web[ASP.NET Core Host]
    Host --> Desktop[Desktop Host]
    Host --> Worker[Background Worker]
    Host --> Serverless[Serverless Host]

    Web --> Rest[REST and SSE]
    Web --> Grpc[gRPC]
    Desktop --> LocalApi[Embedded HTTP/gRPC]
    Worker --> Queue[Queue / Batch Jobs]
    Serverless --> Func[Azure Functions / Lambda]

    Rest --> Core[Runtime Core]
    Grpc --> Core
    LocalApi --> Core
    Queue --> Core
    Func --> Core

    Core --> Providers[Hosted Providers]
    Core --> Local[Native Local Models]
    Core --> Retrieval[Embeddings and RAG]
    Core --> Tools[Tools and Memory]

For gRPC specifically, ASP.NET Core’s gRPC stack is strong when you need low-overhead internal service boundaries or a separate token/event plane. Microsoft documents that gRPC in ASP.NET Core requires HTTP/2, should be secured with TLS, and can coexist with MVC/controllers in the same routing pipeline; the same framework reference can also be used from WPF, WinForms, or Windows Services that need to host an ASP.NET Core server internally.

For GC and memory policy, the runtime should optimize for steady-state low allocation, not magical low-latency modes as a primary strategy. Microsoft’s guidance says low-latency GC modes suppress certain collections but should be used only for short or contained time-sensitive windows and with minimized allocations, especially LOH and pinned allocations. That makes low-latency modes a tactical tool around streaming bursts or deadline-sensitive sections, not a blanket process setting.

LLM Integration, Retrieval, and Teleodynamic Control Loops

The runtime should expose a model capability descriptor rather than hard-coded provider branches. At minimum, capabilities should include: text generation, JSON/structured output, streaming, embeddings, tool calls, prompt caching, conversation state, background execution, batch support, image input, and local-model compatibility. OpenAI’s Responses API alone already spans streaming, conversation continuity, function calling, built-in tools, stateful interactions, and prompt caching keys; Anthropic and Gemini fill complementary ground in tools, embeddings, and streaming semantics.

For API versus local model routing, the right C# abstraction is a IModelExecutor interface with capability flags and a cost/latency profile. Hosted execution wins on frontier-model quality, multimodal breadth, and rapidly changing platform features. Local execution wins on data residency, deterministic infrastructure control, offline operation, and predictable marginal cost. GGUF’s optimized loading format and executor metadata make it a sensible local artifact format, while llama.cpp remains a practical initial local backend.

For streaming, unify provider deltas into a small event algebra:

• text token delta
• reasoning/thinking delta if surfaced
• tool call started
• tool argument delta
• tool result
• usage update
• completion / error

This event model aligns well with OpenAI SSE streaming, Anthropic’s detailed event flow and incremental tool_use.input JSON deltas, and Gemini streaming endpoints. It also makes backpressure easy to express in Channels.

For batching, you should distinguish between provider-side batch APIs and runtime-side dynamic microbatching. Even when a provider exposes a batch endpoint, local runtime throughput still depends on request coalescing, KV-cache efficiency, and minimizing fragmentation. PagedAttention is a major design reference here: the paper shows how inefficient KV-cache management limits batch size and how paging-style memory management can improve throughput by 2–4× at similar latency. That is a strong argument for making KV cache and batch assembly first-class runtime components even in a C#-centric architecture.

For tokenization, do not pretend one tokenizer abstraction fits all models. You need a pluggable tokenizer layer supporting at least BPE-family tokenizers and SentencePiece-style models. SentencePiece remains important because it supports direct training from raw text, language-independent tokenization, and subword units such as BPE and unigram models; tiktoken illustrates the importance of a fast BPE tokenizer family for OpenAI-class models.

For context-window management, prefer a budgeted policy over naive truncation. At minimum, the runtime should maintain:

• a hard token budget,
• a reserved output token floor,
• tool/schema overhead,
• retrieval budget,
• memory budget,
• prompt cache segment boundaries,
• a compaction policy.

OpenAI’s current APIs explicitly surface conversation continuity and prompt caching controls, which means context strategy is not just “sum tokens until failure” anymore. Teleodynamic’s resource-closure idea suggests promoting context budgeting to a policy engine with declared costs and an explicit no-op/abort when the remaining budget is insufficient.

For embeddings and RAG, the runtime should treat retrieval as its own subsystem, not a helper function hidden behind a prompt builder. OpenAI’s embedding API returns float vectors and token-usage metadata; Gemini’s current embedding API supports multimodal embeddings in a shared embedding space and over 100 languages. RAG remains a foundational architecture because it combines parametric memory with explicit non-parametric memory and improves provenance, factuality, and updatability on knowledge-intensive tasks.

For tool use, the runtime should support both client-side and server-side tools. Anthropic’s current docs are especially clear here: client tools execute in your application after tool_use blocks, while server tools execute on Anthropic infrastructure. That distinction is helpful for a custom runtime generally: some tools should remain in your controlled .NET process, while others can be provider-managed or remote. ReAct and Toolformer further support making tool invocation a first-class part of reasoning rather than a secondary bolt-on.

Recommended module partitioning

Performance, Security, Deployment, Observability, and Developer Ergonomics

Performance work should begin with explicit targets: p50/p95 first-token latency, tokens/sec, requests/sec, batch efficiency, allocation rate, GC pause time, KV-cache footprint, retrieval latency, and tool-call tail latency. Teleodynamic’s evaluation framing is useful here because it insists on viability and review pressure in addition to correctness; performance should therefore be reported with operational metrics and auditability, not just benchmark bragging.

For actual benchmarking, use BenchmarkDotNet for microbenchmarks and .NET diagnostic tools for runtime introspection. BenchmarkDotNet emphasizes reliable, reproducible measurement and guards against common benchmark mistakes; dotnet-counters is suitable for ad hoc health monitoring and first-level investigation; dotnet-trace collects traces for deeper analysis; Visual Studio’s profiler exposes CPU usage, object allocation, memory usage, async analysis, and more. That toolchain is enough to measure almost every hot path in a custom C# runtime.

For kernel-level performance strategy, two research anchors matter. FlashAttention shows why attention performance is often constrained by memory I/O and not just arithmetic, while PagedAttention shows why serving throughput is often constrained by KV-cache fragmentation and inefficient dynamic batching. A C# runtime should therefore avoid premature hand-optimizing business logic while ignoring the two places where real long-context cost often accumulates: attention I/O and KV memory management.

For security and privacy, treat the runtime as a data-handling system first and an ML system second. In development, Microsoft explicitly warns not to store secrets in source or config files and notes that Secret Manager is not an encrypted trusted store; production secrets should come through controlled stores such as Azure Key Vault. For protected state, ASP.NET Core’s data protection stack provides cryptographic protection with key management and rotation. Those are the right defaults for API keys, session state, cached tool results, and encrypted memory snapshots.

For observability, standardize on OpenTelemetry-compatible instrumentation. Microsoft’s .NET guidance is especially clean here: ILogger<T> for logging, Meter for metrics, and ActivitySource/Activity for distributed tracing. That should map directly onto runtime spans such as llm.request, retrieval.query, embedding.batch, tool.call, memory.compact, policy.noop, and native.infer.

For deployment, containers are the portable default. Microsoft documents official .NET production and SDK images, regularly updated patch images, and smaller chiseled images with fewer CVEs and non-root defaults; Kubernetes Deployments add declarative rollout, scaling, and rollback; Azure Functions isolated worker offers process control and version flexibility; AWS Lambda supports managed runtimes, custom runtimes, and container images; Cloud Run has .NET deployment paths plus autoscaling, concurrency controls, secrets, service identity, and even GPU-related configuration paths.

For developer ergonomics, the runtime should feel like a modern .NET SDK, not an academic framework. That means:

• typed clients instead of stringly APIs,
• capability discovery,
• source-generated serialization,
• strongly typed tool contracts,
• testable abstractions,
• consistent async streaming,
• safe buffer ownership,
• provider-agnostic models and prompts,
• reference sample apps for ASP.NET, worker, desktop, and serverless hosts.

This design is supported by IHttpClientFactory, Generic Host patterns, System.Text.Json source generation, Channels, and ASP.NET/gRPC hosting surfaces already available in the .NET ecosystem.

Benchmark plan

Sample microbenchmarks

using System;
using System.Buffers;
using System.Text;
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;

/// <summary>
/// Measures tokenizer framing and UTF-8 parsing overhead for streaming paths.
/// </summary>
[MemoryDiagnoser]
public class Utf8TokenFramingBenchmarks
{
    private byte[] _payload = default!;

    /// <summary>
    /// Payload size in bytes.
    /// </summary>
    [Params(256, 1024, 4096)]
    public int PayloadBytes { get; set; }

    /// <summary>
    /// Prepares a representative UTF-8 payload.
    /// </summary>
    [GlobalSetup]
    public void Setup()
    {
        _payload = Encoding.UTF8.GetBytes(new string('a', PayloadBytes));
    }

    /// <summary>
    /// Copies data into a pooled buffer and decodes it.
    /// </summary>
    /// <returns>The decoded string length.</returns>
    [Benchmark]
    public int PooledDecode()
    {
        using IMemoryOwner<byte> owner = MemoryPool<byte>.Shared.Rent(_payload.Length);
        _payload.AsSpan().CopyTo(owner.Memory.Span);

        ReadOnlySpan<byte> span = owner.Memory.Span[.._payload.Length];
        return Encoding.UTF8.GetCharCount(span);
    }

    /// <summary>
    /// Decodes directly from the existing payload.
    /// </summary>
    /// <returns>The decoded string length.</returns>
    [Benchmark(Baseline = true)]
    public int DirectDecode()
    {
        return Encoding.UTF8.GetCharCount(_payload);
    }
}

public static class Program
{
    public static void Main()
    {
        BenchmarkRunner.Run<Utf8TokenFramingBenchmarks>();
    }
}

using System;
using System.Threading.Channels;
using System.Threading.Tasks;
using BenchmarkDotNet.Attributes;

/// <summary>
/// Measures channel throughput for token event transport.
/// </summary>
[MemoryDiagnoser]
public class ChannelStreamingBenchmarks
{
    /// <summary>
    /// Number of token events to stream.
    /// </summary>
    [Params(512, 4096)]
    public int Count { get; set; }

    /// <summary>
    /// Writes and reads token-sized payloads through a bounded channel.
    /// </summary>
    /// <returns>A task that completes when the stream is fully consumed.</returns>
    [Benchmark]
    public async Task BoundedChannelRoundTrip()
    {
        var channel = Channel.CreateBounded<string>(new BoundedChannelOptions(256)
        {
            SingleReader = true,
            SingleWriter = true
        });

        Task producer = Task.Run(async () =>
        {
            for (int i = 0; i < Count; i++)
            {
                await channel.Writer.WriteAsync("tok");
            }

            channel.Writer.Complete();
        });

        await foreach (string _ in channel.Reader.ReadAllAsync())
        {
        }

        await producer;
    }
}

Core implementation patterns

using System;
using System.Collections.Generic;
using System.Net.Http;
using System.Runtime.CompilerServices;
using System.Text.Json;
using System.Threading;
using System.Threading.Tasks;

/// <summary>
/// Represents a streamed token delta from any provider.
/// </summary>
/// <param name="Text">The emitted text delta.</param>
/// <param name="IsFinal">Whether this is the terminal chunk.</param>
public readonly record struct TokenDelta(string Text, bool IsFinal);

/// <summary>
/// Streams SSE-style deltas from a provider endpoint and normalizes them into token deltas.
/// </summary>
/// <param name="httpClient">The configured HTTP client for the provider.</param>
/// <param name="request">The request message to send.</param>
/// <param name="cancellationToken">The cooperative cancellation token.</param>
/// <returns>An async sequence of normalized token deltas.</returns>
public static async IAsyncEnumerable<TokenDelta> StreamResponseAsync(
    HttpClient httpClient,
    HttpRequestMessage request,
    [EnumeratorCancellation] CancellationToken cancellationToken = default)
{
    using HttpResponseMessage response = await httpClient.SendAsync(
        request,
        HttpCompletionOption.ResponseHeadersRead,
        cancellationToken);

    response.EnsureSuccessStatusCode();

    await using var stream = await response.Content.ReadAsStreamAsync(cancellationToken);
    using var reader = new System.IO.StreamReader(stream);

    while (!reader.EndOfStream)
    {
        string? line = await reader.ReadLineAsync(cancellationToken);
        if (string.IsNullOrWhiteSpace(line) || !line.StartsWith("data:", StringComparison.Ordinal))
        {
            continue;
        }

        string payload = line["data:".Length..].Trim();
        if (payload == "[DONE]")
        {
            yield return new TokenDelta(string.Empty, true);
            yield break;
        }

        using JsonDocument json = JsonDocument.Parse(payload);
        if (json.RootElement.TryGetProperty("delta", out JsonElement delta) &&
            delta.TryGetProperty("text", out JsonElement text))
        {
            yield return new TokenDelta(text.GetString() ?? string.Empty, false);
        }
    }

    yield return new TokenDelta(string.Empty, true);
}

using System;
using System.Buffers;
using System.Collections.Generic;

/// <summary>
/// Token-budget utilities for context assembly.
/// </summary>
public static class ContextBudget
{
    /// <summary>
    /// Trims message segments to stay within a hard token budget while preserving a required tail window.
    /// </summary>
    /// <param name="segments">Ordered message segments from oldest to newest.</param>
    /// <param name="budgetTokens">The hard total token budget.</param>
    /// <param name="reservedOutputTokens">The tokens reserved for model output.</param>
    /// <param name="tokenCounter">A function that estimates token count for a segment.</param>
    /// <returns>The kept segments in original order.</returns>
    public static IReadOnlyList<string> TrimToBudget(
        IReadOnlyList<string> segments,
        int budgetTokens,
        int reservedOutputTokens,
        Func<string, int> tokenCounter)
    {
        int available = Math.Max(0, budgetTokens - reservedOutputTokens);
        var kept = new List<string>(segments.Count);

        int running = 0;
        for (int i = segments.Count - 1; i >= 0; i--)
        {
            int tokens = tokenCounter(segments[i]);
            if (running + tokens > available)
            {
                continue;
            }

            kept.Add(segments[i]);
            running += tokens;
        }

        kept.Reverse();
        return kept;
    }
}

using System;
using System.Buffers;
using System.Runtime.InteropServices;

/// <summary>
/// Demonstrates disciplined pooled memory usage for embeddings or token buffers.
/// </summary>
public static class VectorMath
{
    /// <summary>
    /// Computes a dot product using pooled memory and spans.
    /// </summary>
    /// <param name="left">The left vector.</param>
    /// <param name="right">The right vector.</param>
    /// <returns>The dot product.</returns>
    public static float Dot(ReadOnlySpan<float> left, ReadOnlySpan<float> right)
    {
        if (left.Length != right.Length)
        {
            throw new ArgumentException("Vector lengths must match.");
        }

        float sum = 0f;
        for (int i = 0; i < left.Length; i++)
        {
            sum += left[i] * right[i];
        }

        return sum;
    }

    /// <summary>
    /// Copies an input vector into a pooled buffer to create a leased working set.
    /// </summary>
    /// <param name="input">The source vector.</param>
    /// <returns>A leased memory owner containing the copied vector.</returns>
    public static IMemoryOwner<float> Lease(ReadOnlySpan<float> input)
    {
        IMemoryOwner<float> owner = MemoryPool<float>.Shared.Rent(input.Length);
        input.CopyTo(owner.Memory.Span);
        return owner;
    }
}

Roadmap, Recommendations, and Open Questions

The most credible implementation roadmap is incremental and review-gated.

The architectural recommendation is therefore:

That includes sessions, prompts, tools, memory, retrieval, evaluation, telemetry, policy, and host integration.

1. Own orchestration in C# immediately.

Use LibraryImport, Span<T>, Memory<T>, IMemoryOwner<T>, and explicit lease semantics to minimize overhead and preserve debuggability.

1. Own native interop next.

Those layers often produce larger end-to-end gains than a heroic early rewrite of matmul kernels.

1. Own batching, KV policy, and context budgets before rewriting kernels.

That decision should be benchmark-driven and may end with a hybrid answer: some pieces belong in C#, some do not.

1. Only then decide whether “rewrite llama.cpp in C#” is justified.

Open questions and limitations

The largest limitation in this report is the attached research corpus. The uploaded documents clearly point toward a direct-to-metal C# LLM runtime agenda, but the body text of those files was not queryable in the tool context available here, so their fine-grained technical arguments, benchmarks, and internal proposals remain unspecified in this synthesis.

A second open question is how far you want “fully custom” to go. There are at least three materially different interpretations:

• custom orchestrator over hosted APIs,
• custom orchestrator plus native local backends,
• full managed inference stack including kernels and format loaders.

Those are not the same project. The report’s recommendation is to start with the second, measure ruthlessly, and only pursue the third where data justifies it.

A third open question is target workload shape. The optimal runtime for chat UX, enterprise RAG, offline batch processing, and agentic tool flows is not identical. The more your workload skews toward long-context serving and concurrent sessions, the more KV-cache management, batching, and retrieval economics dominate architecture decisions.