Rewriting llama.cpp in C

Full report

On this page

Executive summary

A C# rewrite of llama.cpp is technically feasible, but the answer depends on what “rewrite” means. If the goal is a managed-first reimplementation of model loading, tokenization, scheduling, CLI/server tooling, basic CPU inference, and backend abstraction, C# is a strong fit because .NET gives you Span<T>, Memory<T>, SIMD, hardware intrinsics, memory-mapped files, source-generated P/Invoke, channels, and increasingly capable AOT/JIT tooling. If the goal is full parity with current llama.cpp performance and hardware breadth on every backend and all quantized kernels, a pure C# rewrite is high risk and likely slower to reach parity than a hybrid design. The current llama.cpp codebase is no longer “just a C++ LLaMA inference toy”; it is a broad inference platform centered on ggml, GGUF, a large model-architecture matrix, many quantization families, a substantial CLI/server/tool ecosystem, and a wide backend surface that includes CPU, BLAS, Metal, SYCL, CUDA, HIP, Vulkan, OpenCL, OpenVINO, WebGPU, and others.

The most defensible strategy is therefore hybrid and staged. Build the top half in C#—GGUF parser, tokenizer layer, scheduler, sampling, KV-cache management, server/CLI, benchmarking, tests, packaging, and interop abstractions—while keeping three execution modes: a managed CPU baseline, optional native-kernel backends via P/Invoke, and optional out-of-process accelerators for deployments that prefer operational isolation. That architecture gives you a credible path to a fully open, idiomatic .NET codebase without tying success to immediate reimplementation of every CUDA/HIP/Metal/Vulkan kernel or every new quantization variant that lands in ggml.

The engineering conclusion is blunt: do not begin by porting kernels line-for-line. Begin by reproducing the contract surface of llama.cpp: GGUF compatibility, tokenizer behavior, graph/model abstraction, CLI/server semantics, benchmarking/test harnesses, and backend pluggability. Then optimize the hot kernels that measurement proves matter on your target hardware. The current llama.cpp project itself reflects that architecture: GGUF conversion, architecture registration, graph building, multimodal add-ons, examples/tools, and backend-specific implementation layers are explicitly separated in its add-model and build documentation.

Scope and feature inventory

llama.cpp describes its primary goal as LLM inference with minimal setup and state-of-the-art performance across a wide range of hardware, locally and in the cloud. The repository is organized as a real platform, with top-level directories for ggml, src, conversion, tests, tools, examples, gguf-py, common, docs, benches, and packaging/build metadata. That is the correct scope assumption for any rewrite: you are not rewriting one inference loop, you are rewriting a layered platform.

At the model-format and artifact level, the project is centered on GGUF, which the ggml documentation defines as a binary format for inference models designed for fast loading/saving, full self-description, extensibility, and mmap compatibility. GGUF naming and metadata support not only base tensor models, but also LoRA sidecars, vocab-only files, multimodal projector sidecars (mmproj), and multi-token prediction sidecars (mtp), plus sharding. In practice, this means a C# rewrite must treat “model load” as a metadata-driven artifact system rather than a flat weight blob loader.

At the tokenizer level, GGUF documents multiple embedded tokenizer modes and metadata conventions. The spec explicitly names llama and replit SentencePiece-derived vocabularies, gpt2-style BPE, rwkv, and optionally an embedded Hugging Face tokenizer.json. It also includes special-token identifiers and a tokenizer.chat_template key for Jinja-style prompt formatting. In other words, your C# tokenizer subsystem must not be “a LLaMA tokenizer”; it must be a dispatcher over model metadata with support for at least SentencePiece-like vocab/scoring, GPT-2/BPE merges, special-token semantics, and chat-template formatting.

At the quantization and tensor-encoding level, llama.cpp’s documented surface is large and still evolving. The README summarizes the project as supporting 1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization. The tensor-encoding wiki and current CPU backend trait tables show a broader family including floating-point and integer tensor types (F64, F32, F16, BF16, I8, I16, I32, I64) and multiple quantized families such as Q4_0, Q4_1, Q5_0, Q5_1, Q8_0, Q8_1, Q2_K, Q3_K, Q4_K, Q5_K, Q6_K, i-quant variants like IQ3_XXS, IQ4_NL, IQ4_XS, plus newer trait-table entries such as Q1_0, MXFP4, NVFP4, TQ1_0, and TQ2_0. The CPU trait tables are especially revealing because they map each type to quantization/dequantization and vec_dot implementations, which is exactly where a C# rewrite will spend its performance budget.

At the inference-loop level, the project already distinguishes between prompt processing, text generation, and combined prompt-generation workflows. llama-bench supports all three (pp, tg, pg), carries context-depth prefilling for the KV cache, and varies batch size, threads, layer offload, backend, cache types, and device selection. This matters for C# design because different performance bottlenecks show up in prefill versus decode, and because the benchmark tool explicitly notes that its measurements exclude tokenization and sampling—a useful reminder that tokenizer and sampler performance need their own benchmark harness in your rewrite.

At the memory-management and file-I/O level, GGUF was explicitly designed for mmap, and llama-bench exposes runtime controls for --mmap, --direct-io, K/V cache types, host/device placement, tensor overrides, and GPU layer offload. A rewrite that eagerly copies weights into managed arrays will give away one of the format’s principal advantages. Model loading in C# should therefore be built around MemoryMappedFile, pointer-backed spans, and lazy tensor materialization where possible.

At the multithreading and scheduling level, the CLI/server surface already exposes thread counts, separate thread counts for prompt/batch work, CPU masks/ranges, thread priority, and polling levels. The server adds parallel decoding, multi-user support, and continuous batching. This is a strong signal that the managed rewrite needs both a low-level kernel scheduler and a higher-level request scheduler; one abstraction is not enough. Compute kernels want tight control and affinity; server workflows want bounded queues, admission control, and backpressure.

At the backend level, the build guide is unambiguous: llama.cpp can be built with CPU, BLAS, Metal, SYCL, CUDA, MUSA, HIP, Vulkan, CANN, ZenDNN, Arm KleidiAI, OpenCL, Android, OpenVINO, and WebGPU support, and can dynamically load backends at runtime if built accordingly. This breadth is exactly why a greenfield “pure C# everywhere” objective is strategically fragile. A C# rewrite should instead define a stable backend contract and accept that some implementations are managed and some are native.

At the CLI, tools, server, benchmarks, and tests level, the repo includes a wide operational surface: the root repo contains tools, tests, examples, benches, conversion scripts, and server/security/contributing metadata. The server README alone documents OpenAI-compatible completions, chat completions, responses, embeddings, Anthropic-compatible messages, reranking, multimodal requests, tool calling, schema-constrained JSON, speculative decoding, monitoring endpoints, the Web UI, and LoRA adapter management. The repository test tree spans allocator tests, argument parsing, backend ops, grammar/GBNF, GGUF loading, chat/template/Jinja parsing, reasoning budget, RoPE, sampling, save/load state, thread safety, tokenizers, quantization, multimodal C API, and server tests written in pytest. That is the real target inventory. A serious rewrite must plan for feature parity in subsystems, not in a single predict() method.

The following table captures the rewrite surface at a subsystem level.

C# implementation strategy and architecture

The best C# design is a modular vertical slice: Core for model/session abstractions; GGuf for metadata and tensor maps; Tokenizers for vocab/merge/template handling; Kernels.Cpu for a managed baseline; Backends.Native for CUDA/HIP/Metal/Vulkan/OpenCL/OpenVINO bindings or shims; Sampling for logits processors and grammar constraints; Server and Cli for host tools; and Tests/Benchmarks for parity and performance. That mirrors llama.cpp’s own separation between conversion, architecture registration, graph building, backend support, multimodal add-ons, tools, and tests.

A useful architectural rule is: metadata and orchestration in safe managed code, kernels in constrained unsafe code, optional vendor acceleration behind narrow interfaces. .NET’s memory model supports this cleanly. Span<T> and ReadOnlySpan<T> are appropriate for synchronous hot-path slices because they can wrap managed, unmanaged, or stack memory, but they are stack-only and cannot cross await or be stored as ordinary heap fields. Memory<T> and ReadOnlyMemory<T> exist precisely for heap-stored or async-crossing ownership. That split maps almost perfectly onto llama.cpp’s distinction between tightly scoped compute buffers and longer-lived model/session/request state.

For compute, System.Numerics.Vector<T> is the right portable SIMD baseline, while System.Runtime.Intrinsics.X86 and .Arm are the correct escape hatches for architecture-specific kernels. Microsoft’s docs make the trade-off explicit: Vector<T> is hardware-accelerated and CPU-register dependent, while the intrinsics namespaces expose instruction-set-specific APIs for x86 and Arm. A rewrite should therefore implement each hot kernel in three tiers: scalar reference, portable Vector<T>, and intrinsic specializations for the few kernels that dominate profiles.

For interop, prefer source-generated P/Invoke with LibraryImport over ad hoc DllImport for new code. Microsoft’s interop documentation specifically calls out LibraryImport source generation, reduced runtime stub generation, and better alignment with NativeAOT scenarios; the runtime also provides NativeLibrary.Load and SetDllImportResolver for controlled native loading and RID-specific dispatch. That makes it practical to ship a managed core with selectable native acceleration packages.

For scheduling, reserve the managed thread pool for short-lived coordination work and request pipelines; the .NET docs position ThreadPool and tasks as the general-purpose substrate for short background work, and TaskCreationOptions.LongRunning / dedicated threads for coarse-grained long-lived operations. That maps well to inference: request handling, batching, and streaming can live on tasks/channels, while a pinned decode worker or backend submission thread can be dedicated where measurement justifies it. System.Threading.Channels is particularly attractive for server batching because it is explicitly designed for asynchronous producer-consumer pipelines.

The table below maps common C++/llama.cpp implementation idioms to recommended C# equivalents.

A concrete architecture can look like this:

flowchart TD
    CLI[CLI]
    Server[HTTP Server]
    Bench[Benchmarks]
    Tests[Parity & Regression Tests]

    CLI --> Runtime
    Server --> Runtime
    Bench --> Runtime
    Tests --> Runtime

    Runtime[Inference Runtime]
    Runtime --> Sessions[Session & KV Cache Manager]
    Runtime --> Sampling[Sampling & Grammar]
    Runtime --> Tokenizers[Tokenizer Layer]
    Runtime --> Loader[GGUF Loader]
    Runtime --> BackendAbstraction[Backend Abstraction]

    Loader --> Metadata[GGUF Metadata Parser]
    Loader --> Tensors[Tensor Catalog / Views]
    Tokenizers --> Embedded[Embedded GGUF Tokenizers]
    Tokenizers --> HF[HF Tokenizer Adapter Optional]

    BackendAbstraction --> CpuManaged[Managed CPU Kernels]
    BackendAbstraction --> NativeBackends[Native Backends]
    BackendAbstraction --> RemoteBackends[Remote / Out-of-Proc Backends]

    NativeBackends --> CUDA[CUDA / cuBLAS]
    NativeBackends --> HIP[HIP / ROCm]
    NativeBackends --> DML[DirectML / Windows ML]
    NativeBackends --> ORT[ONNX Runtime / TensorRT / EPs]

The llama.cpp add-model guide gives a useful architectural clue for your own domain model: it treats “supporting a model” as four separate steps—convert to GGUF, define architecture metadata, build the execution graph, and optionally add multimodal encoders. That suggests a C# design with explicit interfaces such as IModelArtifactLoader, IModelArchitecture, IGraphBuilder, IBackend, ITokenizer, and IMultimodalEncoder, rather than a single inheritance tree full of special cases.

Suggested code patterns

The snippet below shows the right shape for memory-mapped GGUF access in C#: open the file once, create a read-only view, acquire a pointer, and expose span slices over the mapped region. That preserves GGUF’s intended mmap-friendly loading model.

using System;
using System.IO;
using System.IO.MemoryMappedFiles;
using Microsoft.Win32.SafeHandles;

public static class GgufMap
{
    /// <summary>
    /// Maps a GGUF file into memory and invokes a callback with a read-only byte span.
    /// </summary>
    /// <param name="path">Absolute or relative path to the GGUF file.</param>
    /// <param name="reader">Callback that consumes the mapped bytes synchronously.</param>
    public static unsafe void WithMappedFile(string path, Action<ReadOnlySpan<byte>> reader)
    {
        if (path is null) throw new ArgumentNullException(nameof(path));
        if (reader is null) throw new ArgumentNullException(nameof(reader));

        var fileInfo = new FileInfo(path);
        if (!fileInfo.Exists) throw new FileNotFoundException("GGUF file not found.", path);
        if (fileInfo.Length == 0) throw new InvalidDataException("GGUF file is empty.");

        using var mmf = MemoryMappedFile.CreateFromFile(
            path,
            FileMode.Open,
            mapName: null,
            capacity: 0,
            MemoryMappedFileAccess.Read);

        using var accessor = mmf.CreateViewAccessor(
            offset: 0,
            size: fileInfo.Length,
            access: MemoryMappedFileAccess.Read);

        byte* ptr = null;
        accessor.SafeMemoryMappedViewHandle.AcquirePointer(ref ptr);
        try
        {
            // AcquirePointer returns a pointer to the start of the view.
            var span = new ReadOnlySpan<byte>(ptr, checked((int)fileInfo.Length));
            reader(span);
        }
        finally
        {
            accessor.SafeMemoryMappedViewHandle.ReleasePointer();
        }
    }
}

For a portable first-pass dot product / matmul inner primitive, Vector<float> is the right baseline. The code below is intentionally simple; in a production rewrite you would add alignment-aware intrinsic variants and quantized kernels separately.

using System;
using System.Numerics;

public static class SimdKernels
{
    /// <summary>
    /// Computes the dot product of two equal-length float vectors.
    /// </summary>
    /// <param name="left">Left-hand vector.</param>
    /// <param name="right">Right-hand vector.</param>
    /// <returns>The dot product.</returns>
    public static float Dot(ReadOnlySpan<float> left, ReadOnlySpan<float> right)
    {
        if (left.Length != right.Length)
            throw new ArgumentException("Input spans must have the same length.");

        int width = Vector<float>.Count;
        int i = 0;
        Vector<float> acc = Vector<float>.Zero;

        for (; i <= left.Length - width; i += width)
        {
            var vx = new Vector<float>(left.Slice(i, width));
            var vy = new Vector<float>(right.Slice(i, width));
            acc += vx * vy;
        }

        float sum = Vector.Sum(acc);

        for (; i < left.Length; i++)
            sum += left[i] * right[i];

        return sum;
    }
}

For tokenizer integration, a useful pattern is to dispatch from GGUF metadata to a tokenizer implementation. The metadata keys are standardized enough to drive that decision directly.

using System;
using System.Collections.Generic;

public interface ITokenizer
{
    /// <summary>
    /// Encodes input text into token IDs.
    /// </summary>
    /// <param name="text">Input text.</param>
    /// <param name="addBos">Whether to prepend BOS.</param>
    /// <param name="addEos">Whether to append EOS.</param>
    /// <returns>Encoded token IDs.</returns>
    int[] Encode(string text, bool addBos = false, bool addEos = false);
}

public static class TokenizerFactory
{
    /// <summary>
    /// Creates a tokenizer from GGUF metadata.
    /// </summary>
    /// <param name="meta">GGUF metadata key-value pairs.</param>
    /// <returns>A tokenizer implementation.</returns>
    public static ITokenizer Create(IReadOnlyDictionary<string, object> meta)
    {
        if (meta.TryGetValue("tokenizer.huggingface.json", out var hfJson))
        {
            return new HuggingFaceTokenizer((string)hfJson);
        }

        string model = (string)meta["tokenizer.ggml.model"];
        return model switch
        {
            "llama" or "replit" => new SentencePieceStyleTokenizer(meta),
            "gpt2"              => new BpeTokenizer(meta),
            "rwkv"              => new RwkvTokenizer(meta),
            _                   => throw new NotSupportedException($"Unsupported tokenizer model '{model}'.")
        };
    }
}

For native interop, prefer source-generated bindings and stable C ABI entry points. The following signature style is appropriate whether you wrap libllama directly or a thin C shim that you own.

using System;
using System.Runtime.InteropServices;

internal static partial class LibLlama
{
    private const string LibraryName = "llama";

    /// <summary>
    /// Initializes backend-wide state.
    /// </summary>
    [LibraryImport(LibraryName, EntryPoint = "llama_backend_init")]
    [UnmanagedCallConv(CallConvs = [typeof(CallConvCdecl)])]
    internal static partial void BackendInit();

    /// <summary>
    /// Frees backend-wide state.
    /// </summary>
    [LibraryImport(LibraryName, EntryPoint = "llama_backend_free")]
    [UnmanagedCallConv(CallConvs = [typeof(CallConvCdecl)])]
    internal static partial void BackendFree();

    /// <summary>
    /// Loads a model from file.
    /// </summary>
    /// <param name="path">UTF-8 or platform-default encoded file path, depending on the exported ABI.</param>
    /// <param name="parameters">Native model parameters struct by reference.</param>
    /// <returns>Opaque model handle or null on failure.</returns>
    [LibraryImport(LibraryName, EntryPoint = "llama_model_load_from_file", StringMarshalling = StringMarshalling.Utf8)]
    [UnmanagedCallConv(CallConvs = [typeof(CallConvCdecl)])]
    internal static partial IntPtr ModelLoadFromFile(string path, ref LlamaModelParams parameters);
}

[StructLayout(LayoutKind.Sequential)]
internal struct LlamaModelParams
{
    public int NGpuLayers;
    public int MainGpu;
    public int UseMmap;
    public int UseMlock;
}

Performance, acceleration, and interop

The most performance-critical insight from current llama.cpp sources is that type-specific vector dot products and quantization/dequantization are first-class dispatch points in the CPU backend. The ggml-cpu.c trait tables map data types to from_float, vec_dot, and vec_dot_type implementations, which is strong evidence that your hottest CPU kernels will be: quantized block unpacking, dot products / GEMV-style operations, and the operations fused around attention and projection paths. That is exactly where C# needs careful use of unsafe, SIMD, and architecture-specific intrinsics.

Your optimization priorities should therefore be ordered like this. First, eliminate allocations on hot paths using spans, pools, and stack-local scratch buffers. ArrayPool<T> exists precisely to reduce GC pressure from frequently created/destroyed arrays, and Microsoft’s interop guidance also calls out pooled buffers as a good fit for native boundary work. Second, use Vector<T> for portable baseline vectorization. Third, add explicit intrinsics for the few kernels that dominate p95 runtime on each architecture. Fourth, benchmark JIT settings, not assumptions: Tiered PGO is configurable in .NET, and ReadyToRun and NativeAOT solve startup/deployment problems differently.

On JIT and AOT, the trade-offs are straightforward. Dynamic PGO and tiered compilation are runtime optimization tools suited to long-lived managed processes. ReadyToRun is aimed at reducing startup work by precompiling assemblies, but Microsoft also notes the size trade-off. NativeAOT produces a self-contained native binary and removes JIT from the deployment equation, which is attractive for CLI tools and small utilities, but it also introduces AOT-compatibility constraints and requires more discipline around reflection and dynamic code. For an inference library, the best path is usually: JIT + TieredPGO for the core engine and benchmark process; ReadyToRun or NativeAOT for utility executables if startup or single-file deployment matters enough.

On GC tuning, use policy sparingly and structure aggressively. Microsoft’s GC docs describe server GC as throughput/scalability oriented, and document low-latency/no-GC options for critical regions, but the safest primary strategy for inference remains “allocate less.” In practice, that means large long-lived weight buffers outside the managed heap when possible, pooled temporary arrays, stable per-session KV-cache buffers, and minimal object churn during streaming decode. Server GC makes sense for a daemon-like HTTP service; interactive local tools should benchmark both defaults and tuned modes before locking policy in.

The backend decision is partly technical and partly product-strategic. The table below summarizes the main options.

The interop choice is equally important, because it shapes maintainability as much as raw speed.

One .NET 9-specific temptation is Tensor<T>. Microsoft’s docs describe it as experimental, built on TensorPrimitives, and optimized for AI-library interop. That makes it promising for future higher-level tensor plumbing, but it should not be the foundation of a first production rewrite of llama.cpp; the API is explicitly experimental and your core abstraction needs long stability.

Implementation roadmap, testing, and benchmarks

The llama.cpp maintainers’ own add-model guide is the best roadmap seed: conversion to GGUF, architecture definition, graph construction, optional multimodal implementation, and validation across key backends/tools. A C# rewrite should mirror that sequence, but with an additional early emphasis on differential testing against llama.cpp.

The table below proposes a practical phased plan.

A realistic timeline for one strong team is closer to months than weeks, and the sequence matters more than the exact dates.

gantt
    title Suggested delivery timeline
    dateFormat  YYYY-MM-DD
    axisFormat  %b

    section Foundation
    Project skeleton and CI               :a1, 2026-06-20, 14d
    GGUF parser and memory mapping        :a2, after a1, 28d
    Differential test harness             :a3, after a1, 21d

    section Core runtime
    Tokenizer subsystem                   :b1, after a2, 35d
    Session runtime and KV cache          :b2, after a2, 35d
    Sampling and grammar                  :b3, after b2, 21d

    section Compute
    Managed CPU baseline kernels          :c1, after b2, 42d
    Benchmarking and hotspot analysis     :c2, after c1, 21d
    First native backend                  :c3, after c2, 42d

    section Product surface
    CLI parity                            :d1, after b3, 21d
    HTTP server parity                    :d2, after d1, 28d
    Packaging and docs                    :d3, after d2, 14d

    section Extended scope
    LoRA and advanced quant coverage      :e1, after c3, 35d
    Multimodal support                    :e2, after e1, 35d

For testing, your strongest asset is that llama.cpp already exposes what it considers important enough to test. The tests/ tree includes GGUF parsing, tokenizer-specific tests, grammar and GBNF validators, chat and template parsers, Jinja, quantization statistics and performance, RoPE, sampling, state save/restore, thread safety, backend ops, and multimodal C API tests; server tests live separately under tools/server/tests as Python pytest scenarios targeting GitHub workflow runners. The rewrite should not invent an entirely new test taxonomy—port the categories.

For benchmarking, adopt the same logical split as llama-bench: prefill (pp), decode (tg), and combined (pg). Preserve knobs for context depth, batch size, thread count, GPU-layer offload, and device selection so your results remain comparable to upstream expectations. Because llama-bench excludes tokenization and sampling, add separate microbenchmarks for tokenizer throughput, KV-cache append/update, logits processors, grammar validation, and each quantized dot-product kernel. On the .NET side, BenchmarkDotNet is the standard choice and is explicitly designed for reliable, repeatable measurement with dedicated benchmark processes.

A representative benchmark matrix should include at least these scenarios:

Packaging, licensing, security, and contributor guidance

llama.cpp is published under the MIT license, and the repository explicitly exposes CONTRIBUTING.md and SECURITY.md alongside the license. A clean-room or source-attributed C# rewrite can therefore also be MIT-licensed, but you should preserve notices and be disciplined about provenance if any code is transliterated rather than independently reimplemented.

For distribution, split deliverables into a managed core package and RID-specific optional accelerator packages. .NET packaging and publishing make this natural: dotnet pack produces NuGet packages, and the RID catalog exists specifically for platform-specific assets. This maps well to a package layout such as YourProject.Core, YourProject.Cpu, YourProject.Cuda, YourProject.DirectML, YourProject.Cli, and YourProject.Server. For deployable tools, use dotnet publish with PublishReadyToRun or NativeAOT where startup, self-contained deployment, or operational simplicity matter enough to justify the trade-offs.

For contributor guidance and reproducibility, require three things from day one. First, pin the SDK with global.json; Microsoft documents this as the mechanism for selecting the .NET SDK used by CLI commands. Second, enable package lock files for repeatable restore; NuGet’s lock-file behavior is explicitly designed to make restores deterministic when dependencies have not changed. Third, enable deterministic compilation in release builds; the C# compiler docs describe deterministic compilation as useful for verifying whether binaries are built from trusted source. Those three measures give you much better build reproducibility than many greenfield rewrites ever achieve.

Security needs to be treated as part of the design, not a checklist after the port. GGUF parsing should validate magic, version, metadata types, key structure, tensor counts, dimensions, alignment, and offsets before exposing mapped views; the GGUF spec is explicit about magic bytes, typed metadata, hierarchical keys, and aligned tensor offsets. Because the project will inevitably use unsafe code and native interop in hot paths, follow Microsoft’s native-interop and unsafe-code best practices: keep unsafe regions small, prefer generated interop stubs, use pooled buffers rather than per-call array allocations, and wrap native lifetime in safe abstractions.

Reproducibility of numerical behavior also deserves special handling. Official .NET tensor docs note that some numeric operations may use underlying C runtime calls or architecture-specific instructions, and exact results can differ across operating systems or architectures. That means your parity protocol should distinguish between byte-for-byte deterministic outputs where required and tolerance-based comparisons where backend math legitimately differs. In practice: exact tokenizer parity, exact GGUF metadata parity, golden logit comparisons within tolerance, and end-to-end text-output comparisons only under pinned seeds, prompts, templates, and backend configurations.

Open questions and limitations

This report is based on the upstream llama.cpp and related primary documentation as visible in the June 2026 repository snapshot and official platform docs. The largest moving target is quantization and backend coverage: ggml continues to add new tensor encodings and backends, and some backend ecosystems are themselves in motion—for example, DirectML is in sustained engineering while Windows ML is the newer Windows strategy, and ONNX Runtime’s ROCm EP has already shifted toward MIGraphX. Any serious rewrite plan should therefore treat backend/quantization expansion as an ongoing compatibility program rather than a one-time porting exercise.

The other important limitation is strategic rather than factual: a report can identify the right architecture, but actual viability depends on target hardware priorities. If your primary target is CPU-only local inference and .NET ergonomics, a managed-first rewrite is attractive. If your primary target is parity with the fastest CUDA paths, native acceleration becomes central much earlier. Because the original request left the target platform unspecified, this report intentionally recommends a cross-platform modular design rather than a platform-maximalist one.