Architectural Blueprint: Engineering a Native C\ Large Language Model Inference Engine

Full report

On this page

The Paradigm Shift in Managed Inference Ecosystems

The proliferation of localized Large Language Models (LLMs) has fundamentally altered the computational requirements of modern software ecosystems. The industry standard for local inference is llama.cpp, an open-source inference engine developed by Georgi Gerganov that achieves remarkable cross-platform performance via pure C/C++ implementations of tensor operations, integer quantization, and hardware-accelerated backends1. Operating under the MIT license, this framework allows enterprises to deploy massive foundation models on consumer-grade hardware with zero reliance on cloud providers, ensuring absolute data privacy and sovereignty1. However, for enterprise ecosystems heavily invested in the .NET framework, integrating a native C/C++ engine introduces substantial architectural friction. Historically, the .NET ecosystem has approached machine learning and deep learning through wrappers and bindings. The SciSharp stack—which includes libraries such as TensorFlow.NET, Keras.NET, and NumSharp—demonstrates the viability of bringing data science tools to C\# developers, yet these tools still predominantly act as Managed wrappers over native C++ implementations5. Similarly, the primary mechanism for utilizing llama.cpp within C\# applications has been via libraries like LLamaSharp8. LLamaSharp relies on Platform Invocation Services (P/Invoke) to interface with pre-compiled native llama.cpp dynamic link libraries (.dll or .so)4. While LLamaSharp provides robust abstractions—such as LLamaContext and InteractiveExecutor—and seamlessly integrates with orchestration frameworks like BotSharp and Microsoft’s Semantic Kernel11, this paradigm retains the constraints of unmanaged code. Developers must manage complex cross-compilation toolchains, navigate versioning conflicts across hardware backends (e.g., varying CUDA versions), and accept the opacity of the memory boundary between the .NET Garbage Collector (GC) and the native heap1. Recent advancements in the .NET runtime have catalyzed a paradigm shift, proving that a pure, native C\# inference engine is not only viable but highly competitive. Projects such as TensorSharp2 and dotLLM illustrate that the entirety of the llama.cpp stack—from weight ingestion to token sampling—can be rewritten in C\# targeting .NET Core and .NET 1013. By leveraging modern C\# features such as Hardware Intrinsics (SIMD), unmanaged memory manipulation via NativeMemory, and Native Ahead-of-Time (AOT) compilation, developers can achieve throughput comparable to llama.cpp while retaining the safety, observability, and deployment simplicity of a fully managed environment14. The following architectural report dissects the engineering requirements for translating the llama.cpp framework into a native C\# architecture, detailing the mechanics of GGUF parsing, vector-accelerated mathematics, direct GPU driver interfacing, and advanced memory orchestration.

Storage and State Orchestration: The GGUF Specification

The foundation of an inference engine is its capacity to ingest massive neural network weights rapidly and without exhausting physical system memory. The GPT-Generated Unified Format (GGUF) was engineered specifically to supersede the legacy GGML format2. While GGML functioned adequately as an initial tensor library, it suffered from rigidity; modifying hyperparameters or introducing new model architectures routinely broke backward compatibility3. GGUF resolves this by introducing an extensible, hierarchical key-value metadata store alongside the binary payload, allowing the format to seamlessly incorporate new information without invalidating legacy parsers3.

Binary Layout and Metadata Extraction

A native C\# parser must strictly adhere to the GGUF binary specification to dynamically reconstruct the neural network's topology at runtime16. The file is structured sequentially, beginning with metadata and terminating with a massive, memory-aligned payload of tensor weights.

In a native C\# implementation, the parsing pipeline reads the metadata key-value pairs into a managed dictionary. The parser extracts the architecture type (e.g., llama, qwen, phi) from the general.architecture key, using it to dynamically construct the remaining keys needed to populate a ModelConfig struct16. For example, the engine dynamically looks up llama.embedding\_length to determine the hidden size, and llama.rope.freq\_base to establish the Rotary Positional Embedding (RoPE) parameters16. After processing the tensor\_count, the parser calculates the exact start of the data payload by rounding the current file stream position up to the specified alignment boundary16.

Demand Paging and Unmanaged Memory Mapping

A critical architectural strategy employed by llama.cpp is the utilization of memory-mapped files (mmap), a technique that delegates memory management directly to the host operating system's page cache1. Rather than eagerly loading a massive 70-gigabyte model into physical RAM, the engine maps the file into the virtual address space. The operating system transparently loads memory pages from the solid-state drive (SSD) into physical RAM only when specific tensors are accessed during the forward pass1. This permits inference on hardware with limited physical memory and enables multiple processes to share the same cached physical pages14. Replicating this behavior in C\# requires circumventing limitations within the standard .NET Base Class Library. The standard MemoryMappedViewAccessor used for memory-mapped files is restricted by a 2-gigabyte limit per view, rooted in legacy 32-bit indexing constraints16. To overcome this and map arbitrarily large foundation models, a native C\# engine utilizes MemoryMappedFile.CreateFromFile in conjunction with SafeMemoryMappedViewHandle.AcquirePointer16. This unsafe operation yields a raw, unmanaged byte\* pointer to the base of the memory-mapped payload. Accessing a specific tensor becomes an [Figure omitted from source export] pointer arithmetic operation: [Figure omitted from source export] This architecture achieves true zero-allocation loading14. Because the .NET GC is entirely unaware of unmanaged pointers, the presence of multi-gigabyte neural network weights generates absolutely no GC pressure or heap fragmentation14.

Threat Mitigation and Memory Safety

Historically, parsing complex binary formats in C/C++ has introduced severe security vulnerabilities. For instance, CVE-2024-25664 details a heap overflow vulnerability within the GGML library, where unchecked metadata n\_kv loop counters and unvalidated length-encoded strings allowed malicious GGUF files to overwrite adjacent heap memory, potentially resulting in arbitrary code execution19. A pure C\# architecture intrinsically neutralizes these vectors. While pointer arithmetic is reserved exclusively for the tensor data payload, the initial parsing of the GGUF header and metadata strings is handled using modern, memory-safe constructs like Span\<T\> and ReadOnlySpan\<T\>19. These structures enforce rigorous, hardware-accelerated bounds checking at the runtime level. Any attempt by an adversarial GGUF file to declare a malicious allocation size or wrap a 64-bit integer will instantly trigger a managed ArgumentOutOfRangeException, halting execution safely before memory corruption can manifest19.

Integer Quantization Topologies and Block Layouts

The computational bottleneck of localized LLM inference is not processing power, but memory bandwidth. During the autoregressive decoding phase (where tokens are generated one by one), the engine must stream the entire weight matrix from RAM to the CPU registers for every single token21. To mitigate this bottleneck, models are aggressively quantized. Quantization divides the massive floating-point matrices into small, contiguous blocks of weights, mapping high-precision 16-bit or 32-bit floats down to 8-bit, 4-bit, or even 2-bit integers, sharing a common scaling factor across the block to retain dynamic range1.

GGML Type Structures

A C\# engine must map these quantized memory blocks perfectly using unmanaged structs. The block configurations are highly optimized for parallel processing.

Modern distributions rarely use a uniform quantization type across the entire network. Mixed-precision topologies, such as the Q4\_K\_M format, map the heavily utilized attention layers to the higher-fidelity Q6\_K format, while the immense Feed-Forward Network (FFN) layers are compressed down to Q4\_K23. A robust inference engine must dynamically dispatch specific matrix-vector (GEMV) multiplication kernels based on the ggml\_type metadata attached to each individual tensor23. Furthermore, extreme sub-4-bit quantization methodologies are emerging to run models on severely constrained hardware. Formats mapping weights to 1, 2, or 3 bits per parameter rely on fractional bit rates rather than discrete bit depths, presenting unique challenges. Engines implementing these extreme compressions require highly specialized kernels capable of evaluating arbitrary bit rates on the fly, balancing the significant drop in memory footprint against the increased computational instruction overhead required to unpack such irregular bitwise data24.

SIMD Intrinsics and Hardware Vectorization

To attain throughput parity with the native llama.cpp C executable, the C\# engine must execute dot products directly against quantized blocks without ever expanding them into intermediate 32-bit floating-point (FP32) tensors25. Expanding a Q4\_0 tensor back to FP32 in RAM before multiplication would instantly saturate the memory bus and cripple performance.

System.Runtime.Intrinsics Acceleration

The .NET runtime exposes low-level CPU instructions via the System.Runtime.Intrinsics namespace. This allows C\# code to directly emit Single Instruction, Multiple Data (SIMD) assembly instructions, specifically AVX2 and AVX-512 on x86 architectures, and NEON on ARM26. SIMD enables a single CPU core to load 256-bit or 512-bit vectors, performing simultaneous mathematical operations across 8, 16, or 32 values within a single clock cycle26. When executing a dot product between a Q8\_0 weight block and a dynamically quantized activation vector, the optimized C\# execution pipeline operates entirely within SIMD registers:

1. Two 256-bit vectors are loaded containing the packed int8 data29.
2. The engine invokes Avx2.MultiplyAddAdjacent (wrapping the \_mm256\_maddubs\_epi16 intrinsic). This executes a fused dot product on adjacent byte pairs, yielding 16-bit integer results (INT8 \-\> INT16) natively in the hardware23.
3. A subsequent Avx2.MultiplyAddAdjacent operation multiplies these 16-bit integers by a vector of 1s, widening the accumulators further into 32-bit integers (INT16 \-\> INT32) without data loss23.
4. Finally, the 32-bit integer accumulators are converted to floating-point representation, multiplied by the combined scale factors extracted from the block headers, and accumulated into the final sum using Fused Multiply-Add (Fma.MultiplyAdd) instructions23.

To utilize these advanced instructions, developers must ensure strict memory alignment. Instantiating arrays via the standard new\[\] keyword in C\# provides no alignment guarantees, causing AVX-512 instructions to fault or suffer massive performance penalties when executing unaligned memory loads14. Therefore, all tensor memory buffers are allocated using NativeMemory.AlignedAlloc(size, 64), ensuring strict 64-byte boundary alignment required for maximum AVX-512 throughput14.

Cache Locality and Row-Interleaved Repacking

Despite rigorous SIMD optimization, the autoregressive decode phase frequently stalls due to CPU cache thrashing. When a CPU core computes a slice of the output matrix, it must read across multiple rows of the weight matrix. In standard row-major memory layouts, traversing down a column involves jumping vast distances in memory, resulting in continuous Translation Lookaside Buffer (TLB) misses and invalidating the L1/L2 cache21. To circumvent this, advanced C\# engines execute "Row-Interleaved Weight Repacking" immediately after loading the GGUF file22. For an AVX2-optimized pipeline, the engine converts the matrices into a Q8\_0\_R4 format21. It extracts four consecutive rows of quantized blocks from the memory-mapped file and repacks them contiguously, column-by-column, into a fresh, unmanaged buffer21. Consequently, when the SIMD lanes execute the matrix multiplication, they read sequentially from contiguous memory rather than striding across disparate rows. This optimization transforms the memory access pattern from an Array of Structures (AoS) to a Structure of Arrays (SoA), virtually eliminating cache misses and significantly boosting decoding tokens-per-second21. The memory overhead and latency of this repacking are incurred only once during startup and are completely amortized over the inference lifespan of the model22.

Direct GPU Interfacing and Driver-Level Execution

While CPUs orchestrate the pipeline, achieving state-of-the-art inference speeds requires offloading computation to the massive parallel throughput of Graphics Processing Units (GPUs). llama.cpp supports an array of hardware backends, primarily NVIDIA CUDA and Apple Metal1. Standard .NET applications usually rely on heavy, pre-compiled unmanaged C++ wrappers (like cudart.dll) to interface with the GPU. A fully native C\# engine bypasses these wrappers entirely by interacting directly with the low-level graphics drivers.

The CUDA Driver API and PTX String Compilation

The NVIDIA compute ecosystem is distinctly split into the High-Level Runtime API and the Low-Level Driver API33. By invoking P/Invoke signatures directly against cuda.dll (on Windows) or libcuda.so (on Linux), the C\# application commands the GPU at the lowest possible software layer33. This architecture avoids static binary compilation. Instead of distributing massive .cubin files compiled specifically for older GPU architectures, the C\# engine embeds Parallel Thread Execution (PTX) instruction strings directly within the application binary33. PTX serves as NVIDIA’s intermediate assembly language. During initialization, the engine executes cuModuleLoadDataEx to pass the PTX string directly to the NVIDIA driver33. The host machine's graphics driver intercepts this string and performs Just-In-Time (JIT) compilation, translating the PTX into optimized machine code tailored explicitly for the specific microarchitecture present in the host system (e.g., maximizing tensor core usage on Hopper architecture while falling back gracefully on legacy Pascal architecture)33. The engine establishes a hardware session using cuCtxCreate, defining a unique CUDA context analogous to a CPU process namespace33. Utilizing unmanaged memory handles (CUdeviceptr), the C\# logic coordinates memory transfers (cuMemcpyHtoD) and dispatches highly optimized, quantized GEMV custom kernels alongside standard cuBLAS HGEMM operations for the prefill phases15. This completely severs the reliance on a C++ compiler toolchain; the C\# inference engine is distributed purely as managed assemblies, drastically simplifying continuous integration and deployment pipelines.

DirectX 12 and Compute Shaders via Roslyn

Relying entirely on CUDA alienates host machines lacking NVIDIA hardware. For environments possessing Intel Integrated Graphics or AMD GPUs, the C\# ecosystem possesses a unique capability: transpiling C\# directly into High-Level Shader Language (HLSL) via libraries such as ComputeSharp37. Utilizing advanced Roslyn Source Generators, C\# structures representing computational kernels and decorated with attributes like \[ThreadGroupSize\] are parsed at compile-time37. The source generator automatically outputs optimized DirectX 12 compute shaders and generates the voluminous boilerplate code necessary to orchestrate buffers and descriptor heaps37. This enables the inference engine to dynamically dispatch hardware-accelerated quantized GEMV operations across arbitrary DirectX 12-compatible hardware37. If a physical GPU is completely absent, the framework automatically falls back to the Windows Advanced Rasterization Platform (WARP), executing the HLSL shaders on the CPU via an emulation layer, ensuring guaranteed execution across any Windows environment without altering the codebase38.

Advanced Memory Orchestration and Threading Topologies

The raw computational throughput of hardware intrinsics and GPU kernels is irrelevant if the engine's orchestrator is bottlenecked by threading contention or memory fragmentation. Enterprise-grade inference requires complex memory management and precise thread synchronization.

PagedAttention and Copy-on-Write Caching

The Key-Value (KV) cache is the memory structure wherein the neural network stores the intermediate mathematical states of all previously processed tokens, allowing the model to "remember" context without recalculating it13. A naive implementation statically allocates a contiguous multidimensional array for the KV cache, sized to accommodate the absolute maximum sequence length for a given batch40. In environments handling continuous, asynchronous chat requests of varying lengths, static allocation results in catastrophic internal memory fragmentation, routinely wasting up to 95% of allocated VRAM40. Inspired by operating system virtual memory management and frameworks like vLLM, a robust C\# engine implements Paged KV-Caching13. The engine segments the cache memory into a global KvBlockPool containing small, fixed-size physical blocks (e.g., accommodating 16 tokens each)40. Every concurrent request maintains a KvBlockTable—a logical page table mapping the continuous sequence to non-contiguous physical blocks40. This architecture unlocks highly advanced capabilities:

• Zero-Waste Allocation: Memory blocks are dynamically allocated strictly on demand as the text generation progresses, ensuring nearly 100% memory utilization40.
• Copy-on-Write (CoW): When executing Beam Search (a sampling technique that explores multiple probable sentence branches simultaneously), all beams initially share an identical prompt prefix. Through atomic reference counting (Interlocked.Increment), the engine allows multiple sequences to point to the exact same physical blocks in RAM. The memory is only duplicated when a specific beam diverges and requires a unique mutation, saving vast amounts of memory40.
• Prefix Caching: Frequently utilized system prompts are maintained in a global Least Recently Used (LRU) cache. When a new query arrives featuring a known system prompt, the engine instantly maps the new sequence to the existing physical blocks, entirely bypassing the computationally expensive prefill phase for the prompt prefix40.

NUMA Awareness and Dispatch Fusion

The standard .NET ThreadPool is engineered for highly asynchronous, I/O-bound operations (like handling HTTP requests)21. It is entirely unsuited for orchestrating parallel matrix multiplication, where threads must synchronize within microseconds. Consequently, a high-performance C\# engine must implement a custom ComputeThreadPool21. Profiling reveals several major bottlenecks that the custom thread pool must mitigate:

1. Kernel Transition Overhead: Issuing independent parallel loops for the neural network's Query, Key, and Value (Q/K/V) projections requires the threads to wait at synchronization barriers (ManualResetEventSlim) between every operation. The C\# engine optimizes this by fusing the Q/K/V projections into a single, unified thread dispatch, calculating all three matrices sequentially per partition42. Similarly, the Feed-Forward Gate and Up projections are fused42. This fusion eliminates hundreds of kernel-level context switches per token, resulting in measurable throughput enhancements42.
2. Hybrid CPU Asymmetry: Modern processors (e.g., Intel's Alder Lake architecture) feature asymmetric cores, split between high-performance P-cores and low-power E-cores43. If the OS scheduler arbitrarily assigns a matrix multiplication thread to an E-core, the faster P-cores complete their partitions rapidly and sit idle at the synchronization barrier, stalling the entire inference step43. The C\# orchestrator queries low-level OS APIs to map the processor topology, explicitly utilizing thread affinity masks to pin the compute workers exclusively to P-cores44.
3. Adaptive Spin-Waiting: During autoregressive decoding, the work required per layer is exceedingly brief (often under 10 microseconds). In such scenarios, invoking standard thread sleeping mechanisms introduces unacceptable system call latency. The engine employs adaptive spin-waiting: threads actively burn CPU cycles checking a volatile flag for a brief duration before yielding44. For short decode sequences, this avoids kernel sleep latency entirely, accelerating token generation44.
4. NUMA Node Replication: On massive multi-socket server motherboards, fetching tensor data from RAM physically attached to a remote CPU socket cripples memory bandwidth44. The engine detects Non-Uniform Memory Access (NUMA) topologies, strategically replicating the loaded weights across sockets and pinning compute threads to local memory controllers to preserve maximum bandwidth44.

Tokenization and Structured Constraint Decoding

Translating text into tokens—and vice versa—requires precise string manipulation. Traditional Byte-Pair Encoding (BPE) implementations rely heavily on Regular Expressions (Regex). In a high-throughput environment, evaluating complex regex patterns across large prompt sequences incurs massive CPU penalties45. Optimized C\# tokenizers abandon regex in favor of deterministic finite state machines, such as Trie structures or Aho-Corasick automatons43. These structures parse the input string character-by-character without backtracking, operating in strict [Figure omitted from source export] time complexity to rapidly isolate multi-token patterns and specific control tokens43. Once tokens are generated, modern enterprise applications require Structured Output, guaranteeing that the LLM response strictly conforms to a defined format, such as a specific JSON schema or a precise Regular Expression. A naive implementation of this feature forces the engine to clone the entire tracking state for the model's entire vocabulary (often exceeding 128,000 tokens) at every generation step to evaluate which tokens are valid. This cloning generates massive garbage collection pressure, routinely allocating over 160MB of memory per token cache miss43. To resolve this, the native engine employs Pushdown Automata (PDA) and advanced Finite State Machines (FSM) to calculate logical logit masks14. By employing simulate-and-rollback mechanics and utilizing InlineArray stack structures allocated entirely on the stack (avoiding the heap entirely), the engine selectively masks out syntactically invalid tokens before the softmax sampling phase occurs14. This guarantees that the final output perfectly adheres to the defined grammar or JSON schema, eliminating the need for expensive post-generation validation and retries, with virtually zero impact on the Garbage Collector14. Similarly, when executing Top-K sampling (truncating the vocabulary pool to the highest probability candidates), invoking standard sorting algorithms like Array.Sort() across 128,000 floating-point logits is catastrophic to performance, as full sorts operate in [Figure omitted from source export] complexity43. The C\# engine instead implements introselect or partial quicksort methodologies, operating in [Figure omitted from source export] complexity to isolate only the top K bounds, radically accelerating the sampling pipeline43. To further amplify text generation speed, the engine can utilize speculative decoding methodologies such as the Draft-Verify-Accept pattern14. In this configuration, a highly efficient, smaller secondary model generates a rapid sequence of "draft" tokens. The primary, massive model then evaluates this entire sequence of tokens in a single, parallel forward pass. If the primary model agrees with the draft sequence, all tokens are accepted instantly, effectively allowing the engine to generate multiple tokens per forward pass and multiplying decode speed14.

Extensibility: Dynamic LoRA Adapters and Multi-Modal Vision

Modern inference architectures must be highly extensible. Rather than fine-tuning a massive foundational model entirely, the industry relies on Low-Rank Adaptation (LoRA). LoRA introduces small, low-rank matrices into the attention layers that shift the model's behavior3. Instead of statically merging these adapters into the foundational weights—which forces a complete reload of the model to change tasks—a robust C\# engine applies the adapter dynamically during the forward pass using the formula [Figure omitted from source export]46. Because these adapter files (often loaded via the SafeTensors format) only span 10MB to 100MB, they are retained in CPU or GPU RAM46. This enables "Hot Loading," allowing the server to swap adapters dynamically between individual requests46. In continuous batching scenarios, the engine partitions the batch, executing the base matrix multiplication for all sequences simultaneously, and subsequently calculating the unique LoRA deltas only for the specific sequences requesting that adapter46. Furthermore, the engine supports multi-modal inference by ingesting .mmproj (multi-modal projector) sidecar files16. These visual projectors translate external pixel or audio data into embedded vectors that the language model can comprehend, allowing the C\# environment to seamlessly execute computer vision and image classification tasks using standard language model orchestration pathways47.

Concluding Perspectives

The systematic reconstruction of the llama.cpp inference stack into a fully native C\# architecture represents a significant technological milestone for the .NET ecosystem. By meticulously parsing the GGUF specification, exploiting unmanaged memory mapped files to bypass 32-bit legacy limits, and orchestrating cycle-accurate AVX2 and AVX-512 SIMD intrinsics, the severe latency and throughput penalties traditionally associated with managed languages have been eradicated. This architecture fundamentally validates that the modern .NET runtime is highly capable of executing the lowest-level mathematical operations required for artificial intelligence. Furthermore, it introduces distinct advantages that unmanaged C/C++ environments struggle to replicate natively: absolute immunity to buffer overflow vulnerabilities during file parsing, the unparalleled parallel orchestration capabilities of modern asynchronous patterns, and the capacity to compile directly via Native AOT to achieve deployment footprints devoid of fragile binary dependencies. As localized AI continues to integrate into highly secure, complex enterprise environments, the fusion of zero-allocation tensor mechanics with the intrinsic safety and scalability of the C\# ecosystem establishes a formidable foundation for the future of intelligent systems.

Works cited

1. Llama.cpp \- Run LLM Inference in C/C++, https://llama-cpp.com/
2. GGUF · Hugging Face, https://huggingface.co/docs/hub/gguf
3. GGUF versus GGML \- IBM, https://www.ibm.com/think/topics/gguf-versus-ggml
4. LM-Kit.NET vs LLamaSharp, On-Device .NET AI SDK Comparison, LM-Kit, https://lm-kit.com/why-local-ai/compare/lm-kit-vs-llama-sharp/
5. SciSharp STACK, https://scisharp.github.io/SciSharp/
6. SciSharp \- NuGet Gallery, https://www.nuget.org/profiles/SciSharp
7. SciSharp repositories \- GitHub, https://github.com/orgs/SciSharp/repositories
8. LLamaSharp Documentation, https://scisharp.github.io/LLamaSharp/0.5/
9. Orfeous/llamacpp.net: C\#/.NET binding of llama.cpp \- GitHub, https://github.com/Orfeous/llamacpp.net
10. Running Local AI with LlamaSharp in .NET: A Developer's Guide \- C\# Corner, https://www.c-sharpcorner.com/article/running-local-ai-with-llamasharp-in-net-a-developers-guide/
11. Architecture \- LLamaSharp Documentation, https://scisharp.github.io/LLamaSharp/0.14.0/Architecture/
12. Architecture \- LLamaSharp Documentation, https://scisharp.github.io/LLamaSharp/0.4/Architecture/
13. TensorSharp: Open Source Local LLM Inference Engine written by C\# : r/dotnet \- Reddit, https://www.reddit.com/r/dotnet/comments/1u4vzgw/tensorsharp\_open\_source\_local\_llm\_inference/
14. dotLLM — Native .NET LLM Inference, https://dotllm.dev/
15. Introducing dotLLM \- Building an LLM Inference Engine in C\# | Konrad 'Dev Nerd' Kokosa, https://kokosa.dev/blog/2026/dotllm/
16. dotLLM/docs/GGUF\_FORMAT.md at main \- GitHub, https://github.com/kkokosa/dotLLM/blob/main/docs/GGUF\_FORMAT.md
17. MemoryMappedFile CreateViewAccessor throws "Not enough storage is available to process this command." \- Stack Overflow, https://stackoverflow.com/questions/15662455/memorymappedfile-createviewaccessor-throws-not-enough-storage-is-available-to-p
18. Memory Mapped File to Read End of File? \- Stack Overflow, https://stackoverflow.com/questions/4402725/memory-mapped-file-to-read-end-of-file
19. GGML GGUF File Format Vulnerabilities | Databricks Blog, https://www.databricks.com/blog/ggml-gguf-file-format-vulnerabilities
20. 10x Performance with SIMD Vectorized Code in C\#/.NET \- xoofx, https://xoofx.github.io/blog/2023/07/09/10x-performance-with-simd-in-csharp-dotnet/
21. dotLLM/docs/ROADMAP.md at main \- GitHub, https://github.com/kkokosa/dotLLM/blob/main/docs/ROADMAP.md
22. Step 25: Row-interleaved weight repacking — contiguous multi-row block layout · Issue \#52 · kkokosa/dotLLM \- GitHub, https://github.com/kkokosa/dotLLM/issues/52
23. dotLLM/docs/QUANTIZATION.md at main \- GitHub, https://github.com/kkokosa/dotLLM/blob/main/docs/QUANTIZATION.md
24. picoLLM — Inference Engine for X-Bit Quantized LLMs \- Picovoice, https://picovoice.ai/blog/picollm-inference-engine-for-x-bit-quantized-llms/
25. SciSharp/TensorSharp2: A C\# inference engine for running large language models (LLMs) locally using GGUF model files. TensorSharp provides both a console application and a web-based chatbot interface for multi-turn conversations with multimodal models. · GitHub, https://github.com/SciSharp/TensorSharp2
26. SIMD-accelerated types in .NET \- Microsoft Learn, https://learn.microsoft.com/en-us/dotnet/standard/simd
27. Avx2.Multiply Method (System.Runtime.Intrinsics.X86) \- Microsoft Learn, https://learn.microsoft.com/en-us/dotnet/api/system.runtime.intrinsics.x86.avx2.multiply?view=net-10.0
28. SIMD Accelerated Numeric Types in C\# \- Code Maze, https://code-maze.com/csharp-simd-accelerated-numeric-types/
29. .NET Core Concepts (SIMD, AVX, Intrinsics) | Marian Todorov \- Medium, https://medium.com/@meriffa/net-core-concepts-simd-avx-intrinsics-0e30c845ebca
30. Step 10: SIMD kernel tuning — benchmark-driven Q8\_0/Q4\_K GEMV optimization · Issue \#26 · kkokosa/dotLLM \- GitHub, https://github.com/kkokosa/dotLLM/issues/26
31. goldshtn/simd-workshop: Exercises and sample code for a C\# SIMD (vectorization) workshop \- GitHub, https://github.com/goldshtn/simd-workshop
32. Add GPU support to ggml · ggml-org llama.cpp · Discussion \#915 \- GitHub, https://github.com/ggml-org/llama.cpp/discussions/915
33. 3.3. The CUDA Driver API — CUDA Programming Guide, https://docs.nvidia.com/cuda/cuda-programming-guide/03-advanced/driver-api.html
34. Porting CUDA driver API — HIP 6.3.42133 Documentation, https://rocm.docs.amd.com/projects/HIP/en/docs-6.3.1/how-to/hip\_porting\_driver\_api.html
35. 1\. Introduction — PTX Compiler API 13.3 documentation, https://docs.nvidia.com/cuda/ptx-compiler-api/index.html
36. Passing the PTX program to the CUDA driver directly \- Stack Overflow, https://stackoverflow.com/questions/15842507/passing-the-ptx-program-to-the-cuda-driver-directly
37. Announcing ComputeSharp 2.0 — run C\# on the GPU with ease through DirectX 12 and D2D1\! \- Sergio Pedri, https://sergiopedri.medium.com/announcing-computesharp-2-0-run-c-on-the-gpu-with-ease-through-directx-12-and-d2d1-be4f3f2312b4
38. ComputeSharp 3.2.0 \- NuGet, https://www.nuget.org/packages/ComputeSharp
39. ComputeSharp, Run C\# on the GPU \- Hacker News, https://news.ycombinator.com/item?id=26234384
40. dotLLM/docs/KV\_CACHE.md at main \- GitHub, https://github.com/kkokosa/dotLLM/blob/main/docs/KV\_CACHE.md
41. dotLLM/docs/SAMPLING.md at main \- GitHub, https://github.com/kkokosa/dotLLM/blob/main/docs/SAMPLING.md
42. Step 23: Decode dispatch optimization — fused projections and pre-quantization reuse · Issue \#50 · kkokosa/dotLLM \- GitHub, https://github.com/kkokosa/dotLLM/issues/50
43. Wave 7: CPU performance — partial sort, schema cache, AVX2 gaps, tokenizer · Issue \#109 · kkokosa/dotLLM \- GitHub, https://github.com/kkokosa/dotLLM/issues/109
44. Step 30: NUMA-aware threading — spin-wait, topology detection, CPU pinning · Issue \#57 · kkokosa/dotLLM \- GitHub, https://github.com/kkokosa/dotLLM/issues/57
45. Gyula Rabai's Projects, https://gyularabai.com/p\_8846-projects.html
46. dotLLM/docs/LORA.md at main \- GitHub, https://github.com/kkokosa/dotLLM/blob/main/docs/LORA.md
47. LlamaSharp \- Voxta Docs, https://doc.voxta.ai/docs/server/services/llm/llamasharp