Advanced Engineering of Distilled .slm Models for Browser-Local Rust/WASM Inference

Full report

On this page

Introduction to Browser-Local Inference Dynamics

The deployment of large language models directly into the browser environment represents a fundamental shift in privacy-preserving, edge-compute architectures. For platforms like TinyRustLM, executing distilled models entirely within the client's local execution context eliminates network latency, guarantees cryptographic-level data privacy by ensuring zero data exfiltration, and entirely removes server-side inference compute costs. However, deploying useful language models into a WebAssembly (WASM) sandbox presents severe systems engineering challenges that require a radical departure from traditional server-side inference engine design. WebAssembly environments are strictly constrained by the underlying physics of the virtual machine. Foremost is the 4GB contiguous linear memory boundary inherent to the wasm32 specification, which imposes a hard ceiling on the combined footprint of the model weights, the dynamic key-value (KV) cache, and the operational heap1. Furthermore, browser execution environments restrict multi-threading capabilities, enforce synchronous execution limitations on the main thread to prevent user interface blocking, and lack native operating system memory management primitives such as hardware-accelerated memory mapping (the mmap syscall)1. To achieve viable token generation rates while adhering to these constraints, inference engines must synthesize optimized model architectures, aggressive weight quantization schemes, custom Single Instruction Multiple Data (SIMD) dot-product kernels, and highly specialized memory management strategies designed specifically for the WASM memory model. This analysis details the architectural, mathematical, and low-level optimizations required to engineer a highly performant, browser-local Rust/WASM inference engine for sub-billion parameter models.

Model Architecture Choices for the WASM Execution Environment

When targeting the browser, parameter scale is strictly bounded by the memory capabilities of the client device and the architectural limits of the wasm32 target. While proposals such as WebAssembly Garbage Collection (WasmGC) and Memory64 are advancing, the current pragmatic limit for deterministic cross-browser support dictates that the entire execution state must fit within a 4GB address space2. Consequently, the optimal model size for broad device compatibility falls within the 135M to 360M parameter range, with absolute maximum bounds approaching 1.5B parameters for client hardware featuring sufficient unified memory6. Distilled models such as SmolLM2-135M and SmolLM2-360M exhibit architecture patterns highly suited for WASM deployment8. These models prioritize network depth over width and utilize Grouped-Query Attention (GQA) rather than standard Multi-Head Attention (MHA). GQA is mathematically critical for edge inference because it drastically reduces the memory bandwidth required during the autoregressive decode phase. By sharing key and value projection heads across multiple query heads, the footprint of the KV cache is reduced by a factor equal to the group size. In environments where memory allocations cannot be easily swapped to disk, delaying the onset of out-of-memory errors through GQA reduces the pressure on the WASM linear memory allocator. The depth-over-width architectural choice further complements the browser execution model. Narrower hidden dimensions reduce the size of the intermediate activation tensors that must be allocated dynamically during the forward pass, effectively minimizing the peak transient memory required for each generation step.

Recurrence Equivalence and Parameter Sharing

To maximize logical depth without exhausting the 4GB WASM linear memory limit, architectures can leverage parameter sharing via looped transformers. Implementing a prelude-recur-coda architecture template allows the engine to repeatedly execute a shared recurrent block of transformer layers11. By fixing a prelude and coda layer set, the recurrent block can be cycled multiple times. The effectiveness of this parameter sharing is mathematically described by the recurrence-equivalence exponent [Figure omitted from source export]. Empirical scaling laws demonstrate that looping a block [Figure omitted from source export] times is not strictly equivalent to [Figure omitted from source export] unique blocks (which would yield [Figure omitted from source export]), nor does it yield zero capacity gain ([Figure omitted from source export]). Instead, recent evaluations recover a recurrence-equivalence exponent of [Figure omitted from source export]11. This means that a looped model can perform on par with a non-looped model possessing significantly more parameters, but with a vastly reduced static memory footprint. For instance, a 410M parameter looped model can achieve the validation loss of a 580M parameter non-looped model, provided the inference engine accepts the higher computational cost of repeated forward passes11. In a WASM environment where memory capacity is a harder limit than raw CPU compute time, leveraging architectures with high [Figure omitted from source export] values allows TinyRustLM to punch above its weight class, delivering superior semantic coherence while remaining safely within the bounds of wasm32 memory constraints.

Memory-Mapped vs Streaming .slm Loading Mechanisms

A primary obstacle in browser-based machine learning is initialization latency. In native environments, loading a multi-gigabyte model is almost exclusively accelerated using mmap, which maps the file directly from the filesystem into the process's virtual address space, deferring actual I/O until specific memory pages are accessed. Because WebAssembly lacks access to the OS-level mmap syscall, a naive implementation that fetches a model via standard HTTP requests and loads it entirely into a JavaScript ArrayBuffer forces the browser to allocate memory in the V8 heap, copy the data into the WASM linear memory, and subsequently trigger massive garbage collection sweeps. This sequence leads to severe initialization latency spikes, memory fragmentation, and potential browser tab crashes4. To circumvent this constraint, the engine must utilize the Origin Private File System (OPFS) combined with the FileSystemSyncAccessHandle API, executing exclusively within a Dedicated Web Worker4. The OPFS provides a highly optimized, origin-sandboxed virtual file system that interacts directly with the local disk. By downloading the quantized .slm model weights via chunked HTTP byte-range requests and writing them sequentially to an OPFS file, the engine avoids pinning massive data blobs in the JavaScript heap4. During inference initialization, the Rust WASM module requests a synchronous access handle to the OPFS file. Instead of loading the entire file into memory, the engine dynamically reads specific byte ranges directly into pre-allocated WASM linear memory segments just-in-time for the forward pass, effectively emulating an out-of-core streaming architecture without violating the browser sandbox. The .slm file format utilized by TinyRustLM must be explicitly structured to support this lazy-loading mechanism. The Safetensors format provides an optimal baseline for this endeavor15. It begins with an 8-byte unsigned little-endian integer denoting the exact length of the subsequent JSON header. The engine reads only these initial bytes to calculate the header size, fetches the JSON metadata, and parses the exact byte offsets of every tensor in the model without touching the heavy weight payloads16. This precise byte-offset mapping allows the engine to stream individual layers directly into the CPU cache equivalents during the prefill and decode phases, ensuring that the peak memory footprint never substantially exceeds the size of a single transformer block.

Startup Validation Cost

Cryptographic validation of model integrity presents a secondary computational bottleneck at startup. Computing a SHA-256 hash over a file ranging from 400MB to 1.5GB using a WASM-compiled Rust module introduces unacceptable initialization delays. Instead, the engine should rely on Subresource Integrity (SRI) hashes applied at the network fetch layer18. By delegating the cryptographic verification to the browser's highly optimized, native C++ networking stack during the initial download, the application guarantees that the weights stored in the OPFS have not been tampered with or corrupted in transit. Upon subsequent local loads, the engine can bypass full-file hashing entirely. It relies on the sandboxed security guarantees of the OPFS and only parses the JSON header to confirm structural integrity before commencing inference, thus reducing startup time from tens of seconds to mere milliseconds.

Matrix Multiplication Layout and Structural Memory Swizzling

To execute efficiently on edge CPUs, models must undergo aggressive post-training quantization. While 8-bit integer quantization (INT8) offers near-lossless perplexity degradation, 4-bit integer quantization (INT4) represents the optimal Pareto frontier for browser environments. INT4 balances acceptable quality loss with halved memory bandwidth requirements, shrinking a 135M parameter model to approximately 101MB, and a 360M parameter model to roughly 369MB6. Standard block quantization formats, such as Q4\_K\_M, subdivide weight matrices into super-blocks of 256 weights, which are further partitioned into sub-blocks of 32 weights. Each sub-block carries its own independent scaling factor and zero-point12. This localized scaling is vital because it preserves the dynamic range of outlier weights, which are empirically critical for maintaining the reasoning capabilities of heavily distilled models20. The layout of these quantized matrices in the WASM linear memory must be explicitly engineered for the access patterns of the underlying CPU architecture. Traditional row-major formats force the CPU to jump across memory strides when computing column-wise dot products, effectively destroying cache locality. To mitigate this, the engine must implement an offline memory packing phase where the Q4\_K blocks are swizzled into cache-friendly tiles. For example, weights can be interleaved such that 16 adjacent 4-bit values are packed into contiguous 64-bit segments, precisely matching the alignment requirements of 128-bit SIMD registers21. Advanced implementations perform type pre-conversion during this packing phase, where mini-floats or specialized formats are upcast to the compute type once during packing, rather than on every General Matrix Multiply (GEMM) call. This amortizes the conversion cost across all rows of the activation matrix that will be multiplied against the packed weights21. Furthermore, padding rows to match the exact SIMD vector width allows inner loops to load data without executing costly boundary checks, preserving the instruction pipeline.

Quantized Kernel Design: SIMD and Non-SIMD Fallback Paths

The most significant performance multiplier in browser-based CPU inference is the utilization of Single Instruction, Multiple Data (SIMD) capabilities. The WebAssembly simd128 specification introduces the v128 register type and a suite of 236 fixed-width operations that map directly to native NEON instructions on ARM architectures and SSE/AVX instructions on x86 architectures3. Compiling the Rust kernels with the \-C target-feature=+simd128 flag instructs the LLVM backend to auto-vectorize loops. However, for optimal performance, the matrix multiplication inner loops must be written using explicit std::arch::wasm32 intrinsics to avoid compiler heuristics that may sub-optimally unroll the operations23. Standard SIMD execution of a 4-bit quantized dot product requires unpacking 4-bit weights into 8-bit integers, executing a widening multiplication into 16-bit integers using operations like i16x8\_extmul\_low\_i8x16 and i16x8\_extmul\_high\_i8x16, and then performing pairwise additions to accumulate the intermediate sums into a 32-bit register using i32x4\_add26. While this vectorized approach is drastically faster than scalar execution, this multi-instruction sequence still occupies valuable clock cycles and increases register pressure.

Relaxed SIMD and Fused Operations

To achieve maximum throughput, the engine must leverage the WebAssembly Relaxed SIMD proposal, a standard supported in most modern browsers2. Relaxed SIMD deliberately introduces local non-determinism, allowing the WASM runtime to utilize host-specific hardware optimizations—such as Fused Multiply-Add (FMA) instructions—without enforcing bit-perfect cross-platform consistency for edge-case floating-point behaviors like NaN propagation or subnormal rounding2. Crucially, the Relaxed SIMD specification introduces the wasm\_i32x4\_relaxed\_dot\_i8x16\_i7x16\_add instruction. This fused operation accepts two 128-bit vectors of 8-bit integers, computes their dot product, and accumulates the result into a 32-bit vector in a single CPU cycle, bypassing intermediate 16-bit widening28. Integrating this specific instruction into the Q4\_K dot-product kernel fundamentally collapses the inner loop. Benchmarks and recent open-source repository optimizations indicate that utilizing this relaxed dot-product instruction yields a 2x throughput acceleration for quantized matrix multiplications in WebAssembly, effectively bridging the performance gap between browser environments and native CPU execution28.

Single-Threaded Constraints and The Scalar Fallback

WebAssembly multi-threading requires the use of SharedArrayBuffer objects, which are gated behind stringent Cross-Origin Opener Policy (COOP) and Cross-Origin Embedder Policy (COEP) security headers25. In environments where these headers cannot be configured, or on restrictive mobile browsers, the inference engine is forced into a strictly single-threaded execution model. Over-spawning Web Workers can also cause severe context-switching thrashing on mobile devices, necessitating careful thread pooling22. Furthermore, because the target environment encompasses a vast array of devices, including legacy hardware that lacks 128-bit vector support, the engine must compile a secondary, scalar-only fallback path. If the WebAssembly.validate() API detects that simd128 or Relaxed SIMD opcodes are unsupported by the host engine, the runtime dynamically routes to the scalar WASM module24. The scalar fallback must be meticulously unrolled by the Rust compiler to maximize instruction pipelining, ensuring that the application remains functional, albeit at a reduced token generation rate.

Prompt Prefill Versus Decode Dynamics

The inference lifecycle consists of two distinct computational regimes: the prefill phase and the decode phase. These phases are constrained by entirely different hardware bottlenecks, requiring independent optimization strategies within the WASM engine. The prefill phase processes the entire input prompt in parallel to populate the initial KV cache. This operation relies on dense matrix-matrix multiplications (GEMM) and is fundamentally compute-bound by the arithmetic logic units (ALUs) of the CPU12. In a single-threaded WASM environment, passing a 2,048-token prompt through a 360M parameter model will monopolize the JavaScript main thread. This compute saturation locks the browser's UI render loop and triggers watchdog timeouts, leading to an unresponsive page. To mitigate this, the engine must implement chunked prefilling34. By slicing the input prompt into manageable chunks (e.g., 256 tokens) and yielding execution back to the browser's event loop via asynchronous Rust functions and await points, the application maintains high UI responsiveness while amortizing the prefill compute cost over a slightly longer duration. Conversely, the decode phase generates one token at a time autoregressively. This process relies on matrix-vector multiplications (GEMV), which are overwhelmingly memory-bandwidth bound12. During decode, the CPU arithmetic units remain heavily underutilized because they are starved waiting for the model weights to be fetched from system RAM into the L1/L2 caches. Therefore, optimizing the decode phase relies entirely on the efficacy of the Q4\_K memory packing, the reduction of memory bus saturation, and the minimization of pointer chasing during attention computation. Accurate benchmarking of these phases requires a sequential-dispatch measurement method. Naive single-operation measurements often overestimate the per-dispatch cost by conflating the dispatch overhead with pipeline synchronization. By queuing [Figure omitted from source export] dispatches and syncing only once at the end, the engine can accurately isolate the true per-dispatch latency and tune the kernel accordingly35.

KV-Cache Paging, Compression, and Context Length Tradeoffs

During the autoregressive decode phase, the engine must store the Key and Value vectors for every historical token to prevent recomputing the entire sequence context. In a flat tensor allocation model, predicting the maximum sequence length forces the engine to pre-allocate massive contiguous buffers in the WASM heap. If the generation terminates early, this memory is wasted; if the generation exceeds the buffer, the engine is forced to halt, as dynamically growing flat allocations requires a complete copy of the data, resulting in massive relocation overhead and fragmentation12. Furthermore, standard scaled dot-product attention materializes an [Figure omitted from source export] attention mask. At a context length of 8,192 tokens, a single session can allocate over a gigabyte of memory just for the attention masks, which is fatal within a 4GB WASM limit37. The solution requires implementing a PagedAttention architecture natively within the Rust runtime38. PagedAttention decouples the logical sequence of tokens from their physical location in linear memory by managing the KV cache through an internal virtual page table36. The WASM linear memory is divided into a fixed pool of blocks, each sized to hold a discrete number of tokens—typically 16 or 32 tokens per block40. As generation progresses, the sequence manager allocates new blocks on demand from a free list. When a request finishes, its blocks are immediately returned to the pool without triggering a global garbage collection event. This block-based architecture natively supports prefix caching via a radix trie data structure40. For application topologies utilizing a fixed system prompt, the token IDs of the prompt are hashed and stored in the trie. Subsequent inference requests can look up the prompt, immediately retrieve the physical block IDs containing the precomputed KV states, and increment their reference counters40. This zero-compute prefix sharing drastically reduces prefill latency for common prompts and saves tens of megabytes of WASM heap space. To further extend the context window within the strict WASM limits, the KV cache itself must be quantized. Instead of storing Keys and Values in 16-bit floating-point format (BF16 or FP16), the engine should apply a compression scheme analogous to TurboQuant34. By compressing the KV cache down to 4-bit representations (e.g., the turbo4 format), the engine achieves up to a 3.7x reduction in memory consumption43. This requires highly specialized SIMD kernels capable of dequantizing the KV blocks on-the-fly during the attention reduction step. The trade-off is a slight increase in ALU utilization during the memory-bound decode phase, which is generally masked by the latency of fetching the model weights, resulting in a net gain in context capacity with minimal penalty to token generation speed.

Deterministic Sampling, Repetition Penalties, and EOS Handling

Because browser hardware varies wildly, ensuring consistent output for identical inputs across different devices requires strict determinism in the sampling phase. Standard floating-point math in WebAssembly can sometimes exhibit non-determinism, particularly regarding NaN handling and subnormal numbers29. To counteract this during the critical sampling step, the engine must utilize a mathematically deterministic Pseudo-Random Number Generator (PRNG), such as ChaCha8 or PCG32, seeded explicitly by the application logic, and execute all probability calculations using integer or fixed-point arithmetic wherever possible. Repetition degradation is a known failure mode in heavily quantized distilled models, where the model enters an infinite loop of repeating the same phrase. Standard penalty mechanisms involve applying a fixed scalar penalty to the logits of previously generated tokens. However, advanced engines utilize an information-theoretic approach based on sliding window compression algorithms to calculate dynamic penalties44. By simulating a Lempel-Ziv (LZ) compressor over the causal sequence, the engine calculates the change in code length [Figure omitted from source export] for each potential next token. The penalty is applied directly to the logit vector [Figure omitted from source export] in the log-domain: [Figure omitted from source export] This dynamic, compression-based penalty exponentially penalizes loops and repetitive structural syntax without arbitrarily suppressing common syntax tokens (e.g., articles and prepositions), which is a severe flaw in static frequency penalties45. Furthermore, to align the adjusted probabilities back to the target distribution without losing information, the engine can employ logit scaling utilizing a Bi-Proportionate Scaling algorithm, which minimizes information loss during the alignment process46. Handling the End-Of-Sequence (EOS) token requires continuous evaluation of the logit probabilities against a defined threshold. In highly distilled models, the EOS logit can sometimes be overshadowed by hallucinated continuation tokens, especially when the sliding window context drops critical semantic cues. The sampler must employ temperature scaling and top-p (nucleus) filtering to mathematically truncate the long tail of the probability distribution, forcing the model into higher-confidence terminal states when the semantic completion is imminent.

Diagnostics and Smoke Tests for Production Environments

Supporting an inference engine deployed across thousands of heterogeneous browser environments requires robust, low-overhead diagnostics. Memory leaks in the WASM linear memory are fatal, as the buffer cannot automatically shrink once grown; it can only be reclaimed when the entire WASM instance is destroyed47. To track dynamic heap allocations in Rust without the overhead of injecting full malloc tracing into high-performance loops, the engine must bypass standard external profilers48. Instead, it wraps the default allocator abstraction, std::alloc::System, with an atomic counting allocator47. By injecting this tracing allocator, the engine intercepts every alloc and dealloc call, summing the requested layout sizes into an AtomicIsize. This provides the application layer with exact, real-time byte metrics regarding heap fragmentation and KV cache consumption. This telemetry allows the sequence manager to proactively gracefully reject incoming prompts if the predicted KV cache growth exceeds the available remaining linear memory, preventing hard crashes.

Remote Inference Smoke Tests

A core value proposition of browser-local inference is cryptographic privacy. To definitively prove to enterprise clients and security auditors that the inference is strictly local and zero-exfiltration is guaranteed, the application must implement verifiable network isolation smoke tests. This is achieved by instantiating the Rust WASM module inside a Dedicated Web Worker and systematically stripping the worker's global scope of the fetch and XMLHttpRequest APIs immediately after the OPFS weights are successfully mounted. A programmatic smoke test then intentionally attempts to execute an outbound network request from within the WASM module. If the environment is properly secured, the invocation will fail. Additionally, intercepting all outbound requests at the Service Worker level provides a secondary defense layer. If any telemetry, prompt data, or token output attempts to egress the origin via HTTP or WebSockets, the Service Worker triggers an immediate hard exception, mathematically proving that the execution environment is completely disconnected from remote inference servers.

Performance Targets and Comprehensive Benchmark Plans

For an optimized Rust/WASM engine running on consumer-grade hardware (e.g., Apple M-series or Intel 12th-Gen architectures), performance targets must be rigorously defined to ensure practical viability. Benchmarking efforts must measure not only throughput but also energy efficiency (Tokens per Joule, tok/J), which is critical for laptop and mobile battery life9.

Data synthesized from benchmark ranges on 15W to 25W CPU power envelopes utilizing SIMD acceleration9. A comprehensive benchmark plan must isolate the prefill and decode phases. Utilizing the sequential-dispatch measurement method, the engine executes [Figure omitted from source export] sequential prefill blocks, syncing only at the end to isolate the true computational throughput from WASM-to-JS bridge synchronization overhead35. Energy efficiency is calculated by dividing the output token length by the product of the decode power draw and the [Figure omitted from source export] decode latency per token9.

Architecture Recommendations and Acceptance Criteria

To realize the full potential of browser-local AI using TinyRustLM, engineering efforts must aggressively pursue hardware-sympathetic design. Relying on naive ports of standard C++ tensor libraries into WebAssembly ignores the unique constraints of the browser sandbox, resulting in unacceptable memory bloat and sluggish token generation. The definitive architecture recommendation is to establish a data pipeline that streams Safetensor-formatted .slm weights directly into the Origin Private File System, bypassing the JavaScript heap entirely to preserve linear memory for activations. The Rust backend must implement PagedAttention with discrete 16-token block sizes to eliminate virtual memory fragmentation, coupled with a radix trie to ensure zero-copy prompt caching. Computationally, the matrix math core must be hardcoded using std::arch::wasm32 intrinsics, explicitly utilizing the wasm\_i32x4\_relaxed\_dot\_i8x16\_i7x16\_add instruction to collapse the multi-instruction sequence required for processing Q4\_K\_M INT4 quantized blocks. Acceptance Criteria for Production Deployment:

1. Memory Integrity: Peak WASM linear memory usage must remain strictly below 2.5GB for a 360M parameter model with a 4,096-token context window, ensuring absolute safety against 4GB browser crashes.
2. Latency: Time-To-First-Token (TTFT) must not exceed 800 milliseconds for a 512-token prompt utilizing chunked prefilling to maintain UI thread responsiveness.
3. Determinism and Quality: Given an identical seed, the generation sequence must match bit-for-bit across Chrome, Safari, and Firefox, utilizing LZ-based repetition penalties to prevent degradation.
4. Kernel Coverage: The engine must successfully compile and dynamically route logic through both wasm32-simd128 relaxed dot-product kernels and scalar fallbacks based on runtime capability probing.

By unifying advanced quantization heuristics, cache-aware memory paging, and relaxed SIMD vectorization, the engine successfully circumvents the historic limitations of WebAssembly. This synthesis of techniques transforms the web browser into a highly capable, cryptographically private execution environment capable of driving sophisticated, small-scale language models at native-equivalent speeds.

Works cited

1. wasm64: support memory larger than 16 GB · Issue \#1892 · WebAssembly/spec \- GitHub, https://github.com/WebAssembly/spec/issues/1892
2. WebAssembly 3.0 Is Official: Nine Features That Change What Wasm Can Do | byteiota, https://byteiota.com/webassembly-30-spec-release/
3. WebAssembly and WebGPU enhancements for faster Web AI, part 1 | Blog | Chrome for Developers, https://developer.chrome.com/blog/io24-webassembly-webgpu-1
4. Brainwires/rullama: Browser-resident Gemma 4 inference in pure Rust → WebAssembly \+ WebGPU \- GitHub, https://github.com/Brainwires/rullama
5. Rust \+ WebAssembly 2025: Why WasmGC and SIMD Change Everything \- DEV Community, https://dev.to/dataformathub/rust-webassembly-2025-why-wasmgc-and-simd-change-everything-3ldh
6. Run an LLM in Your Browser (2026): Browser-Based AI, No Server | Local AI Master, https://localaimaster.com/blog/run-llm-in-browser
7. On-Device LLMs: State of the Union, 2026 \- Vikas Chandra, https://v-chandra.github.io/on-device-llms/
8. An Evaluation of LLMs Inference on Popular Single-board Computers \- arXiv, https://arxiv.org/html/2511.07425v1
9. Tiny LLM Benchmark: Jetson Orin Nano Super 8GB \- SmolHub, https://www.smolhub.com/posts/jetson-nano-super-benchmark-non-reasoning/
10. SmolLM \- blazingly fast and remarkably powerful \- Hugging Face, https://huggingface.co/blog/smollm
11. How Much Is One Recurrence Worth? Iso-Depth Scaling Laws for Looped Language Models \- arXiv, https://arxiv.org/html/2604.21106v1
12. Llamas on the Web: Memory-Efficient, Performance-Portable, and Multi-Precision LLM Inference with WebGPU \- arXiv, https://arxiv.org/html/2605.20706v1
13. New Project Megathread \- Week of 04 Jun 2026 : r/selfhosted \- Reddit, https://www.reddit.com/r/selfhosted/comments/1tx202z/new\_project\_megathread\_week\_of\_04\_jun\_2026/
14. utooland/opfs-project \- GitHub, https://github.com/utooland/opfs-project
15. Safetensors \- PyTorch, https://pytorch.org/projects/safetensors/
16. GitHub \- safetensors/safetensors: Simple, safe way to store and distribute tensors, https://github.com/safetensors/safetensors
17. Reading Safetensors Headers \- Zenn, https://zenn.dev/platina/articles/e65c73cb01a900?locale=en
18. GitHub \- mlc-ai/web-llm: High-performance In-browser LLM Inference Engine, https://github.com/mlc-ai/web-llm
19. What Is LLM Inference, Really? A Deep Technical Walkthrough \- Karthika Raghavan, https://kraghavan.ca/llm-infrastructure/inference/2026/04/14/re-introduction-to-inference.html
20. MCAP: Deployment-Time Layer Profiling for Memory-Constrained LLM Inference \- arXiv, https://arxiv.org/html/2604.21026v1
21. GitHub \- ashvardanian/NumKong: SIMD-accelerated distances, dot products, matrix ops, geospatial & geometric kernels for 16 numeric types — from 6-bit floats to 64-bit complex — across x86, Arm, RISC-V, and WASM, with bindings for Python, Rust, C, C++, Swift, JS, and Go, https://github.com/ashvardanian/NumKong
22. WASM \+ SIMD for On-Device AI: Private, Fast, Offline | by Thinking Loop | Medium, https://medium.com/@ThinkingLoop/wasm-simd-for-on-device-ai-private-fast-offline-3ef82c47172d
23. core::arch::wasm32 \- Rust, https://doc.rust-lang.org/beta/core/arch/wasm32/index.html
24. WASM SIMD: Browser Support, Features, Limitations | TestMu AI (Formerly LambdaTest), https://www.testmuai.com/learning-hub/wasm-simd-browser-support/
25. 9 WebAssembly \+ Rust Tricks for Native-Speed Web Apps | by Bhagya Rana | Medium, https://medium.com/@bhagyarana80/9-webassembly-rust-tricks-for-native-speed-web-apps-95aff2f8959b
26. ternlight/engine/src/kernels.rs at main \- GitHub, https://github.com/soycaporal/ternlight/blob/main/engine/src/kernels.rs
27. "wasm-relaxed-simd" | Can I use... Support tables for HTML5, CSS3, etc \- CanIUse, https://caniuse.com/wasm-relaxed-simd
28. WASM Relaxed SIMD Enhancement by JeremyCEY · Pull Request \#19590 · ggml-org/llama.cpp \- GitHub, https://github.com/ggml-org/llama.cpp/pull/19590
29. relaxed-simd/proposals/relaxed-simd/Overview.md at main \- GitHub, https://github.com/WebAssembly/relaxed-simd/blob/main/proposals/relaxed-simd/Overview.md
30. Relaxed-math mode, https://sunfishcode.github.io/RelaxedMathMode.pdf
31. Relaxed SIMD · Issue \#1401 · WebAssembly/design \- GitHub, https://github.com/WebAssembly/design/issues/1401
32. Ggml : x2 speed for WASM by optimizing SIMD \- In The News \- Devtalk, https://forum.devtalk.com/t/ggml-x2-speed-for-wasm-by-optimizing-simd/185618
33. DeepSeek-R1 optimizes the llama.cpp WASM runtime by leveraging SIMD instructions \-x2 Speed Increase \- the whole PR is 99% R1 : r/singularity \- Reddit, https://www.reddit.com/r/singularity/comments/1ibew93/deepseekr1\_optimizes\_the\_llamacpp\_wasm\_runtime\_by/
34. EricLBuehler/candle-vllm: Efficent platform for inference and serving local LLMs including an OpenAI compatible API server. \- GitHub, https://github.com/EricLBuehler/candle-vllm
35. Measuring and Reducing WebGPU Dispatch Overhead for LLM Inference \- OpenReview, https://openreview.net/pdf?id=TR7wZmeyXZ
36. 10 Transformer Inference Hacks for Faster TPS | by Modexa \- Medium, https://medium.com/@Modexa/10-transformer-inference-hacks-for-faster-tps-19f61358427e
37. 98× Faster LLM Routing Without a Dedicated GPU: Flash Attention, Prompt Compression, and Near-Streaming for the vLLM Semantic Router \- arXiv, https://arxiv.org/html/2603.12646v1
38. ruvllm \- crates.io: Rust Package Registry, https://crates.io/crates/ruvllm
39. AtomaAI/atoma-infer: Fast serverless LLM inference, in Rust. \- GitHub, https://github.com/atoma-network/atoma-infer
40. kv-cache-scheduler — Rust utility // Lib.rs, https://lib.rs/crates/kv-cache-scheduler
41. kv-cache-scheduler 0.1.0 on Cargo \- Libraries.io, https://libraries.io/cargo/kv-cache-scheduler
42. Blog \- vLLM, https://vllm.ai/blog
43. guoqingbao/xinfer: Blazing-fast LLM inference in pure Rust. No PyTorch and Python runtime. \- GitHub, https://github.com/guoqingbao/xinfer
44. slidingWindow() | Data Analysis 1.241.0-1.247.0 | LogScale Documentation, https://library.humio.com/data-analysis/functions-slidingwindow.html
45. LZ Penalty: An information-theoretic repetition penalty for autoregressive language models, https://arxiv.org/html/2504.20131v3
46. Logit Scaling: A General Method for Alignment in Microsimulation models, https://microsimulation.pub/download/aHR0cDovL3dlYjo4MDgyLzAwMTQ0L2lqbS0wMDE0NC5wZGY=/ijm-00144.pdf?\_hash=v3WQyUGg61wDrBcVMJF9S59jXLYnbpCqzs%2FM4KLFWJ0%3D
47. How do I log WASM heap memory usage from Rust? \- Stack Overflow, https://stackoverflow.com/questions/78246635/how-do-i-log-wasm-heap-memory-usage-from-rust
48. Tracing Large Memory Allocations in Rust with BPFtrace \- ReadySet.io, https://readyset.io/blog/tracing-large-memory-allocations-in-rust-with-bpftrace
49. rustwasm/wasm-tracing-allocator \- GitHub, https://github.com/rustwasm/wasm-tracing-allocator
50. System in std::alloc \- Rust, https://doc.rust-lang.org/std/alloc/struct.System.html