Extreme Model Compression and .slm Quantization for Browser-Local Inference

Full report

On this page

Update Ledger

• Verification Date: 2026-07-11 UTC
• Target Scope: 0.5B – 2B parameter instruction-tuned models for deterministic browser-local execution.
• Reference Architectures: TinyRustLM, GGUF/SLM1.
• Confidence Label: High. Evidence grounded in peer-reviewed literature, public GitHub repositories, and observed SLM1 format constraints up to July 2026\.

1. Executive Summary and Teleodynamic Context

The deployment of small language models (SLMs) in the 0.5B to 2B parameter range directly to the browser environment represents a fundamental shift in client-side computational intelligence. Operating entirely on local compute through WebAssembly (WASM) and WebGPU, these models circumvent server latency, preserve user privacy, and operate within strict memory and power budgets. However, bridging the gap between a trained FP32 or BF16 baseline and a highly compressed browser-local artifact demands extreme model compression—typically reducing parameters to an average of 2 to 4 bits. Traditional uniform quantization approaches, such as standard round-to-nearest (RTN) INT4, cause catastrophic degradation in sub-2B models. This degradation manifests as the destruction of conversational coherence, practical judgment, and lawful adult inquiry, as critical parameter distributions collapse into undifferentiated zero-bins1. Norm vectors, continuous recurrent states, attention projections, and outlier channels exhibit acute sensitivity to low-bit distortion3. Furthermore, the target deployment format—a black-box .slm container defined by TinyRustLM—imposes strict architectural, alignment, and metadata parsing constraints that preclude the use of non-standard runtime environments4. This report provides an exhaustive, expert-level analysis of extreme post-training quantization (PTQ) methodologies. It formulates a deterministic Rust conversion architecture, defines exact mathematical byte-boundaries for memory allocation, compares current state-of-the-art compression algorithms for browser suitability, and proposes three rigorous precision profiles. The objective is to achieve a pristine, structurally validated model artifact that maintains conversation parity while strictly adhering to the architectural limits of the WebGPU and WASM environments.

2. Browser Execution Environment and SLM1 Constraints

Before evaluating numerical compression algorithms, it is critical to map the mathematical constructs of model parameters to the physical and architectural boundaries of the browser runtime. Browser execution relies on WASM for CPU logic and WebGPU for hardware acceleration, both of which introduce unique memory and latency constraints. Historically, WASM was bound by a 4GB (32-bit) linear memory limit, which severely restricted the deployment of memory-intensive models5. The introduction of the Memory64 standard expands this theoretical limit to 16 exabytes, though modern browser engines typically clamp this allocation at 16GB6. While 16GB provides ample overhead for a 2B parameter model, the process of transferring a model from network storage to the GPU introduces severe memory duplication. Loading an artifact from a server or an Origin Private File System (OPFS) into a JavaScript ArrayBuffer, transferring it to WASM linear memory, and staging it to a GPUBuffer causes the peak memory to spike to nearly three times the artifact's size8. To mitigate this, modern implementations rely on queue.writeBuffer() for streaming data to the GPU in discrete chunks, which avoids the persistent intermediate staging buffers required by mapAsync() workflows10. Furthermore, WebGPU's security-focused design imposes per-operation validation and command buffer submission checks. These security measures introduce a per-dispatch overhead of [Figure omitted from source export] on Metal (Apple Silicon) and [Figure omitted from source export] on Vulkan, resulting in a total per-operation latency of approximately [Figure omitted from source export] when including framework overhead11. At a batch size of 1, a 32-layer transformer evaluating queries, keys, values, and outputs as separate dispatches incurs immense latency. Operator fusion—combining linear projections and element-wise additions into a single compute shader—is therefore mandatory to amortize this API cost9. This heavily biases the selection of quantization algorithms toward those that can be evaluated in dense, fused kernels rather than those requiring multiple sparse lookups or complex branching. These execution realities must conform to the observed evidence from the .slm v1 container format, which enforces a strict binary layout. The parser validates a 108-byte header starting with a 4-byte magic sequence (SLM1), followed by dimension metadata (vocabulary, layers, heads, etc.), floating-point values for RoPE theta and RMS epsilon, and a non-cryptographic checksum at offset 1004. Following the header, the format requires 64-byte tensor directory entries specifying the FNV-1a name hash, dtype, rank, dimensions, and data offsets. Crucially, the parser enforces that every tensor payload must be exactly 64-byte aligned4. If the exact shapes, magics, or alignment boundaries are breached, the deterministic parser rejects the model entirely, rendering unstructured sparsity or custom byte-packing schemes unviable unless they conform to the supported f32, q8\_0, or q4\_0 dtype contracts4.

3. Explicit Byte Formulas and Resource Allocation

Precise resource estimation is required to guarantee that a 0.5B-2B model fits within the browser's active execution budget. The formulation of these bytes dictates the acceptable precision levels for each tensor class. For weight tensors, let [Figure omitted from source export] represent a linear projection matrix. For uniform blockwise quantization (such as [Figure omitted from source export] or [Figure omitted from source export]) with a block size [Figure omitted from source export], the total number of blocks is defined as [Figure omitted from source export]. Each block requires a scaling factor, typically stored as a 16-bit float (2 bytes), followed by the packed integer data. For a 4-bit quantizer ([Figure omitted from source export]) with a block size of 32, the memory required is: [Figure omitted from source export] For an 8-bit quantizer ([Figure omitted from source export]) with the same block size: [Figure omitted from source export] Because the SLM1 format requires 64-byte alignment for all tensor payloads, the final stored byte count for any tensor is mathematically padded: [Figure omitted from source export] Embedding and output head bytes constitute a massive percentage of the parameter count in sub-2B models. Given a vocabulary size [Figure omitted from source export] and a hidden dimension [Figure omitted from source export], an untied architecture (separate embedding and output heads) kept in FP32 to preserve exact logit parity for the first token requires: [Figure omitted from source export] [Figure omitted from source export] If the architecture supports tied embeddings (indicated by bit 0 of the flags field at offset 16 in the SLM1 header), the output head matrix is eliminated, reducing overall model memory by up to [Figure omitted from source export] in extremely small models, though this requires the base model to have been explicitly trained with weight tying4. The KV cache footprint scales linearly with context length [Figure omitted from source export] and becomes the primary dynamic memory consumer during auto-regressive generation. For a model with [Figure omitted from source export] layers, [Figure omitted from source export] key-value heads, and head dimension [Figure omitted from source export], the KV cache must store both keys and values. Assuming a 2-bit quantization scheme with FP16 scales evaluated per group size [Figure omitted from source export]: [Figure omitted from source export] [Figure omitted from source export] [Figure omitted from source export] The SLM1 reference runtime avoids WASM memory fragmentation by allocating this maximum KV cache capacity statically ahead of time during the model-load transaction14. System overhead also includes forward-pass scratch buffers and logits arrays. According to the SLM1 runtime constraints, a single forward pass requires a scratch allocation to hold intermediate attention and feed-forward states, sized as [Figure omitted from source export] bytes, where [Figure omitted from source export] is the hidden dimension of the SwiGLU feed-forward network14. The logits buffer for a single token requires [Figure omitted from source export] bytes. Additional WASM module overhead, tokenizer manifesting, and sampler candidate arrays (strictly capped at 1,024 candidates) add a fixed overhead of approximately 5 MB14. Finally, for Peer-to-Peer (P2P) distribution, models must be chunked and hashed to allow deduplication and resumption. If a model of size [Figure omitted from source export] is divided into chunks of size [Figure omitted from source export] (e.g., 8 MiB) using Content-Defined Chunking (CDC), the overhead consists of SHA-256 hashes (32 bytes per chunk) and metadata manifests: [Figure omitted from source export] This minimal overhead allows browser clients to safely cache chunks in IndexedDB and verify structural integrity before initiating the expensive WebGPU materialization phase.

4. Literature Review: Core Quantization Methodologies

The mathematical strategies for compressing language models have evolved rapidly, navigating the trade-offs between activation outliers, computational throughput, and extreme low-bit representation. This section critically compares primary-source methods for their applicability to the 0.5B-2B browser inference regime.

4.1 Weight-Only Scaling and Activation-Aware Methods (GPTQ, AWQ, HQQ)

The foundational approach to post-training quantization involves determining optimal scaling factors and zero-points for blocks of weights. GPTQ utilizes approximate second-order information (the inverse Hessian matrix) to find per-column quantization scales, effectively compressing models to 3 or 4 bits with minimal perplexity degradation15. It operates layer-by-layer, compensating for the quantization error of one weight by adjusting the unquantized weights in the same row. While effective, GPTQ requires substantial calibration data (e.g., 128 samples of C4 or WikiText) and can sometimes over-fit to the calibration distribution, leading to subtle reasoning failures in smaller models. AWQ (Activation-aware Weight Quantization) improves upon this by recognizing that approximately 1% of weight channels are disproportionately salient, based on the magnitude of the activations that pass through them17. AWQ protects these salient channels via per-channel scaling, mitigating the need for backpropagation or complex inverse Hessian calculations. AWQ operates under MIT and Apache licenses18 and is highly deterministic. HQQ (Half-Quadratic Quantization) represents a shift toward calibration-free quantization. It uses an iterative half-quadratic splitting optimization process to find optimal scales and zero-points without requiring a calibration dataset, making it incredibly fast and completely immune to calibration over-fitting17. HQQ operates under an Apache 2.0 license. Hardware Suitability: All three methods produce standard dense affine integer weights (e.g., 4-bit integers with FP16 scales). The decoding operator is a simple linear dequantization step [Figure omitted from source export], which is easily vectorizable in WASM SIMD and highly parallelizable in WebGPU compute shaders without conditional branching. They are universally suitable for the SLM1 format constraints.

4.2 Outlier Migration and Distribution (SmoothQuant, OmniQuant, ZeroQuant)

Because activations frequently exhibit long-tailed outlier channels with values 10 to 100 times larger than the median, low-bit activation quantization is notoriously difficult. SmoothQuant is a training-free method that migrates the quantization difficulty from activations to weights. It computes a channel-wise scaling factor [Figure omitted from source export] offline, scaling down activation outliers while correspondingly scaling up the weights to maintain mathematical equivalence: [Figure omitted from source export]20. This enables W8A8 (8-bit weight, 8-bit activation) execution. OmniQuant extends this by introducing Learnable Weight Clipping (LWC) and Learnable Equivalent Transformation (LET), operating within a differentiable framework to optimize parameters via block-wise error minimization, allowing for W4A4 configurations24. ZeroQuant relies on token-wise activation quantization and layer-by-layer knowledge distillation to reduce precision21. Hardware Suitability: While SmoothQuant and OmniQuant elegantly solve the activation outlier problem, their reliance on dynamic activation quantization (W8A8 or W4A4) introduces runtime overhead in the browser. WebGPU currently lacks the highly optimized INT8/INT4 tensor core intrinsics available in proprietary CUDA environments (like cutlass or DP4a instructions across all backends). Consequently, for 0.5B-2B models where memory bandwidth—not pure compute—is the bottleneck, weight-only quantization (W4A16) is vastly preferred in WebGPU over weight-activation quantization.

4.3 Sparse-Dense Outlier Decomposition (SpQR, SqueezeLLM)

Rather than mathematically migrating outliers, these methods isolate them physically. SpQR (Sparse-Quantized Representation) and SqueezeLLM decompose the weight matrix into two distinct components: a dense, ultra-low-precision matrix (3–4 bit) containing 99% of the weights, and a highly sparse, high-precision matrix (FP16) containing the outlier weights1. SqueezeLLM further incorporates sensitivity-based non-uniform quantization based on second-order information2. Hardware Suitability: Poor for browser deployment. Sparse matrix multiplication relies on formats like Compressed Sparse Row (CSR), which require complex pointer chasing, conditional branching, and scattered memory accesses. While proprietary CUDA kernels can hide this latency via warp-level programming, WebGPU and WASM lack native sparse-hardware schedulers. The branch divergence severely penalizes WebGPU compute shaders, and the structural complexity violates the SLM1 format's requirement for contiguous 64-byte aligned dense tensor blocks.

4.4 Rotational Dispersion and Lattice Codebooks (QuaRot, SpinQuant, QuIP, QuIP#, AQLM)

For extreme sub-3-bit compression, scalar quantization fails because the step sizes become too large to capture the underlying distribution. QuaRot and SpinQuant apply orthogonal transformations (e.g., randomized Hadamard matrices) to the activations, weights, and KV caches28. By rotating the weight matrices, outliers are dispersed, yielding a sub-Gaussian distribution28. This computational invariance allows uniform 4-bit quantization across the entire model without the need for mixed-precision outlier retention. QuIP and QuIP\# build upon this incoherence processing. QuIP\# introduces a set of hardware-efficient multi-dimensional codebooks based on the highly symmetric [Figure omitted from source export] lattice, which achieves optimal 8-dimensional unit-ball packing32. This allows weights to be quantized as vectors rather than scalars, achieving viable 2-bit models. AQLM (Additive Quantization of Language Models) generalizes classic additive quantization by representing groups of weights as the sum of multiple vector codes, updating codebooks and scaling factors via joint optimization across layers35. Hardware Suitability: QuaRot's Hadamard rotations map cleanly to WebGPU compute shaders, as they rely on dense matrix arithmetic and fast Walsh-Hadamard transforms requiring [Figure omitted from source export] time. However, AQLM and QuIP\# require multi-codebook lookups during the decoding phase. In WebGPU, this necessitates placing codebooks in uniform or storage buffers, potentially causing cache thrashing in the limited shared memory of integrated mobile GPUs (typically 16-48 KB per workgroup37). While QuIP\#'s [Figure omitted from source export] lattice can be decoded algebraically with [Figure omitted from source export] instructions per weight34, integrating lattice decoders into the rigid SLM1 scalar runtime requires significant architectural extensions.

4.5 Low-Rank Quantization Error Correction (QLoRA, LoftQ)

To recover accuracy lost during extreme PTQ, methods like QLoRA and LoftQ introduce trainable low-rank adapters ([Figure omitted from source export])38. LoftQ optimally initializes these adapters via Singular Value Decomposition (SVD) of the quantization error40, avoiding the zero-initialization trap of standard LoRA. Hardware Suitability: High. The base weights remain in standard 4-bit storage. For WebGPU, executing the adapter dynamically ([Figure omitted from source export]) requires two additional small matrix multiplications per layer. If the adapter weights are small enough, they can be merged directly back into the quantized base weights post-training, though this alters the zero-points and scales, requiring re-calibration.

4.6 Method Comparison Matrix

5. KV Cache and Recurrent State Quantization

In browser environments, the KV cache footprint scales linearly and rapidly consumes the WebAssembly heap during long-context generation. Quantizing the KV cache to 2 or 3 bits is non-negotiable for serving 0.5B-2B models with [Figure omitted from source export] context windows. KIVI (Key-Value Cache Quantization) recognizes that key and value tensors exhibit different sensitivities. KIVI implements an asymmetric approach: Keys are quantized per-channel, while Values are quantized per-token44. This aligns perfectly with the streaming nature of auto-regressive generation, allowing newly generated tokens to be continuously appended to the buffer without recalculating channel-wide statistics. KIVI maintains robust quality at 2-bit quantization, allowing up to a 2.6x reduction in peak memory usage. TurboQuant, developed by Google DeepMind, achieves near-lossless 3-bit KV caching using a two-stage process. First, PolarQuant converts vectors to polar coordinates after a Hadamard rotation, bypassing block-normalization. Second, a 1-bit Quantized Johnson-Lindenstrauss (QJL) transform is applied to correct the bias in inner-product attention score estimations46. RotateKV employs explicit rotations on the key vectors before quantization, enabling extreme 2-bit quantization with less than 0.3 perplexity degradation on standard benchmarks49. Browser Implications: WebGPU compute shaders executing attention variants must dequantize the KV cache on the fly. TurboQuant's requirement of orthogonal rotation un-packing combined with QJL residual correction maps effectively to fused CUDA kernels, but in WebGPU WGSL, local caching of orthogonal projection matrices within the tight 16 KB shared memory limits may create a bottleneck37. KIVI's asymmetric design is considerably more memory-friendly for WASM/WebGPU, as it relies on simple scalar tracking and per-token appends, avoiding complex matrix inversions during the critical path of the decoding phase.

6. Tokenizer, Metadata, and Distribution Compression

The SLM1 format embeds the tokenizer—either BTOK or BPE1—directly into the artifact to prevent mismatch between the model weights and the vocabulary space4. To ensure the metadata remains safely inspectable, the JSON manifests and vocabulary arrays are stored uncompressed within the binary envelope, separated by explicit byte offsets. While applying Zstandard or Brotli compression to the entire .slm file reduces wire transfer size, it forces the browser to decompress the entire artifact into a massive secondary ArrayBuffer before parsing, exacerbating the memory duplication crisis. Instead, distribution relies on uncompressed, memory-mapped-friendly structures combined with Peer-to-Peer (P2P) chunking. Content-Defined Chunking (FastCDC) segments the artifact into addressable boundaries based on rolling hashes. If the user possesses an earlier version of the model or a different quantization variant sharing the exact same FP32 base layers, the browser can deduplicate identical chunks in IndexedDB. Immutable manifests signed with SHA-256 hashes guarantee that resumed downloads or stitched chunks strictly reproduce the original deterministic artifact, satisfying the teleodynamic requirement for verifiable evidence boundaries14.

7. Layer-Sensitivity and Tensor-Sensitivity Procedures

Achieving ultra-compact compression without destroying the model's common sense, practical judgment, or capacity for lawful adult inquiry requires more than uniform minimization. Catastrophic outlier loss destroys the semantic boundaries of safety training, leading to hallucination or refusal loops. A structured 9-step calibration and sensitivity suite must be executed prior to committing the .slm container. 1\. Source Tensor Range Audit & Outlier Mapping: A complete pass over the FP32/BF16 weights establishes a baseline. The algorithm calculates the kurtosis and max-to-mean ratios per channel. Tensors with a maximum value exceeding [Figure omitted from source export] of the layer mean are flagged as "outlier-dense." These tensors are protected from aggressive 3-bit quantization and routed to Q8\_0 or subjected to orthogonal dispersion (QuaRot). 2\. Calibration Set Design: Calibration datasets (typically 128–256 prompts) are constructed to encompass natural dialogue, instruction-following, and safety-bound inputs. Hard Exclusion Rule: No strings from target evaluation benchmarks (e.g., MMLU, GSM8K, MT-Bench) may be present in the calibration set. This guarantees that the quantization parameters do not over-fit the test distribution. 3\. First-Token Top-K Logit Parity: The fundamental test of structural integrity. A prompt is passed through the BF16 source model and the quantized candidate. The probability mass over the top-K logits (e.g., [Figure omitted from source export]) is calculated. The Kullback-Leibler (KL) divergence is computed: [Figure omitted from source export]. If the divergence exceeds [Figure omitted from source export], the structural validation fails, serving as a hard rejection criterion. 4\. Divergence Metrics (Teacher-Forced vs. Free-Generation): In Teacher-Forced testing, the original BF16 output sequence is fed to the quantized model, and per-token cross-entropy loss (perplexity) is measured. In Free-Generation, the quantized model generates autoregressively for 256 tokens. The cosine similarity of the final semantic embedding is measured against the BF16 trajectory. This evaluates compounding errors over time, ensuring the current-turn state remains coherent. 5\. Layer-by-Layer Replacement Test: Inference is executed where a single Layer [Figure omitted from source export] is quantized, but layers [Figure omitted from source export] remain in BF16. This isolates the specific quantization damage caused by [Figure omitted from source export]. Early layers (embedding and Layer 0\) and late layers (final norm and logit projection) consistently exhibit the highest sensitivity and are typically pinned to Q8\_0 or FP32. 6\. Quantized Adapter Merging vs. Separate Execution: If fine-tuning recovery was performed using QLoRA, merging the low-rank adapter back into the 4-bit weights mathematically alters the scales and zero-points. The KL divergence of the merged state is tested against executing the adapter dynamically ([Figure omitted from source export]). For WebGPU, merging is highly preferred to reduce the dispatch count, provided the divergence penalty remains under the threshold. 7\. Mixed-Profile Search: Using Integer Linear Programming (ILP) or a greedy solver, bits are allocated to each tensor to minimize the global divergence loss function subject to the hard artifact memory budget [Figure omitted from source export]. 8\. Corruption, Truncation, and Validation: Malformed scales (e.g., [Figure omitted from source export]) and extreme out-of-bounds tokens are injected into the quantized artifact. The Rust SLM1 parser must successfully trap the error, issue a bounded state reset, and return a stable error code without crashing the main browser thread or leaking WASM memory4. 9\. Behavior-Gate Comparison: A secondary suite of prompts tests practical judgment and lawful adult inquiry. If the quantized model collapses into repetitive loops or refuses benign prompts due to semantic blurring in the attention heads, the precision profile is rejected, regardless of its mathematical perplexity score. Smaller bytes do not compensate for broken conversation behavior.

8. Proposed Precision Profiles and Tensor Hierarchy

To balance hardware capabilities with the target size, three distinct precision profiles are formulated for an exemplary 1.5B parameter architecture (Hidden dimension [Figure omitted from source export], Vocabulary [Figure omitted from source export], Layers [Figure omitted from source export], FFN dimension [Figure omitted from source export]). Hard Rejection Criteria: Any profile that fails to parse structural headers, exceeds 128 MiB per contiguous tensor transfer, or deviates [Figure omitted from source export] on Top-5 retention against the BF16 baseline is immediately discarded.

Profile 1: Ultra-Compact (Target: ~950 MB)

Optimized for maximum compatibility on integrated graphics (Intel UHD, Adreno) with strict memory ceilings.

• Embeddings & Output Head: Q8\_0 (Block 32). High precision is required to preserve vocabulary distributions.
• Attention Q/K/V Projections: Q4\_0 (Block 32).
• Attention O (Output) Projections: Q4\_0 (Block 32).
• FFN Up/Gate/Down Projections: Q4\_0 (Block 32).
• Layer Norms & Biases: FP32. Crucial for statistical stability, with a negligible byte footprint.
• KV Cache: 2-bit Asymmetric (KIVI) to prevent OOM errors during long context generation.
• Transformation: QuaRot (Orthogonal rotation) applied pre-quantization to allow uniform Q4\_0 without catastrophic outlier collapse.

Profile 2: Balanced (Target: ~1.2 GB)

Optimized for M-series Apple Silicon and discrete mobile GPUs. Prioritizes reasoning over absolute size.

• Embeddings & Output Head: Q8\_0 (Block 32\) (Untied).
• Early/Late Layers (Layer 0, Layer 23): Elevated to Q8\_0 to preserve initial token extraction and final logit refinement.
• Attention Projections: Q4\_0 (Block 32).
• FFN Up/Gate: Q5\_0 (Block 32). Provides higher parameter capacity for factual recall.
• FFN Down: Q4\_0 (Block 32).
• Layer Norms: FP32.
• KV Cache: 3-bit (TurboQuant-style).

Profile 3: Quality (Target: ~1.6 GB)

Optimized for high-end local workstations where quality supersedes extreme loading times.

• Embeddings & Output Head: FP32. Zero quantization loss at the model boundaries.
• Attention & FFN Projections: Q6\_K or Q8\_0.
• Layer Norms: FP32.
• KV Cache: 4-bit (Standard uniform).
• Transformation: AWQ scaling to preserve the exact activation logic without modifying the underlying architecture.

9. Deterministic Rust Conversion Architecture

A robust, enterprise-grade Rust packing architecture—acting as the bridge between PyTorch/Safetensors checkpoints and the SLM1 container—must safely convert models without exposing proprietary logic. It relies on stable, open-source contracts and a deterministic state machine4. The converter employs a bounded streaming architecture. Instead of mapping the entire multi-gigabyte source file into memory, it utilizes an iterative read process. The source .safetensors file is memory-mapped, and tensors are yielded sequentially. An strict allowlist validation step ensures the tensor name matches a pre-defined architectural regex (e.g., ^model\\.layers\\.\\d+\\.mlp\\.down\_proj\\.weight$). Unknown tensors trigger an immediate abort, preventing malicious or malformed injections. Dimensional and range checks are enforced prior to quantization. The system verifies that the rank is [Figure omitted from source export], asserts that all FP32 values are finite (\!val.is\_nan() && \!val.is\_infinite()), and checks for integer overflows in shape multiplication. The selected quantization algorithm (such as HQQ or AWQ scale calculation) is then applied in memory. The packer writes the 64-byte directory entry to a staging manifest, pads the quantized payload to the strict 64-byte boundary required by SLM1, and streams the chunk directly to the output file using atomic writes. To support enterprise governance, the artifact is accompanied by an immutable JSON manifest. This manifest stores the original source hash (SHA-256 of the FP32 model), the exact quantization recipe, the layer sensitivity divergence scores, and cryptographic hashes of the calibration datasets used. If an interruption occurs during network distribution, the browser uses the manifest's Merkle tree to resume P2P chunk downloading, recovering narrowly without triggering an unnecessary reload of the entire artifact14.

10. Phased Implementation Plan and Conclusion

Moving from theoretical framework to hardened deployment requires a methodical approach:

• Phase 1: Deterministic Baseline. Implement the strict Safetensors-to-SLM1 Rust parser. Enforce the 108-byte header and 64-byte tensor alignment. Achieve 100% FP32 logit parity against the source model inside a local WASM loopback server4.
• Phase 2: Uniform Quantization and Verification. Introduce standard symmetric Q8\_0 and Q4\_0 blocks. Implement bounded streaming in the packer to prevent RAM exhaustion. Validate via KL divergence on the first token.
• Phase 3: Outlier Dispersion and Mixed Precision. Integrate AWQ or HQQ scaling into the packer. Update the WGSL shaders to support these layouts. Execute the layer-by-layer replacement test to establish the Balanced Profile.
• Phase 4: KV-Cache Compression. Implement asymmetric KV cache quantization (KIVI). Modify the WebGPU attention kernel to dequantize the cache within the threadgroup memory on the fly.
• Phase 5: Teleodynamic System Integration. Integrate the immutable manifests and peer-to-peer chunking logic into the worker-owned incremental runtime. Validate OPFS persistence and zero-copy ArrayBuffer transfers to eliminate peak memory spikes.

Extreme model compression for browser-local execution is constrained not by theoretical mathematics, but by the physical realities of memory bandwidth, WebGPU dispatch overhead, and deterministic structural parsing. Standard uniform quantization fails structurally and linguistically on 0.5B-2B models. By leveraging advanced post-training quantization methodologies—such as activation-aware scaling (AWQ), calibration-free half-quadratic splitting (HQQ), and asymmetric Key-Value caching (KIVI)—models can be reduced to an average of 3 to 4 bits per parameter while maintaining critical capabilities like conversational coherence and common-sense judgment. The proposed deterministic Rust architecture guarantees that these heavily optimized artifacts comply mathematically with the strict SLM1 container specifications, providing an unbroken, verifiable chain of cryptographic provenance from training weights to browser-local execution.

Works cited

1. SPQR: A SPARSE-QUANTIZED REPRESENTATION FOR NEAR-LOSSLESS LLM WEIGHT COMPRESSION \- ICLR Proceedings, https://proceedings.iclr.cc/paper\_files/paper/2024/file/1787533e171dcc8549cc2eb5a4840eec-Paper-Conference.pdf
2. SqueezeLLM: Dense-and-Sparse Quantization \- Stat.berkeley.edu, https://www.stat.berkeley.edu/\~mmahoney/pubs/icml2024\_kim24f.pdf
3. Awesome-LLM-Quantization/README.md at main \- GitHub, https://github.com/pprp/Awesome-LLM-Quantization/blob/main/README.md
4. Implementation \- MiRust, https://mirust.com/implementation/
5. WebAssembly.Memory() constructor \- MDN Web Docs \- Mozilla, https://developer.mozilla.org/en-US/docs/WebAssembly/Reference/JavaScript\_interface/Memory/Memory
6. WebAssembly in 2026: From Browsers to Edge Computing and Beyond | Zylos Research, https://zylos.ai/research/2026-02-05-webassembly-ecosystem-2026/
7. Wasm 3.0 Completed \- WebAssembly, https://webassembly.org/news/2025-09-17-wasm-3.0/
8. Discover | web.dev, https://web.dev/discover
9. 7 WebGPU \+ WASM Plays for Private On-Device ML | by Bhagya Rana | Medium, https://medium.com/@bhagyarana80/7-webgpu-wasm-plays-for-private-on-device-ml-a114bbba8c58
10. GPUQueue: writeBuffer() method \- Web APIs | MDN, https://developer.mozilla.org/en-US/docs/Web/API/GPUQueue/writeBuffer
11. Characterizing WebGPU Dispatch Overhead for LLM Inference Across Four GPU Vendors, Three Backends, and Three Browsers \- arXiv, https://arxiv.org/html/2604.02344v1
12. GetMappedRange doesn't feel very intuitive and requires a complex tracking logic in the backend \#4901 \- GitHub, https://github.com/gpuweb/gpuweb/discussions/4901
13. (PDF) Characterizing WebGPU Dispatch Overhead for LLM Inference Across Four GPU Vendors, Three Backends, and Three Browsers \- ResearchGate, https://www.researchgate.net/publication/403529940\_Characterizing\_WebGPU\_Dispatch\_Overhead\_for\_LLM\_Inference\_Across\_Four\_GPU\_Vendors\_Three\_Backends\_and\_Three\_Browsers
14. Implementation operations \- MiRust, https://mirust.com/implementation-operations/
15. Extreme Compression of Large Language Models via Additive Quantization \- arXiv, https://arxiv.org/html/2401.06118v2
16. SEPTQ: A Simple and Effective Post-Training Quantization Paradigm for Large Language Models \- arXiv, https://arxiv.org/html/2604.10091v1
17. MCAP: Deployment-Time Layer Profiling for Memory-Constrained LLM Inference \- arXiv, https://arxiv.org/html/2604.21026v1
18. LICENSE · TrueCourse/Qwen3.6-35B-AWQ at main \- Hugging Face, https://huggingface.co/TrueCourse/Qwen3.6-35B-AWQ/blob/main/LICENSE
19. dropbox/hqq: Official implementation of Half-Quadratic Quantization (HQQ) \- GitHub, https://github.com/dropbox/hqq
20. SmoothQuant: Efficient PTQ for Large Models \- Emergent Mind, https://www.emergentmind.com/topics/smoothquant
21. GitHub \- aiha-lab/qllm-infer: Quantization Framework for LLM Inferences, https://github.com/aiha-lab/qllm-infer
22. \[2211.10438\] SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models \- arXiv, https://arxiv.org/abs/2211.10438
23. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models, https://par.nsf.gov/servlets/purl/10467173
24. OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models \- arXiv, https://arxiv.org/html/2308.13137v3
25. LLM Quantization, https://www.cse.iitd.ac.in/\~rijurekha/col851/quantization.pdf
26. SqueezeLLM: Dense-and-Sparse Quantization \- arXiv, https://arxiv.org/html/2306.07629v1
27. \[2306.07629\] SqueezeLLM: Dense-and-Sparse Quantization \- ar5iv, https://ar5iv.labs.arxiv.org/html/2306.07629
28. QuaRot: Orthogonal Rotation in LLM Quantization \- Emergent Mind, https://www.emergentmind.com/topics/quarot
29. SpinQuant: LLM Quantization with Learned Rotations \- OpenReview, https://openreview.net/forum?id=ogO6DGE6FZ
30. SPINQUANT: LLM QUANTIZATION WITH LEARNED ROTATIONS \- ICLR Proceedings, https://proceedings.iclr.cc/paper\_files/paper/2025/file/e5b1c0d4866f72393c522c8a00eed4eb-Paper-Conference.pdf
31. KurTail : Kurtosis-based LLM Quantization \- ACL Anthology, https://aclanthology.org/2025.findings-emnlp.943.pdf
32. QuIP\#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks \- PMC, https://pmc.ncbi.nlm.nih.gov/articles/PMC12395268/
33. QuIP\#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks \[Quick Review\] \- Liner, https://liner.com/review/quip-even-better-llm-quantization-with-hadamard-incoherence-and-lattice
34. QuIP\# Even Better LLM Quantization with Hadamard, https://icml.cc/media/icml-2024/Slides/34816.pdf
35. The AQLM Quantization Algorithm, Explained | by Pierre Lienhart | TDS Archive \- Medium, https://medium.com/data-science/the-aqlm-quantization-algorithm-explained-8cf33e4a783e
36. Yet another state of the art in LLM quantization : r/LocalLLaMA \- Reddit, https://www.reddit.com/r/LocalLLaMA/comments/1al58xw/yet\_another\_state\_of\_the\_art\_in\_llm\_quantization/
37. WebGPU Memory Limits: maxStorageBufferBindingSize \- Ayoob AI, https://ayoob.ai/blog/webgpu-maxstoragebufferbindingsize-limits-enterprise
38. Quantized Low-Rank Adaptation (QLoRA) \- Emergent Mind, https://www.emergentmind.com/topics/quantized-low-rank-adaptation-qlora
39. LQ-LORA: LOW-RANK PLUS QUANTIZED MATRIX DECOMPOSITION FOR EFFICIENT LANGUAGE MODEL FINETUNING, https://par.nsf.gov/servlets/purl/10512534
40. QERA: an Analytical Framework for Quantization Error Reconstruction \- OpenReview, https://openreview.net/forum?id=LB5cKhgOTu¬eId=KkgmTCe65a
41. SqueezeLLM/LICENSE at main \- GitHub, https://github.com/SqueezeAILab/SqueezeLLM/blob/main/LICENSE
42. QuIP\#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks, https://huggingface.co/papers/2402.04396
43. GitHub \- alvinreal/awesome-opensource-ai: Curated list of the best truly open-source AI projects, models, tools, and infrastructure. Daily updated., https://github.com/alvinreal/awesome-opensource-ai
44. KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache \- GitHub, https://raw.githubusercontent.com/mlresearch/v235/main/assets/liu24bz/liu24bz.pdf
45. \[ICML 2024\] KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache \- GitHub, https://github.com/jy-yuan/KIVI
46. Google's TurboQuant reduces AI LLM cache memory capacity requirements by at least six times — up to 8x performance boost on Nvidia H100 GPUs, compresses KV caches to 3 bits with no accuracy loss | Tom's Hardware, https://www.tomshardware.com/tech-industry/artificial-intelligence/googles-turboquant-compresses-llm-kv-caches-to-3-bits-with-no-accuracy-loss
47. TurboQuant: 31GB AI Memory Down to 4GB, Explained | TECHSY, https://techsy.io/en/blog/google-turboquant-ai-memory-compression
48. What Is Google TurboQuant and What Does It Mean for Open Source Inference? \- Deep Infra, https://deepinfra.com/blog/google-turboquant
49. RotateKV: Accurate and Robust 2-Bit KV Cache Quantization for LLMs via Outlier-Aware Adaptive Rotations \- IJCAI, https://www.ijcai.org/proceedings/2025/0690.pdf
50. Tiny Model Composer \- MiRust, https://mirust.com/composer/
51. Documentation \- MiRust, https://mirust.com/docs/
52. Guides \- MiRust, https://mirust.com/guides/