Extreme Quantization Strategies for TinyRustLM: Architecting the SLM1 Format for Browser-Native Inference

Full report

On this page

The deployment of Small Language Models (SLMs) in resource-constrained, browser-native environments represents a convergence of two strict engineering limitations: the memory bandwidth wall of host devices and the execution constraints of WebAssembly (WASM). As systems like TinyRustLM push inference directly to the client via scalar and Single Instruction, Multiple Data (SIMD) CPU execution, the underlying memory footprint becomes the primary latency bottleneck. Models ranging from 0.5 billion to 7 billion parameters must be heavily compressed to fit within tight operational ceilings, such as a 128 MiB transfer boundary or a 1 to 2 GB resident memory limit1. This report provides an exhaustive analysis of extreme quantization strategies, evaluating format trade-offs, outlier-aware compression architectures, and optimal Rust WASM decoding mechanics. Furthermore, it delivers a comprehensive proposal for upgrading the proprietary .slm format to support advanced quantization, ensuring that TinyRustLM can execute 4-bit and mixed-precision models with minimal perplexity degradation and maximum throughput.

1. The Quantization Spectrum and Signal-to-Noise Dynamics

Quantization reduces the numerical precision of a model's parameters, shrinking the storage requirement from 16-bit floating-point (FP16 or BF16) formats down to 8-bit, 4-bit, or even 2-bit integer representations. While massive parameter models (e.g., 70B parameters and above) possess enough redundant capacity to absorb quantization noise smoothly, SLMs exhibit extreme sensitivity to precision loss4. The fundamental metric defining the fidelity of a quantized signal is the Signal-to-Quantization-Noise Ratio (SQNR). For an [Figure omitted from source export]\-bit uniform quantizer, the SQNR is mathematically approximated as: [Figure omitted from source export] This formulation demonstrates that each bit removed from the parameter representation halves the number of discrete levels and doubles the step size, thereby increasing the quantization noise power by approximately 6 dB6. In the context of causal language modeling, this reduction in signal fidelity manifests as an increase in perplexity—a measurement of the model's uncertainty in predicting the next token. The theoretical bounds of this compression have been rigorously studied, with findings indicating an "expressive collapse" at 1.58 bits per weight (a ternary representation). At this threshold, the expressive capacity of weight-quantized models degrades polynomially, and the universal approximation properties of the attention mechanism begin to fail9. Consequently, identifying the optimal quantization bit-width is a delicate exercise in balancing signal retention against memory reduction.

The analysis indicates a sharp Pareto frontier around the 4-bit mark. At 4-bit precision, models retain the vast majority of their conversational and structural abilities. Below 4 bits, SLMs suffer catastrophic skill loss because the step size between quantization bins becomes too large to accurately represent the varied magnitudes of the weights5. Therefore, 4-bit precision, augmented by strategic mixed-precision layering, serves as the theoretical and practical minimum for reliable SLM deployment in the browser.

2. Granularity: From Per-Tensor to Groupwise Scaling

The granularity at which quantization scales and zero-points are computed profoundly impacts the resulting SQNR. Early quantization methods applied a single global scale across an entire weight tensor. However, modern approaches recognize that the dynamic range of weights varies wildly across different dimensions of a neural network4.

2.1 Per-Channel and Groupwise Quantization

Moving from per-tensor to per-channel quantization—where each output channel (or row) of a weight matrix maintains its own scale—significantly tightens the quantization bins around the actual weight distribution. However, given the massive width of transformer layers, even per-channel scaling suffers from localized outliers. The industry standard has thus shifted to groupwise (or block-wise) quantization. In this paradigm, consecutive weights within a channel are divided into small, fixed-size blocks (e.g., 32, 64, or 128 weights). Each block independently tracks its own scale and zero-point metadata13. This localized scaling ensures that a single high-magnitude weight only distorts the precision of the few neighboring weights within its immediate block, rather than devastating an entire channel. Determining the optimal group size is critical. Research indicates a non-monotonic relationship in group sizes; reducing the group size (thereby increasing the total number of groups and the metadata overhead) does not uniformly improve performance. Evolutionary search strategies applied to models like Diffusion Transformers (DiTs) demonstrate that varying group sizes across different layers—such as assigning 64 to highly sensitive layers and 128 to robust layers—yields superior Fréchet Inception Distance (FID) and perplexity scores compared to a uniform configuration18. For SLMs, where parameter capacity is highly restricted, maintaining a group size of 32 or 64 is generally optimal for minimizing quantization noise13.

2.2 Hierarchical Block Layouts (K-Quants)

To maximize the efficiency of 4-bit representation while mitigating the storage overhead of dense block metadata, the GGML ecosystem pioneered "K-quants." K-quants nest standard blocks into larger "super-blocks"21. A standard super-block in the Q4\_K layout encompasses 256 weights, divided into 8 sub-blocks of 32 weights. The super-block maintains high-precision scales (typically FP16), while the 8 sub-blocks store heavily quantized 6-bit integer scales relative to the super-block's FP16 baseline12. The exact memory layout of the Q4\_K format is designed to align with memory access patterns: [Figure omitted from source export] This structural hierarchy achieves superior SQNR for the same average bit-width compared to legacy linear quantization formats (like the original Q4\_0, which lacked super-blocks)21.

3. Addressing the Outlier Dilemma

Standard block quantization relies on simple round-to-nearest (RTN) arithmetic. However, LLMs naturally develop systematic "outliers"—a tiny fraction of dimensions (often less than 0.1%) that spontaneously grow to magnitudes 50 to 100 times larger than the remaining network23. These outliers are not random noise; they consistently appear in specific channels across every input token, acting as critical feature detectors. If these outliers are subjected to standard RTN quantization, the block's scale factor becomes dominated by the massive outlier value. Consequently, the surrounding normal weights are squashed into a tiny fraction of the available integer grid, losing almost all their precision and functionally destroying the layer's learned representations23.

3.1 Sparse-Quantized Representations (SpQR and SqueezeLLM)

One architectural solution to the outlier problem is structural isolation, championed by frameworks such as Sparse-Quantized Representation (SpQR) and SqueezeLLM. These frameworks identify sensitive outlier weights based on second-order Hessian sensitivity and explicitly extract them from the primary weight matrix24. The weights are decomposed into a Dense-and-Sparse format:

1. Dense Matrix: The remaining 99.9% of the weights, now stripped of extreme outliers, exhibit a tight, normalized distribution. They are grouped and quantized to 3 or 4 bits with incredibly high fidelity because the quantization grid is no longer stretched.
2. Sparse Matrix: The extracted outliers are stored in full FP16 precision using a Compressed Sparse Row (CSR) or Padded Transposed CSR (PTCSR) format25.

During inference, the dense quantized matrix multiplication is computed, and the sparse FP16 matrix multiplication is added to the result. While this preserves near-lossless performance at sub-4-bit levels, CSR sparse-dense matrix multiplication poses significant branching and latency penalties on CPU and WASM targets. Hardware prefetchers and SIMD pipelines struggle with the irregular, non-contiguous memory access patterns inherent in CSR layouts, resulting in cache misses that counteract the theoretical speedups27. Therefore, for browser-based SLM deployments utilizing WASM, pure sparse-dense splitting is suboptimal.

3.2 Mixed-Precision Layering

Rather than isolating individual weights, an alternative approach is to isolate entire layers or channels. The Outlier-aware Weight Quantization (OWQ) method and mixed-precision K-quants (such as Q4\_K\_M) apply varying bit-widths across the network29. The Q4\_K\_M format dynamically preserves weights in highly sensitive layers—such as early attention layers, token embeddings, or output projections—at 6-bit precision, while heavily compressing denser, more robust feed-forward network (FFN) blocks to 4-bit11. By targeting precision specifically where activation magnitudes and outlier densities are highest, mixed-precision formats achieve a superior trade-off between model footprint and perplexity31.

4. Activation-Aware Weight Quantization (AWQ)

To resolve the outlier dilemma without the computational overhead of sparse CSR layouts, Activation-Aware Weight Quantization (AWQ) has emerged as the state-of-the-art paradigm. AWQ strictly maintains a dense matrix format, making it vastly superior for SIMD-accelerated local inference33. The fundamental insight of AWQ is that the importance of a weight is proportional to the magnitude of the activation it multiplies against. Quantization error on high-activation weights dominates the final layer-output error. Therefore, AWQ shifts the quantization difficulty upstream by manipulating the mathematical identity of linear projections35. Given an input activation [Figure omitted from source export] and a weight matrix [Figure omitted from source export], the layer output is [Figure omitted from source export]. AWQ introduces a per-channel scaling vector [Figure omitted from source export]: [Figure omitted from source export] Before quantization, the weight matrix is multiplied by the inverse scale [Figure omitted from source export]. Channels corresponding to massive activation outliers are scaled down in the weight matrix. This reduces their magnitude relative to non-outlier channels, yielding a compressed weight distribution that fits tightly into a uniform 4-bit grid without squashing the normal weights35. During inference, the input activations are scaled up by the vector [Figure omitted from source export] prior to the matrix multiplication to mathematically compensate.

4.1 Grid Search for Optimal Scaling

AWQ does not derive the scaling vector [Figure omitted from source export] analytically. Instead, it relies on a grid search over an exponent [Figure omitted from source export] acting on the mean activation magnitude [Figure omitted from source export]: [Figure omitted from source export] The algorithm sweeps [Figure omitted from source export] across a discrete range (typically [Figure omitted from source export] in steps of 0.05). For each candidate [Figure omitted from source export], the weights are inverse-scaled, quantized, and dequantized. The resulting output error against a small calibration dataset (e.g., 128 samples) is measured. The [Figure omitted from source export] that minimizes the mean squared error of the layer's output is permanently baked into the model35. Because the compensating scale factor [Figure omitted from source export] can be fused directly into the pre-existing normalization layers (such as RMSNorm or LayerNorm) that precede the linear projection, AWQ introduces absolutely zero computational overhead at runtime. The matrix remains purely dense and block-quantized, perfectly compatible with WebAssembly execution pipelines35.

4.2 AWQ vs. GPTQ for Small Language Models

Generative Pre-trained Transformer Quantization (GPTQ) is another dominant methodology that compensates for quantization error by iteratively updating unquantized weights layer-by-layer using the inverse Hessian matrix33. However, GPTQ operates exclusively on weight sensitivity and is functionally blind to activation magnitudes. For SLMs (like TinyLlama 1.1B or Qwen 1.5B), representation capacity is brittle. Empirical benchmarks demonstrate that AWQ preserves the reasoning capacity of instruction-tuned SLMs noticeably better than GPTQ at identical 4-bit sizes34. AWQ's activation-aware scaling cuts the INT4 quantization penalty drastically, outperforming GPTQ on logical reasoning benchmarks while maintaining identical latency profiles42. AWQ is therefore the recommended algorithmic foundation for processing .slm artifacts prior to packaging.

5. Quantization-Aware Fine-Tuning (QAT)

When compressing SLMs to extreme low-bit regimes (e.g., tight 4-bit or 3-bit), post-training quantization (PTQ) techniques like AWQ may still exhibit residual degradation. To fully bridge the accuracy gap, Quantization-Aware Training (QAT) integrates simulated quantization noise directly into the fine-tuning process, forcing the model to adapt its parameters to the reduced precision grid43.

5.1 The Initialization Problem in QLoRA

The contemporary standard for parameter-efficient fine-tuning is QLoRA, which freezes the base model in a 4-bit format (such as NormalFloat4) and attaches 16-bit Low-Rank Adaptation (LoRA) adapters. The modified weight formulation is: [Figure omitted from source export] where [Figure omitted from source export] and [Figure omitted from source export] are low-rank matrices, [Figure omitted from source export] is the rank, and [Figure omitted from source export] is a scaling factor46. However, QLoRA suffers from a critical initialization flaw. Standard LoRA initializes matrix [Figure omitted from source export] with zeros, meaning the adapters initially contribute nothing. Because the base weights are already degraded by quantization noise, the adapters must waste their highly limited parameter budget acting as error-correction mechanisms for the base matrix, rather than learning new semantic features48.

5.2 LoftQ: Optimizing the Quantization Matrix

To maximize the inference performance for .slm payloads, models should instead be fine-tuned using LoftQ (LoRA-Fine-Tuning-Aware Quantization). LoftQ addresses the initialization discrepancy by jointly optimizing the quantized base matrix [Figure omitted from source export] and the low-rank adapters [Figure omitted from source export] and [Figure omitted from source export] before training even begins. The objective is to minimize the Frobenius norm of the difference between the original high-precision pretrained weight [Figure omitted from source export] and the new adapted structure43: [Figure omitted from source export] By utilizing Singular Value Decomposition (SVD) to alternatingly project the quantization error into the low-rank matrices, LoftQ ensures that the initial state of the network is mathematically indistinguishable from the original unquantized model51. When fine-tuning completes, the adapters can be losslessly merged back into the quantized base matrix (or merged using specialized quantization-aware methods like QA-LoRA or L4Q). This produces a highly accurate, fully integer-quantized model ready for direct browser deployment, completely eliminating the runtime latency penalty associated with computing separate mixed-precision adapter pathways44.

6. WASM SIMD, Relaxed Math, and Execution Mechanics

A profound anti-pattern in early browser-based AI frameworks was the "dequantize-to-cache" execution flow. Frameworks would read a 4-bit block from memory, expand it into a full 32-bit floating-point (f32) vector, cache it in a dense memory arena, and then perform a standard f32 matrix-vector multiplication56. Because memory bandwidth—not compute capability—is the primary bottleneck for batch-size-1 autoregressive decoding on CPUs, this strategy is fatal. Expanding 4-bit weights to 32-bit floats demands moving 4 bytes per element across the memory bus instead of 0.5 bytes, resulting in a massive 4x to 8x DRAM bandwidth penalty and devastating the tokens-per-second throughput33. Modern Rust WASM execution engines must perform fused dequantization and dot products entirely within the CPU's vector registers, leveraging WebAssembly SIMD without touching intermediate RAM.

6.1 WebAssembly SIMD128 and v128 Intrinsics

The WebAssembly specification provides the v128 vector type, exposing a 128-bit register capable of processing multiple data lanes simultaneously (e.g., 16 i8 integers or 4 f32 floats). In Rust, these are accessed via the core::arch::wasm32 module57. When decoding a Q4\_K layout, the runtime utilizes a single v128\_load instruction to read 16 bytes directly from the quantized memory buffer (fetching 32 packed 4-bit weights instantly). Using bitwise SIMD instructions (v128\_and, u8x16\_shr), the runtime isolates the lower and upper nibbles into two separate integer vectors. These vectors are subsequently multiplied against the incoming activation vectors, accumulating the sums directly in the registers57.

6.2 The Relaxed SIMD Paradigm

While standard SIMD128 guarantees bit-for-bit deterministic results across all hardware architectures, it lacks certain advanced arithmetic operations. A critical advancement for browser-based inference is the WebAssembly "Relaxed SIMD" proposal, accessed in Rust via the target\_feature \= "relaxed-simd" flag57. Relaxed SIMD intentionally sacrifices strict cross-platform determinism to map operations directly to the host CPU's native fused-multiply-add hardware (e.g., VPDPBUSD on Intel AVX512 VNNI, or sdot on ARM NEON architectures)59. For the highest-performance inner loop, the intrinsic i32x4\_relaxed\_dot\_i8x16\_i7x16\_add is utilized61. This instruction operates as follows:

1. It consumes an i8x16 signed activation vector.
2. It consumes an i7x16 unsigned weight vector (representing the extracted 4-bit weights).
3. It computes the dot product of the 16 elements.
4. It adds the result to an existing i32x4 accumulator vector, all within a single CPU cycle61.

By keeping the inner multiplication strictly within the integer domain and only multiplying the final i32 accumulator by the floating-point block scale, the system minimizes expensive float conversions. This "zero-copy" fused SIMD pipeline moves 3.76x less data through the memory bus compared to f32 inference, directly yielding up to a 3.0x speedup on memory-bound autoregressive decoding56.

7. Architectural Proposal: The SLM1 Quantization Framework

Synthesizing the demands of outlier resilience, memory bandwidth conservation, and WASM vector alignment, the TinyRustLM execution engine must adopt a rigorous format layout. We propose integrating SLM\_AWQ4\_K (Activation-Aware 4-bit K-quant) into the proprietary .slm container.

7.1 Byte Layout and Memory Design

The .slm file format relies on a strict 108-byte header and 64-byte tensor directory66. The 108-byte header must be augmented to encode quantization parameters explicitly:

• Bytes 0-3: Magic number (SLM1).
• Bytes 4-7: Version (e.g., 0x00000002).
• Bytes 8-11: Quantization ID (0x00000004 mapped to AWQ4\_K).
• Bytes 12-15: Teleodynamic routing flags (defining the model's intent role, e.g., router, summarizer, or verifier)68.
• Bytes 16-107: Vocabulary size, dimensions, maximum context length, and a non-cryptographic checksum (such as FNV-1a or CRC32) for load-time integrity validation67.

To facilitate v128 aligned loads without suffering from CPU striding or unaligned access penalties65, the weight blocks must be explicitly structured to match the boundaries of the SIMD lanes. The proposed SuperBlockAWQ4\_K structure encompasses 256 weights, organized hierarchically:

Rust \#\[repr(C, align(32))\] pub struct SuperBlockAWQ4\_K { // 16 bytes: 8 localized float scales (one for each sub-block) pub scales: \[f16; 8\],

// 128 bytes: 8 blocks of 32 4-bit weights packed into 16 bytes each pub sub\_blocks: \[SubBlock; 8\], }

\#\[repr(C)\] pub struct SubBlock { pub qs: \[u8; 16\], // 32 weights packed as 4-bit nibbles }

Total Size: 144 bytes per 256 weights.Bits per Weight (bpw): [Figure omitted from source export]. This layout achieves the optimal 4.5 bpw target11. Crucially, by storing the 8 f16 scales consecutively at the start of the super-block, a single v128\_load instruction can pull all scaling metadata into registers simultaneously, eliminating costly pointer arithmetic during the hot loop execution.

7.2 Expected Memory Savings and Quality Projection

Applying this architecture to a representative 1.5 billion parameter SLM (e.g., Qwen 1.5B or TinyLlama):

• FP16 Baseline Memory: \~3.0 GB resident RAM.
• SLM\_AWQ4\_K Memory: \~840 MB resident RAM.

Quality Projection: Because the framework employs Activation-Aware Weight Quantization (AWQ) parameter pre-scaling combined with LoftQ fine-tuning adaptation, perplexity loss is strictly bounded. Empirical models indicate that language generation tasks will experience less than a 2% degradation in coherence compared to the FP16 baseline. Furthermore, an 840 MB footprint comfortably satisfies browser IndexedDB cache limits and avoids triggering WebAssembly's punitive garbage collection pauses during large allocation spikes65.

7.3 Runtime Decode Strategy

The tinyrustlm-runtime must implement a dynamic execution branch capable of downgrading gracefully if advanced hardware features are missing65.

1. Feature Detection: During WASM initialization, the runtime must invoke WebAssembly.validate() to confirm the presence of relaxed-simd within the host browser57.
2. Kernel Dispatch:

• Path A (Optimal): If relaxed-simd is enabled, loop over SuperBlockAWQ4\_K, extract the packed weights into v128 registers, and execute i32x4\_relaxed\_dot\_i8x16\_i7x16\_add against dynamically scaled integer activations. Multiply the integer sum by the extracted f16 scale.
• Path B (Fallback): If only standard simd128 is available, the runtime must unpack the 4-bit nibbles using u8x16\_and and u8x16\_shr. The integers are converted to f32x4 via i32x4\_trunc\_sat\_f32x4, followed by standard f32x4\_mul and f32x4\_add math58.

1. Workspace Buffers: Forward scratch spaces (approximated at 48 MiB) must be allocated once sequentially as a contiguous memory block and reused iteratively. This design eliminates runtime allocation latency and suppresses OS-level paging delays68.

7.4 Validation Protocols for Quantized Tensors

Because WASM lacks the deep inspection and debugging tools natively available to C++ binaries, evaluating quantization fidelity requires rigorous property-based testing and error-bound checking prior to runtime promotion67:

1. Deterministic RMSE Validation: A scalar FP32 test harness must generate a matrix of pseudo-random weights, compress them to the AWQ4\_K layout using the Rust encoder, and decode them via the active SIMD path. The Root Mean Square Error (RMSE) must be calculated. If the RMSE exceeds a pre-defined tolerance threshold (e.g., \> 0.05), the CI/CD pipeline must reject the artifact.
2. Alignment Assertion: The Rust implementation must enforce strict alignment via macros: static\_assert(std::mem::size\_of::\<SuperBlockAWQ4\_K\>() \== 144\) and std::mem::align\_of must guarantee 32-byte alignment. This explicitly prevents cache-line straddling and unaligned vector loads, which would severely degrade SIMD performance65.
3. Outlier Resistance Testing: An automated test must inject a synthetic super-outlier channel (with [Figure omitted from source export] baseline variance) into the validation matrix. The pipeline must verify that the AWQ scaling correctly adjusts the activation vectors and scaling metadata, ensuring that the non-outlier channels maintain their sub-block distributions without collapsing to zero.

8. Conclusion

Achieving robust, high-performance machine intelligence locally within a browser environment mandates uncompromising architectural precision. The proposed evolution for the TinyRustLM ecosystem establishes a mathematically sound paradigm for overcoming the WebAssembly memory bandwidth wall. By integrating Activation-Aware Weight Quantization (AWQ) for outlier resilience, LoftQ for optimal low-rank error recovery, and WebAssembly Relaxed SIMD for zero-copy dot products, the system mathematically circumvents the latency penalties of standard execution. The final SLM\_AWQ4\_K layout, structured at exactly 4.5 bits-per-weight with strict 32-byte vector alignment, confines highly capable 1.5B parameter SLMs to an \~840 MB memory footprint. This architecture enables local, private, and deterministic generation natively on the client, unlocking the true potential of edge-scale cognitive systems.

Works cited

1. Implementation operations \- MiRust, https://mirust.com/implementation-operations/
2. From SLM Fundamentals to webSLM: A Practical Path to Domain-Specific Browser AI, https://dev.to/vishalmysore/from-slm-fundamentals-to-webslm-a-practical-path-to-domain-specific-browser-ai-5hla
3. Small Language Models (SLMs): Comprehensive Guide 2026 \- CogitX, https://cogitx.ai/blog/small-language-models-slms-comprehensive-guide-2026
4. The Complete Guide to LLM Quantization \- LocalLLM.in, https://localllm.in/blog/quantization-explained
5. how much Quantization decrease model's capability? : r/LocalLLaMA \- Reddit, https://www.reddit.com/r/LocalLLaMA/comments/1ja3vjf/how\_much\_quantization\_decrease\_models\_capability/
6. https://grokipedia.com/page/Signal-to-quantization-noise\_ratio\#:\~:text=SQNR%3D10log10(A2%2F(3,g10(4n).\&text=SQNR%20%E2%89%88%206.02%20n%20%2B%201.76%20dB%20.
7. Signal-to-quantization-noise ratio \- Grokipedia, https://grokipedia.com/page/Signal-to-quantization-noise\_ratio
8. A Scaling Law for Bandwidth Under Quantization \- arXiv, https://arxiv.org/html/2602.23252v1
9. On the Expressive Power of Weight Quantization in Large Language Models \- arXiv, https://arxiv.org/pdf/2606.22249
10. BiLLM achieving for the first time high-accuracy inference (e.g. 8.41 perplexity on LLaMA2-70B) with only 1.08-bit weights across various LLMs families and evaluation metrics, outperforms SOTA quantization methods of LLM by significant \- Reddit, https://www.reddit.com/r/LocalLLaMA/comments/1am9v02/billm\_achieving\_for\_the\_first\_time\_highaccuracy/
11. LLM Quantization Explained: Q4\_K\_M vs Q4\_0 vs Q8\_0 (2026) \- PromptQuorum, https://www.promptquorum.com/local-llms/llm-quantization-explained
12. GGUF · Hugging Face, https://huggingface.co/docs/hub/gguf
13. Which Quantization Should I Use? A Unified Evaluation of llama.cpp Quantization on Llama-3.1-8B-Instruct \- arXiv, https://arxiv.org/html/2601.14277v1
14. Understanding AI/LLM Quantisation Through Interactive Visualisations \- smcleod.net, https://smcleod.net/2024/07/understanding-ai/llm-quantisation-through-interactive-visualisations/
15. Analysis of size-perplexity tradeoff for k-quantization : r/LocalLLaMA \- Reddit, https://www.reddit.com/r/LocalLLaMA/comments/145tf00/analysis\_of\_sizeperplexity\_tradeoff\_for/
16. MobileQuant: Mobile-friendly Quantization for On-device Language Models \- arXiv, https://arxiv.org/html/2408.13933v1
17. The Complete Guide to LLM Quantization with vLLM: Benchmarks & Best Practices, https://jarvislabs.ai/blog/vllm-quantization-complete-guide-benchmarks
18. Q-DiT: Accurate Post-Training Quantization for Diffusion Transformers \- arXiv, https://arxiv.org/html/2406.17343v2
19. Q-DiT: Accurate Post-Training Quantization for Diffusion Transformers \- arXiv, https://arxiv.org/html/2406.17343v1
20. Q-DiT: Accurate Post-Training Quantization for Diffusion Transformers, https://mn.cs.tsinghua.edu.cn/xinwang/PDF/papers/2025\_Q-DiT%20Accurate%20Post-Training%20Quantization%20for%20Diffusion%20Transformers.pdf
21. Quantization \- CRAN, https://cran.r-project.org/web/packages/ggmlR/vignettes/quantization.html
22. llama.cpp: Definations of Q2\_K, Q3\_K, Q4\_K, Q5\_K, Q6\_K, and Q8\_K Structures \- EADST, http://eadst.com/blog/232
23. The Outlier Problem: How a Few Rogue Numbers Are Breaking LLM Quantization \- Medium, https://medium.com/@serahabensour/the-outlier-problem-how-a-few-rogue-numbers-are-breaking-llm-quantization-510a25fe2ed4
24. SPQR: A SPARSE-QUANTIZED REPRESENTATION FOR NEAR-LOSSLESS LLM WEIGHT COMPRESSION \- ICLR Proceedings, https://proceedings.iclr.cc/paper\_files/paper/2024/file/1787533e171dcc8549cc2eb5a4840eec-Paper-Conference.pdf
25. SqueezeLLM: Dense-and-Sparse Quantization \- arXiv, https://arxiv.org/html/2306.07629v1
26. SqueezeLLM: Dense-and-Sparse Quantization \- arXiv, https://arxiv.org/pdf/2306.07629
27. An Efficient Inference Kernel for SpQR \- TU Graz repository, https://repository.tugraz.at/publications/marc21/x84kd-60h12/files/87816.pdf?download=1
28. From 2:4 to 8:16 sparsity patterns in LLMs for Outliers and Weights with Variance Correction \- arXiv, https://arxiv.org/pdf/2507.03052
29. OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Models, https://ojs.aaai.org/index.php/AAAI/article/view/29237/30335
30. \[2306.02272\] OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Models \- arXiv, https://arxiv.org/abs/2306.02272
31. Algorithm--Hardware Co-Design of Digital Compute-in-Memory Architecture Supporting Flexible and Temporal N:M Sparsity, https://www.computer.org/csdl/journal/si/5555/01/11506321/2ggMy22g8x2
32. From 2:4 to 8:16 sparsity patterns in LLMs for Outliers and Weights with Variance Correction, https://arxiv.org/html/2507.03052v1
33. A practical guide to INT4 quantization for SLMs: GPTQ vs AWQ, Olive, and real‑world results, https://medium.com/data-science-at-microsoft/a-practical-guide-to-int4-quantization-for-slms-gptq-vs-awq-olive-and-real-world-results-2f63d6963d1d
34. Which Quantization Method Is Best for You?: GGUF, GPTQ, or AWQ... | E2E Networks, https://www.e2enetworks.com/blog/which-quantization-method-is-best-for-you-gguf-gptq-or-awq
35. AWQ — Activation-Aware Weight Quantization \- ZeroEntropy, https://zeroentropy.dev/concepts/awq/
36. AWQ: Activation-Aware Weight Quantization \- Lei Mao's Log Book, https://leimao.github.io/blog/AWQ-Activation-Aware-Weight-Quantization/
37. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration \- Self-supervised Models, https://self-supervised.cs.jhu.edu/fa2024/files/presentations/10-10-Quantization-Li-Ramesh.pdf
38. AWQ: Activation-Aware Weight Quantisation \- Hamza's Blog, https://hamzaelshafie.bearblog.dev/awq-activation-aware-weight-quantisation/
39. Introduction to AWQ Algorithm \- AMD Quark Documentation\!, https://quark.docs.amd.com/latest/pytorch/awq\_document.html
40. GPTQ vs AWQ quantization \- Newline, https://www.newline.co/@zaoyang/gptq-vs-awq-quantization--d792476e
41. Faster LLMs with Quantization \- How to get faster inference times with quantization \- bitbasti, https://bitbasti.com/blog/faster-llms-with-quantization
42. GGUF vs AWQ vs GPTQ vs MLX: LLM Quant Formats 2026 \- Digital Applied, https://www.digitalapplied.com/blog/gguf-vs-awq-vs-gptq-vs-mlx-llm-quantization-formats-2026
43. LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Models \- arXiv, https://arxiv.org/html/2310.08659
44. L4Q: Parameter Efficient Quantization-Aware Fine-Tuning on Large Language Models \- ACL Anthology, https://aclanthology.org/2025.acl-long.99.pdf
45. (Part 2\) Fine-tuning with QAT, QLoRA, and float8 — torchao 0.17 documentation, https://docs.pytorch.org/ao/stable/eager\_tutorials/finetuning.html
46. Master LoRA and QLoRA: Fine-Tuning LLMs on Consumer GPUs | Let's Data Science, https://letsdatascience.com/blog/fine-tuning-llms-with-lora-and-qlora-complete-guide
47. LoTA-QAF: Lossless Ternary Adaptation for Quantization-Aware Fine-Tuning \- NIPS, https://papers.nips.cc/paper\_files/paper/2025/file/9a16935bf54c4af233e25d998b7f4a2c-Paper-Conference.pdf
48. PEFT Beyond QLoRA: DoRA, GaLore & LoftQ Compared \- AppScale Blog, https://appscale.blog/en/blog/parameter-efficient-fine-tuning-peft-lora-qlora-dora-galore-2026
49. LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Models \- Liner, https://liner.com/review/loftq-lorafinetuningaware-quantization-for-large-language-models
50. LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Models \- arXiv, https://arxiv.org/html/2310.08659v1
51. LoftQ: Reimagining LLM fine-tuning with smarter initialization \- Microsoft Research, https://www.microsoft.com/en-us/research/blog/loftq-reimagining-llm-fine-tuning-with-smarter-initialization/
52. GPTQ-intrinsic LoRA: A Near-optimal Algorithm for Low-precision Quantization with Low-rank Adaptation \- arXiv, https://arxiv.org/html/2606.01412v1
53. QERA: an Analytical Framework for Quantization Error Reconstruction \- OpenReview, https://openreview.net/forum?id=LB5cKhgOTu¬eId=KkgmTCe65a
54. QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models, https://openreview.net/forum?id=WvFoJccpo8
55. QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models \- arXiv, https://arxiv.org/abs/2309.14717
56. rage-quant \- crates.io: Rust Package Registry, https://crates.io/crates/rage-quant
57. core::arch::wasm32 \- Rust, https://doc.rust-lang.org/beta/core/arch/wasm32/index.html
58. v128: Wasm value type \- WebAssembly \- MDN Web Docs \- Mozilla, https://developer.mozilla.org/en-US/docs/WebAssembly/Reference/Value\_types/v128
59. WASM SIMD: Browser Support, Features, Limitations | TestMu AI (Formerly LambdaTest), https://www.testmuai.com/learning-hub/wasm-simd-browser-support/
60. core::arch::wasm32 \- Rust, https://doc.rust-lang.org/core/arch/wasm32/index.html
61. Relaxed Integer Dot Product instructions · Issue \#52 · WebAssembly/relaxed-simd \- GitHub, https://github.com/WebAssembly/relaxed-simd/issues/52
62. NumKong: 2'000 Mixed Precision Kernels For All | Ash's Blog, https://ashvardanian.com/posts/numkong/
63. Module relaxed\_simd \- core::core\_arch::wasm32 \- std \- Rust, https://stdrs.dev/nightly/x86\_64-unknown-linux-gnu/core/core\_arch/wasm32/relaxed\_simd/index.html
64. i16x8\_relaxed\_dot\_i8x16\_i7x16 in core::arch::wasm32 \- Rust, https://www.kernel.org/doc/rustdoc/next/core/arch/wasm32/fn.i16x8\_relaxed\_dot\_i8x16\_i7x16.html
65. WASM \+ SIMD for On-Device AI: Private, Fast, Offline | by Thinking Loop | Medium, https://medium.com/@ThinkingLoop/wasm-simd-for-on-device-ai-private-fast-offline-3ef82c47172d
66. Guides \- MiRust, https://mirust.com/guides/
67. Implementation \- MiRust, https://mirust.com/implementation/
68. Tiny Model Composer \- MiRust, https://mirust.com/composer/
69. Models \- MiRust, https://mirust.com/models/
70. Khushiyant/tenso: High-performance zero-copy tensor serialization for Fastest Transmission \- GitHub, https://github.com/Khushiyant/tenso
71. \[Project\] Running quantized BERT in the browser via WebAssembly (Rust \+ Candle) for local Semantic Search : r/LocalLLaMA \- Reddit, https://www.reddit.com/r/LocalLLaMA/comments/1q9j0r8/project\_running\_quantized\_bert\_in\_the\_browser\_via/
72. 7 WebGPU \+ WASM Plays for Private On-Device ML | by Bhagya Rana | Medium, https://medium.com/@bhagyarana80/7-webgpu-wasm-plays-for-private-on-device-ml-a114bbba8c58
73. core::arch::wasm64 \- Rust, https://doc.rust-lang.org/core/arch/wasm64/index.html
74. Documentation \- MiRust, https://mirust.com/docs/
75. ggml-cpu-aarch64.cpp, https://git.ustc.edu.cn/ustc-os-lab/llama.cpp/-/blob/b5184/ggml/src/ggml-cpu/ggml-cpu-aarch64.cpp