Quantization-Aware Distillation And Behavioral Fidelity Across Native And Browser Runtimes

Full report

On this page

1. Executive Recommendation and Assumptions

The deployment of small language models operating within the highly constrained 0.3B to 1.5B parameter regime demands a radical departure from the post-training compression heuristics traditionally applied to macroscopic architectures. When targeting ubiquitous execution environments—specifically native CPU architectures and browser-hosted WebAssembly (WASM) runtimes—the standard industry practice of conflating successful file loading, uncorrupted token generation, and nominal perplexity degradation with the preservation of useful conversational behavior is fundamentally flawed. This report establishes that aggressive quantization of compact models induces severe representational drift, catastrophic alignment amnesia, and recurrent state compounding errors that remain entirely masked by aggregate language modeling losses. As an independent model-compression scientist and numerical-reliability engineer, I recommend that compact student models targeting CPU and WASM boundaries must abandon standalone post-training quantization in favor of Centered Kernel Alignment-guided Quantization-Aware Distillation (CKA-QAD). This approach jointly optimizes output distributions and internal geometric representations, thereby protecting the deep semantic hierarchies required for structured JSON, code generation, and ordinary helpfulness. Furthermore, it is assumed that the source model, the adapted model, the merged model, the quantized artifact, and the final deployable .slm file constitute entirely distinct cryptographic subjects. Their lineage, alongside the tokenizer, prompting template, quantization converter, runtime binaries, sampling profiles, and evaluation manifest, must be cryptographically bound into an immutable Merkle-directed acyclic graph. In accordance with strict operating parameters, no assertions in this report rely upon the inspection, execution, or validation of private TinyRustLM code, proprietary local models, unreleased hardware, or unpublished evaluation metrics. All conclusions are derived from public primary literature, reference implementations, and framework documentation retrieved through July 2026\. Finally, the behavioral fidelity standards proposed herein assume that raw outputs generated by the WASM or native runtimes may not be repaired, retried until cosmetically attractive, or structurally completed by post-processors prior to scoring. The artifact must natively demonstrate numerical stability and alignment integrity.

2. Quantization-Method Comparison

The quantization methodologies applicable to small local models bifurcate into Post-Training Quantization (PTQ) and training-aware paradigms, each imposing distinct requirements on calibration data, computational overhead, and hardware-specific kernel support. In the 0.3B to 1.5B parameter range, these methods exhibit drastically divergent efficacy due to the inherently limited parameter capacity and unique architectural designs characterizing small language models. Post-Training Quantization operates by projecting pre-trained floating-point weights into a discrete low-precision grid without backpropagation. Symmetric weight-only quantization maps the continuous range to a symmetric integer range around zero, eliminating the need to store or compute zero-points during inference, which significantly accelerates matrix multiplications on constrained CPU architectures. Asymmetric quantization tracks both a scaling factor and a zero-point, offering higher representational fidelity for distributions skewed heavily in one direction. These schemes can be applied per-tensor, per-channel, or group-wise. Group-wise quantization (e.g., [Figure omitted from source export]) partitions individual channels into smaller blocks, each with its own scale and zero-point. While group-wise schemes mitigate the impact of localized activation outliers, they incur a substantial metadata tax; at 3-bit or 4-bit precision, the bookkeeping bits required for group scales can consume nearly half the total bit budget1. Advanced PTQ techniques like GPTQ leverage Hessian approximations to iteratively quantize weights while mathematically compensating for the induced error in the remaining unquantized weights within a layer. GPTQ relies on a calibration dataset to compute the inverse Hessian, ensuring that the quantization error is minimized relative to the expected activation distribution2. However, classical GPTQ often omits the first-order Taylor expansion term, assuming the network has converged to a local optimum. In small language models, this assumption fails. The First-Order Enhanced Method (FOEM) demonstrates that accumulated deviations in unquantized SLM weights generate non-negligible gradients, making first-order compensation mandatory to stabilize low-bit representation3. Activation-aware methods, such as AWQ and SmoothQuant, aim to migrate the quantization difficulty away from highly volatile activations toward static weights. SmoothQuant applies a channel-wise scaling operation mathematically equivalent to the original network, smoothing activation outliers before applying static quantization4. AWQ protects specific salient weights identified via representative calibration prompts, acknowledging that a small fraction of weights disproportionately influences the output distribution5. However, the reliance on dynamic per-tensor activation quantization introduces a severe security vulnerability termed "Quantamination." When dynamic quantization parameters are computed globally across a shared activation tensor in batched inference, data-dependent scaling factors leak information across the batch boundary, enabling adversarial recovery of concurrent prompt inputs6. Consequently, SLMs targeting secure runtimes must rely strictly on static activation quantization or deploy solely in isolated, single-batch browser environments. Training-aware methods incorporate simulated quantization noise directly into the forward pass using straight-through estimators. Full Quantization-Aware Training (QAT) updates trainable fake quantization nodes alongside the model weights via backpropagation. While QAT yields the highest accuracy for extreme compression (e.g., ternary 1.58-bit models), its reliance on massive training corpora and full gradient states makes it computationally prohibitive7. QLoRA mitigates this by freezing a 4-bit quantized base model and training a small set of high-precision Low-Rank Adapters, effectively achieving mixed-precision adaptation. Quantization-Aware Fine-Tuning (QA-SFT) and Quantization-Aware Distillation (QAD) offer superior alternatives for SLMs. QAD aligns the quantized student model against a frozen high-precision teacher. It relies heavily on teacher logits rather than ground-truth labels, transferring the dark knowledge of the output distribution via Kullback-Leibler (KL) divergence8. QAD avoids the complex multi-stage supervised fine-tuning required by traditional QAT, providing a highly stable mechanism for sub-2B parameter models.

3. Tensor-Sensitivity and Hybrid-State Analysis

Small language models manifest unique fragility profiles compared to their macroscopic counterparts. The assumption that outlier elimination remains the sole bottleneck for SLM quantization lacks empirical validation, as compact architectures are trained under distinct data mixtures and learning rate schedules that inherently alter weight distribution characteristics10. Embeddings and output heads are exceptionally fragile components. Because the embedding matrix maps discrete, semantically sparse token IDs to a continuous dense latent space, and the output head executes the reverse transformation over tens of thousands of vocabulary items, minor numerical perturbations in these tensors catastrophically degrade the model's fundamental vocabulary distribution. These layers must universally be exempted from low-bit quantization, remaining in BF16 or FP16 formats to prevent vocabulary collapse12. Attention projections exhibit an asymmetric sensitivity profile. Analysis of the Key (K) and Value (V) caches reveals that Key projections contain extreme channel-wise outliers that dominate the quantization range, while Value projections vary predominantly on a per-token basis across the temporal dimension. Quantization methods must exploit this asymmetric sensitivity by applying per-channel quantization to Keys and per-token quantization to Values, preserving retrieval accuracy at extended context lengths1. The feed-forward network (FFN) gates utilized in SwiGLU activation functions are highly sensitive to saturation. Because the sigmoid-like gate determines the magnitude of information flow, aggressive quantization of the gate's input projection risks irreversible state collapse where the gate permanently locks shut, effectively severing the non-linear capacity of the block. Normalization layers (e.g., RMSNorm) are similarly sensitive to dynamic range truncation. If the variance calculation suffers from quantization-induced underflow, the entire layer output becomes wildly unstable. Norms must remain in high-precision floating-point formats. In hybrid models integrating convolutions and recurrent states, the convolution kernels acting as short-term memory over rolling token windows are highly sensitive to phase shifts. Truncating the precision of the 1D convolution weights alters the temporal receptive field, disrupting the model's ability to smoothly transition between adjacent tokens. Furthermore, small outlier channels within the main projection matrices dictate the preservation of internal geometry. The Outlier Suppression+ (OS+) framework demonstrates that activation outliers are highly concentrated and asymmetric across specific channels14. If early layers suffer local quantization degradation, these outlier channels dominate the representational range, suppressing critical information required by downstream computations4. Specific unknowns remain regarding how hardware-specific WASM kernels optimize these irregular outlier channels without triggering fallback scalar execution paths.

4. Hybrid Models and Recurrent State Analysis

Architectures that replace or augment dense attention with State Space Models (SSMs) like Mamba or linear recurrent attention systems like RWKV introduce severe temporal quantization challenges. In a discrete SSM, the hidden state evolves according to the recurrence relation [Figure omitted from source export], where the output is read as [Figure omitted from source export]. Unlike standard transformer attention, where quantization noise is isolated per token pair and computed in a single matrix multiplication, SSMs accumulate error multiplicatively over time. The accumulated drift at time [Figure omitted from source export] in a quantized SSM is expressed as [Figure omitted from source export], where [Figure omitted from source export] represents the quantization noise injected at step [Figure omitted from source export]16. Because errors mix continuously through the state transition rather than appearing additively, post-hoc linear corrections—such as Kalman gain amplification or James-Stein shrinkage—fail entirely in SSMs. These traditional error-correction techniques monotonically increase perplexity as the temporal horizon expands16. Consequently, robust hybrid quantization strictly demands quantization-aware training or distillation to force the weights to naturally compensate for the accumulated numerical drift. The convolution history preceding the SSM state update is equally critical. Mamba utilizes a causal 1D convolution over a localized token window before computing the input-dependent parameters [Figure omitted from source export], [Figure omitted from source export], and [Figure omitted from source export]19. If the convolution outputs are aggressively quantized, the gating mechanisms determining [Figure omitted from source export] (which controls how much the state forgets the past versus ingests the present) suffer from gating saturation. A saturated [Figure omitted from source export] causes the model to either entirely overwrite its state with noise or permanently freeze its memory, leading to catastrophic context decay. In multi-input, multi-output (MIMO) formulations and complex-valued state space models (e.g., Mamba-3), the state tracks phase and magnitude dynamics that demand elevated precision20. The recurrent state size for a 1.5B parameter linear-attention model typically requires 2 to 4 GB of fixed memory21. If the accumulator precision for the recurrent state matrix drops below FP32, the continuous integration of fractional values leads to catastrophic resonance and NaN explosions. The reset behavior of these models in multi-turn environments is also paramount; lingering activations in the fixed state matrix from previous turns will corrupt subsequent generations if the state is not explicitly zeroed out or decayed precisely according to the prompt boundaries. For mixed attention/recurrent blocks, the integration of precise KV cache elements with continuous SSM states creates a dual-precision requirement. The attention KV cache can tolerate asymmetric per-channel/per-token low-bit quantization, but the recurrent blocks interleaved between them must maintain high-precision accumulators to bridge the temporal gaps without shedding context.

5. Training and Distillation Pipeline Comparison

Deploying a mathematically sound SLM relies heavily on the specific sequence of adaptation and compression. Traditional pipelines often follow a "Train then PTQ" or "SFT then PTQ" methodology. While computationally inexpensive, these sequential pipelines maximize the risk of both representational drift and alignment degradation. When PTQ is applied as an afterthought, the model has no mechanism to adjust its manifold to the restricted discrete grid, resulting in a severe degradation of reasoning capabilities5. The "PTQ then adapter tuning" pipeline (e.g., QLoRA) addresses this by freezing the quantized base weights and training high-precision low-rank adapters. While effective for adapting large models to specific tasks, merging the adapters back into a fully quantized artifact for WASM deployment often resurrects the original quantization errors, as the adapter weights must themselves be quantized during the final export. "Distill then PTQ" suffers from the same sequential flaws. The model learns the teacher's distribution in high precision, only to have that delicate probability mass shattered by a subsequent PTQ pass. Quantization-Aware SFT and Quantization-Aware Distillation (QAD) represent vastly superior pipelines. By applying simulated quantization noise during the fine-tuning or distillation phase, the optimizer naturally routes critical information away from fragile quantization bins. However, standard QAD suffers from a hidden flaw when utilizing solely output-level KL-divergence. Research into NVFP4 and low-bit integer distillation proves that KL-only matching masks severe internal geometric degradation. The student model can produce output logits that match the teacher perfectly on the training set, while internally scrambling its intermediate feature representations8. This "representational drift" acutely harms complex reasoning and structural JSON/code generation, which rely on deep semantic hierarchies that do not immediately manifest in single-token predictions. The optimal pipeline for CPU and browser execution is direct teacher-student training against the deployment precision utilizing CKA-QAD (Centered Kernel Alignment-guided Quantization-Aware Distillation). CKA-QAD explicitly aligns the layer-wise Gram matrices between the high-precision teacher and the quantized student, preserving the internal representational geometry8. Because CKA is invariant to isotropic scaling and orthogonal transformations, it provides a stable regularizer that prevents the internal features of the student from collapsing, even as the numerical ranges of activations shift during low-bit training. This pipeline directly optimizes the artifact for the exact precision matrix of the deployment environment.

6. Calibration-Data Contract

The integrity of post-training and quantization-aware parameters depends entirely on the calibration data contract. Inadequate sample selection distorts the dynamic range of weights and activations, leading to severe precision loss that permanently disables the model24. A strict calibration data contract must enforce representativeness and diversity in the activation space without contaminating downstream evaluation metrics. Benchmark contamination is an escalating crisis; ingesting evaluation prompts into the calibration set artificially deflates perplexity and generates false confidence in the quantized artifact's general capabilities25. Private holdouts, evaluation test sets, and their semantic paraphrases must be cryptographically excluded from calibration manifests. If a model overfits to the calibration data, it learns to optimize the quantization bins exclusively for those specific token trajectories, completely collapsing when presented with novel user dialogue26. High-quality calibration requires targeted sequence lengths. Limiting calibration to 512 tokens fails to expose the quantizer to the long-tail activation growth that occurs during extended context decay. Calibration sequences must span at least 2,048 tokens to accurately capture the dynamic range of attention and recurrent accumulators. The linguistic mixture must encompass multiple languages, syntactically rigid code snippets, deeply nested JSON structures, and ordinary conversational dialogue. Outlier coverage is paramount. The calibration set must naturally trigger the extreme activation spikes present in the model's operational envelope. For highly sensitive SLMs, Family-Aware Quantization (FAQ) offers a robust mechanism28. FAQ inputs raw calibration text into a massive, unquantized teacher from the exact same model lineage to regenerate a high-fidelity calibration set. This synthetic regeneration aligns the target model’s internal activation patterns to a highly consistent knowledge system, suppressing idiosyncratic statistical noise that frequently causes Min-Max and percentile-based quantizers to fail.

7. Precision and Kernel Matrix

To accommodate the diverse hardware capabilities of native CPUs, ARM NEON instruction sets, and WASM-based browser engines, a rigorously defined tiered precision matrix is required. Below is the specified quantization configuration targeting optimal fidelity for a sub-2B parameter SLM.

This matrix explicitly forbids the aggressive quantization of tensors responsible for dimensional mapping and temporal state preservation. WASM SIMD instructions operating on WebGPU or native CPU backends require explicit 32-byte block alignments to ensure coalesced memory access, necessitating padding within the serialized payload.

8. Artifact-Size and Memory Formulas

The theoretical limits of execution environments strictly dictate deployment feasibility. Browser-based WebAssembly environments impose rigid addressing constraints that fundamentally alter how models must be packaged and loaded. While the Memory64 proposal enables 64-bit addressing, standard WASM deployments are strictly bounded by a 4GB (32-bit) linear memory limit, which is frequently further constrained by browser-specific ArrayBuffer maximums of approximately 2GB29. The total required runtime memory ([Figure omitted from source export]) for an attention-based SLM in a WASM environment is defined as: [Figure omitted from source export] For Mamba and linear hybrid architectures, the temporally growing [Figure omitted from source export] is replaced by a fixed-size recurrent state footprint [Figure omitted from source export]: [Figure omitted from source export]

• Artifact Alignment ([Figure omitted from source export]): GGUF formatting utilizes the general.alignment metadata key (defaulting to 32 bytes) to ensure that tensor payloads are aligned strictly to memory boundaries32. This alignment is critical for enabling zero-copy mmap executions and ensuring that SIMD vector instructions do not trigger unaligned access faults.
• State and KV Cache ([Figure omitted from source export] vs [Figure omitted from source export]): A Transformer KV cache scales as [Figure omitted from source export]. The size per layer is calculated as [Figure omitted from source export]. A 1.5B parameter model operating at an 8K context window requires hundreds of megabytes of dynamic allocation. Conversely, Mamba’s state size scales as [Figure omitted from source export] relative to sequence length, consuming a constant [Figure omitted from source export] GB across all layers irrespective of the context horizon21. While Mamba is highly efficient for long contexts, its baseline fixed state size can immediately threaten the 4GB WASM linear memory ceiling if not aggressively managed.
• De-quantization Scratch ([Figure omitted from source export]): The runtime requires dedicated intermediate buffers to temporarily expand quantized integer blocks (e.g., Q4) back into FP16 or FP32 formats just before performing the matrix multiplication.
• JS Host Copies ([Figure omitted from source export]): This is the most dangerous variable in browser deployments. Without properly configured SharedArrayBuffer implementations and corresponding Cross-Origin-Opener-Policy (COOP) isolation headers, passing tensor boundaries between the JavaScript main thread and WASM Web Workers invokes deep structural cloning34. This duplication ([Figure omitted from source export]) essentially doubles the memory footprint of the transferred payload, instantaneously triggering out-of-memory (OOM) fatal exceptions in the V8 engine. Furthermore, V8 JavaScript engine Just-In-Time (JIT) compilation pages and baseline RSS overhead dynamically steal hundreds of megabytes from the theoretical heap limit37.

9. Numerical Conformance Suite

Mathematical equivalence must be tracked meticulously across the entire deployment pipeline. Conformance is defined as the verification of deterministic outputs as the model transitions across distinct execution boundaries: Source Framework [Figure omitted from source export] Merged .safetensors [Figure omitted from source export] Quantization Converter [Figure omitted from source export] Native Runtime (e.g., llama.cpp) [Figure omitted from source export] WASM Runtime [Figure omitted from source export] Browser Canvas/DOM.

1. Logit Drift and KL Divergence: Log-likelihood vectors provide an absolute geometric scale for comparing model checkpoints38. Quantization inherently causes logit drift. High Jensen-Shannon or KL divergence at high-entropy token positions forces quantized models to sample "overthinking" tokens (e.g., "Wait," "Alternatively"), ballooning generation length and destroying logical coherence40. Conformance tests must calculate the token-level KL divergence between the native FP16 execution and the WASM Q4 artifact. To pass conformance, the average KL divergence must remain [Figure omitted from source export] bits/byte across a standardized test corpus39.
2. Top-k Rank Agreement: For a deterministic seed (temperature [Figure omitted from source export]), the top-5 predicted token IDs must match the unquantized source model precisely across 98% of standard reasoning sequences. Output probability swapping outside the top-5 is permissible only if the probability mass is mathematically negligible.
3. Generated Token Agreement: The final decoded text strings must demonstrate strict byte-level equivalence between the native C++ runtime and the WASM browser runtime for identical prompt states.
4. NaN Propagation and State Reset: In recurrent models, lingering activations in the fixed state matrix from previous conversation turns can cause NaN explosions if the decay matrices are not explicitly flushed. Multi-turn WASM conformance must trigger a manual state reset via the runtime API and mathematically verify that the logits produced at [Figure omitted from source export] match cold-start logits identically.

10. Behavioral Fidelity and Statistical Design

Successful file loading and uncorrupted token generation are insufficient proofs of utility. A model that speaks fluent English but suffers from safety amnesia or structural hallucination is a failed artifact. The quantized model must undergo rigorous behavioral fidelity testing designed to isolate deterministic precision degradation from hardware flakiness and sampling variance. The evaluation suite must employ fixed-seed sampling, executing unseen paraphrases of prompts to evaluate the following domains without post-processing or repairing the raw outputs:

• Structured Output Validity: Strict adherence to JSON schema geometries and syntactical code compilation. The model must correctly close nested brackets and utilize correct API signatures without falling back to unstructured conversational text.
• Current Preference & Source Authority: Evaluating whether the model retains its RLHF preference for helpfulness and harmlessness. "Alignment degradation" is flagged if the quantized model obeys hostile memory injections, generates unsafe outputs, or provides restricted information that the unquantized source actively refused5.
• Ambiguity and Summarization: Quantized models suffer heavily from context omission in the middle of long prompts. Summaries of complex documents must be evaluated via [Figure omitted from source export]\-gram overlap or automated LLM-as-a-judge cross-examinations against the source artifact to ensure critical nuances and caveats are not erased.
• Unavailable Resources: The model must accurately maintain the capacity to state "I do not know" or recognize when a prompt asks for information outside its operational context, rather than hallucinating plausible but incorrect facts due to flattened logit distributions.

To separate deterministic precision drift from infrastructure flakiness, the testing regime must implement regression budgets. Repeated-seed statistics across 10 iterations per prompt must mathematically prove that accuracy on targeted holdouts does not degrade by more than a 3% absolute margin relative to the FP16 reference model.

11. Native/Browser Performance Methodology

Evaluating performance requires testing on physical devices—such as mobile ARM processors and low-power x86 CPUs—rather than relying on theoretical FLOP calculations or simulated environments. Performance metrics must strictly isolate the following phases:

• Model Load and Peak Memory: The time required to pull the artifact from disk or IndexedDB, allocate the WebAssembly linear memory, and hydrate the tensors. Peak memory must be tracked via the browser's Performance API to identify transient V8 heap spikes37.
• Time-To-First-Token (TTFT) and Prefill: The processing speed of the prompt (tokens/sec). In WASM, this phase often incurs severe latency due to the lack of highly optimized, hardware-specific matrix multiplication kernels compared to native CUDA/Metal backends.
• Decode Throughput and Steady State: The autoregressive generation speed (tokens/sec) once the KV cache or recurrent state is initialized.
• UI Long Tasks and Thread Blocking: Browser execution must ensure that the Web Worker orchestration does not freeze the Document Object Model (DOM). Synchronous WASM tensor allocation or deep memory cloning that exceeds 50 milliseconds triggers the UI Long Tasks API, resulting in a degraded, frozen user experience41.
• Energy and Thermal Throttling: Sustained token-generation on uncooled mobile devices inevitably triggers thermal throttling. Performance claims must specify the drop in decode throughput over a continuous 10-minute generation window to account for mobile heat dissipation limits.

12. Receipt and Provenance Graph

To establish verifiable trust and prevent "catalog theater"—where identical or fundamentally broken byte-streams are repackaged under deceptive naming conventions to feign catalog diversity—the deployment pipeline must produce a content-addressed receipt graph. Derived from verifiable execution protocols like Portable Agent Memory and CommitLLM43, this provenance architecture utilizes a Merkle-Directed Acyclic Graph (DAG) backed by cryptographic hashing (e.g., SHA-256 or BLAKE3).

• Leaves (Inputs): Checksums of the reference base weights, LoRA adapter weights, raw tokenizer vocabularies, prompt templates, and the exact calibration text manifest utilized during QAD or PTQ.
• Nodes (Operations): The hashed configuration of the quantization converter, random seeds, specific group sizes ([Figure omitted from source export]), alignment metadata, and the runtime version.
• Root (Output): The final .slm or .gguf file hash, cryptographically bound to its parent nodes.

If any upstream parameter is altered—such as substituting an invalid calibration set containing evaluation contamination, or utilizing a deprecated conversion script—the downstream Merkle root is immediately invalidated. This provides an immutable cryptographic receipt guaranteeing that the browser-based SLM executed by the end-user is mathematically and behaviorally identical to the artifact approved by the engineering team.

13. Promotion Thresholds and Stop Rules

A quantization tier that successfully loads and produces text is not automatically qualified for deployment. A deployment tier must be halted and stripped of "production" status if it triggers any of the following strict stop rules:

1. Zero-Ratio Collapse: The sparsity of any specific linear layer within an ultra-low precision (e.g., ternary) artifact exceeds the self-regulating threshold (e.g., [Figure omitted from source export]), resulting in irreversible representational capacity loss16.
2. Representational Drift: The layer-wise Centered Kernel Alignment (CKA) metric between the artifact and the unquantized teacher drops below [Figure omitted from source export] in critical mid-depth FFN or Mamba state-update blocks, indicating geometric feature collapse9.
3. Alignment Amnesia: The model complies with [Figure omitted from source export] of adversarial refusal-bypass prompts (hostile memory or jailbreaks) that the unquantized source model successfully recognized and blocked5.
4. Structured Output Failure: Output syntax conformity drops below 95% on structured JSON extraction or code compilation tasks.
5. Phantom Cataloging: A newly generated precision tier (e.g., a Q4\_K\_M variant vs a Q4\_K\_S variant) exhibits a KL divergence of [Figure omitted from source export] bits/byte compared to an existing tier, indicating no statistically meaningful behavioral difference. Relabeled, behaviorally identical bytes must be rejected. Multiple precisions of a single checkpoint only count as useful diversity if they present a measurable, Pareto-optimal tradeoff between WASM memory consumption and behavioral fidelity.

14. Unknowns Requiring Local Models or Hardware

To fully operationalize this pipeline across arbitrary consumer hardware, specific systemic unknowns remain that strictly require access to local physical hardware and raw inference outputs:

• V8 Engine Microarchitecture Quirks: The exact magnitude of transient RSS memory spikes triggered by WASM memory initialization and JS-to-WASM bridge traversals in modern browsers (e.g., Node.js 24 / V8 13.6 allocation patterns). These invisible allocations frequently cause OOM crashes despite the WASM heap remaining below 4GB37.
• Hardware-Specific SIMD Fallbacks: The degree to which specific browser WASM runtimes correctly map quantized vector inner-products to underlying ARM NEON or x86 AVX2 instructions without silently falling back to slow, scalar software emulation47.
• Mobile Thermal Dynamics: The exact relationship between sustained WebGPU/WASM execution and battery drain on physical iOS and Android devices, which dictates practical generation length limits before operating system watchdogs terminate the browser process.

15. Annotated Primary-Source Bibliography

Ganesaraja, R., et al. (June 2026). Ternary Mamba: Grouped Quantization-Aware Training of W1.58A16 State Space Models. (Retrieved July 2026). Identifies the zero-ratio collapse phenomenon in SSMs and proves mathematically that post-hoc linear corrections (Kalman, James-Stein) fail for recurrent drift. Proposes fixed absmean scaling.

Mamba-3, RWKV-7, and Linear Attention GPU Deployment Guides (2026). Details the O(1) inference memory complexity and precise recurrent state size formulas required for hybrid architectures compared to quadratic Transformers. Explores complex-valued state tracking.

ZeroQAT: A Zeroth-Order Optimization Framework for Quantization-Aware Training (September 2025). Introduces forward-only gradient estimation techniques for memory-efficient QAT.

KIVI & KVQuant (2024). Establishes empirically that attention Key projections exhibit massive outlier channels, while Value projections vary temporally, necessitating asymmetric per-channel and per-token KV quantization.

Outlier Suppression+ (OS+). Evaluates channel-wise shifting for asymmetry and scaling for outlier concentration in low-bit transformer LLMs to prevent downstream information starvation.

Various WebAssembly, WebGPU, and JS runtime specifications (2025–2026). Details the performance disparity, deep-copy memory overhead without SharedArrayBuffer, and the absolute necessity of Web Worker orchestration for main-thread offloading.

ONNX Runtime Web and V8 Engine limits (2026). Documents the historical 2GB ArrayBuffer and 4GB 32-bit WASM memory boundaries, detailing the integration of Memory64 proposals.

Quantamination: The Privacy Risks of Dynamic Batched Quantization (April 2026). Discovers that dynamic per-tensor activation quantization acts as a malicious side-channel, leaking concurrent batch inputs in shared environments.

FAQ: Family-Aware Quantization (2026). Validates that high-fidelity calibration datasets can be synthesized using larger teacher models residing in the exact same lineage to prevent PTQ precision collapse.

SLMQuant: Benchmarking Small Language Model Quantization (November 2025). Proves that SLMs exhibit fundamentally different quantization bottlenecks and numerical sensitivities than macroscopic 70B+ LLMs, invalidating pure LLM-PTQ transfers.

Portable Agent Memory & CommitLLM (May 2026). Introduces Merkle-DAG content-addressed provenance graphs to cryptographically verify inference traces, configuration state, and origin weights. (Pinned reference: 25541e8).

Tu, F., et al. (June 2026). Beyond Output Matching: Preserving Internal Geometry in NVFP4 LLM Distillation. (Retrieved July 2026). Proves that KL-only distillation causes severe representational drift. Proposes CKA-QAD to align intermediate representations.

FOEM: First-Order Enhanced Method (2026). Demonstrates the mathematical necessity of including first-order Taylor expansion terms in post-training quantization to compensate for accumulated weight deviations.

GGUF Specification and Internals (2023–2026). Details the metadata-driven alignment layouts required for optimal zero-copy mmap performance.

Logit Drift and Overthinking in Quantized LLMs (June 2026). Quantifies how token-level KL divergence in compressed models triggers high-entropy hesitation markers, reducing reasoning effectiveness and expanding token budgets.

Log-likelihood vector spaces and KL divergence (2025). Establishes absolute scaling rules for quantifying model drift, proving subdiffusive learning trajectories in distribution space.

Alignment-Aware Quantization (AAQ) (November 2025). Proves that aggressive PTQ silently erases RLHF guardrails (Alignment Degradation), necessitating contrastive loss to preserve safety behaviors.

Node 24 / V8 13.6 Memory Allocation Analysis (2026). Identifies transient RSS spikes and JIT external array buffer allocations that bypass expected JS heap limits.

Benchmark Contamination and Calibration Leakage (May 2026). Highlights the statistical invalidity of overlapping evaluation datasets with calibration data, corrupting downstream compression metrics.

• 16
• 20
• 7
• 1
• 4
• 34
• 29
• 6
• 28
• 10
• 43
• 8
• 3
• 32
• 40
• 38
• 5
• 37
• 25

Works cited

1. Understanding Google's TurboQuant Insight \- Cyrus Radfar, https://cyrusradfar.com/thoughts/turboquant-explainer
2. Quantization Error Propagation: Revisiting Layer-Wise Post-Training Quantization, https://neurips.cc/virtual/2025/poster/117304
3. First-Order Error Matters: Accurate Compensation for Quantized Large Language Models \- AAAI Publications, https://ojs.aaai.org/index.php/AAAI/article/download/40123/44084
4. SpikeLLM: Scaling up Spiking Neural Network to Large Language Models via Saliency-based Spiking \- arXiv, https://arxiv.org/html/2407.04752v1
5. Alignment‑Aware Quantization for LLM Safety \- arXiv, https://arxiv.org/html/2511.07842v2
6. : Dynamic Quantization Can Leak Your Data Across the Batch \- arXiv, https://arxiv.org/html/2604.26505v1
7. ZeroQAT: Your Quantization-aware Training but Efficient \- arXiv, https://arxiv.org/html/2509.00031v1
8. Beyond Output Matching: Preserving Internal Geometry in NVFP4 LLM Distillation \- arXiv, https://arxiv.org/pdf/2606.05682
9. Beyond Output Matching: Preserving Internal Geometry in NVFP4 LLM Distillation \- arXiv, https://arxiv.org/html/2606.05682v2
10. SLMQuant:Benchmarking Small Language Model Quantization for Practical Deployment \- arXiv, https://arxiv.org/pdf/2511.13023
11. SLMQuant: Benchmarking Small Language Model Quantization for Practical Deployment \- arXiv, https://arxiv.org/html/2511.13023v1
12. Small Language Models (SLMs): Comprehensive Guide 2026 \- CogitX, https://cogitx.ai/blog/small-language-models-slms-comprehensive-guide-2026
13. Has anyone tested how quantization hits different capabilities separately? My results are surprising. : r/LocalLLaMA \- Reddit, https://www.reddit.com/r/LocalLLaMA/comments/1us7a22/has\_anyone\_tested\_how\_quantization\_hits\_different/
14. Outlier Suppression+: Accurate quantization of large language models by equivalent and effective shifting and scaling \- OpenReview, https://openreview.net/pdf?id=sZEAMUizsd
15. Enhancing Post-Training Quantization via Future Activation Awareness \- arXiv, https://arxiv.org/html/2602.02538v1
16. Ternary Mamba: Grouped Quantization-Aware Training of W1.58A16 State Space Models, https://arxiv.org/html/2606.18114v1
17. Ternary Mamba: Grouped Quantization-Aware Training of W1.58A16 State Space Models \- arXiv, https://arxiv.org/pdf/2606.18114
18. Both Kalman amplification and James-Stein shrinkage increase PPL... | Download Scientific Diagram \- ResearchGate, https://www.researchgate.net/figure/Both-Kalman-amplification-and-James-Stein-shrinkage-increase-PPL-monotonically-from\_fig2\_407171949
19. Chapter 32: Post-Transformer Architectures \- Mamba and Hybrid Models \- Wayland Zhang, https://waylandz.com/llm-transformer-book-en/chapter-32-post-transformer-architectures/
20. MAMBA-3: IMPROVED SEQUENCE MODELING USING STATE SPACE PRINCIPLES \- OpenReview, https://openreview.net/pdf?id=HwCvaJOiCj
21. Deploy xLSTM and RWKV-7 on GPU Cloud: Linear-Attention Alternatives for Million-Token Context Inference (2026) | Spheron Blog, https://www.spheron.network/blog/xlstm-rwkv7-linear-attention-gpu-cloud-deployment-2026/
22. 2Mamba2Furious: Linear in Complexity, Competitive in Accuracy \- arXiv, https://arxiv.org/html/2602.17363v1
23. (PDF) Beyond Output Matching: Preserving Internal Geometry in NVFP4 LLM Distillation, https://www.researchgate.net/publication/406039642\_Beyond\_Output\_Matching\_Preserving\_Internal\_Geometry\_in\_NVFP4\_LLM\_Distillatio
24. Automatic selection of calibration data for post-training quantization of fake audio detection models \- SPIE Digital Library, https://www.spiedigitallibrary.org/conference-proceedings-of-spie/14163/141630N/Automatic-selection-of-calibration-data-for-post-training-quantization-of/10.1117/12.3110977.full
25. LLM Benchmark Datasets Should Be Contamination-Resistant \- arXiv, https://arxiv.org/html/2605.19999v1
26. The Great Quant Wars of 2025 : r/LocalLLaMA \- Reddit, https://www.reddit.com/r/LocalLLaMA/comments/1khwxal/the\_great\_quant\_wars\_of\_2025/
27. The Poisoned Chalice of LLM Evaluation Report \- ResearchGate, https://www.researchgate.net/publication/408623772\_The\_Poisoned\_Chalice\_of\_LLM\_Evaluation\_Report
28. FAQ: Mitigating Quantization Error via Regenerating Calibration Data with Family-Aware Quantization \- ACL Anthology, https://aclanthology.org/2026.findings-acl.1079.pdf
29. Working with Large Models | onnxruntime, https://onnxruntime.ai/docs/tutorials/web/large-models.html
30. Memory limits in webassembly \- browser \- Stack Overflow, https://stackoverflow.com/questions/40417774/memory-limits-in-webassembly
31. Up to 4GB of memory in WebAssembly \- V8 JavaScript engine, https://v8.dev/blog/4gb-wasm-memory
32. GGUF \- Wikipedia, https://en.wikipedia.org/wiki/GGUF
33. A Short Guide to the GGUF Format \- Gianluca Guida's personal page., http://tlbflush.org/post/2025\_02\_17\_gguf\_weekend/
34. SharedArrayBuffer: Browser Support, Isolation, Use Cases \- TestMu AI, https://www.testmuai.com/learning-hub/sharedarraybuffer-browser-support/
35. WebAssembly Multi-threading Cheat Sheet: WASM & SharedArrayBuffer \- Tech Bytes, https://techbytes.app/posts/wasm-multithreading-sharedarraybuffer-cheat-sheet/
36. Threads and messages with Rust and WebAssembly \- Tweag, https://tweag.io/blog/2022-11-24-wasm-threads-and-messages/
37. backup mail report says INTERRUPTED but it's not ? | XCP-ng and XO forum, https://xcp-ng.org/forum/topic/11721/backup-mail-report-says-interrupted-but-it-s-not/32?page=1
38. Establishing a Scale for Kullback–Leibler Divergence in Language Models Across Various Settings \- arXiv, https://arxiv.org/html/2505.15353v3
39. Establishing a Scale for Kullback–Leibler Divergence in Language Models Across Various Settings \- ACL Anthology, https://aclanthology.org/2026.findings-acl.1163.pdf
40. Quantized Reasoning Models Think They Need to Think Longer, but They Do Not \- arXiv, https://arxiv.org/html/2606.00206v1
41. I Built a Browser-Local AI Assistant in Next.js with WebLLM, WASM, ONNX Runtime, Web Workers, and RAG \- DEV Community, https://dev.to/databro/i-built-a-browser-local-ai-assistant-in-nextjs-with-webllm-wasm-onnx-runtime-web-workers-and-58b5
42. 5 WebAssembly Hacks to Speed Up Your Next Web App | by Thinking Loop | Medium, https://medium.com/@ThinkingLoop/5-webassembly-hacks-to-speed-up-your-next-web-app-1e6d05867710
43. A Protocol for Cryptographically-Verified Memory Transfer Across Heterogeneous AI Agents \- arXiv, https://arxiv.org/pdf/2605.11032
44. A Protocol for Provenance-Verified Memory Transfer Across Heterogeneous LLM Agents \- arXiv, https://arxiv.org/html/2605.11032v1
45. CommitLLM: A Commit-and-Audit Protocol for Open-Weight LLM Inference \- GitHub, https://raw.githubusercontent.com/lambdaclass/CommitLLM/main/paper/main.pdf
46. CommitLLM is a cryptographic commit-and-audit protocol for open-weight LLM inference. \- GitHub, https://github.com/lambdaclass/CommitLLM
47. State Space Models as CPU-Native Neural Network Architectures: Experimental Evidence from ARM NEON Inference with 1.58-bit Quant, https://engrxiv.org/preprint/download/6680/10946/9273
48. \[2606.18114\] Ternary Mamba: Grouped Quantization-Aware Training of W1.58A16 State Space Models \- arXiv, https://arxiv.org/abs/2606.18114
49. Mamba Architecture: State Space Models Explained, https://mambaauthority.com/mamba-architecture-overview
50. Transformers vs Mamba vs Linear Attention: Who Wins Long Context? \- Devansh \- Medium, https://machine-learning-made-simple.medium.com/transformers-vs-mamba-vs-linear-attention-who-wins-long-context-f1dc8ceb5ede
51. Mamba (deep learning architecture) \- Grokipedia, https://grokipedia.com/page/mamba\_deep\_learning\_architecture
52. NeurIPS Poster Preserving LLM Capabilities through Calibration Data Curation: From Analysis to Optimization, https://neurips.cc/virtual/2025/poster/117834
53. Preserving LLM Capabilities through Calibration Data Curation: From Analysis to Optimization | OpenReview, https://openreview.net/forum?id=Tf9eoTIIjh\&referrer=%5Bthe%20profile%20of%20Lihao%20Yin%5D(%2Fprofile%3Fid%3D\~Lihao\_Yin2)
54. (PDF) SLMQuant:Benchmarking Small Language Model Quantization for Practical Deployment \- ResearchGate, https://www.researchgate.net/publication/397701254\_SLMQuantBenchmarking\_Small\_Language\_Model\_Quantization\_for\_Practical\_Deployment
55. ggml/docs/gguf.md at master · ggml-org/ggml \- GitHub, https://github.com/ggml-org/ggml/blob/master/docs/gguf.md
56. How to Calculate the KL Divergence for Machine Learning \- MachineLearningMastery.com, https://machinelearningmastery.com/divergence-between-probability-distributions/